Biases in COVID-19 vaccine effectiveness studies using cohort design

Observational studies on COVID-19 vaccine effectiveness (VE) have provided critical real-world data, informing public health policy globally. These studies, primarily using pre-existing data sources, have been indispensable in assessing VE across diverse populations and developing sustainable vaccination strategies. Cohort design is frequently employed in VE research. The rapid implementation of vaccination campaigns during the COVID-19 pandemic introduced differential vaccination influenced by sociodemographic disparities, public policies, perceived risks, health-promoting behaviors, and health status, potentially resulting in biases such as healthy user bias, healthy vaccinee effect, frailty bias, differential depletion of susceptibility bias, and confounding by indication. The overwhelming burden on healthcare systems has escalated the risk of data inaccuracies, leading to outcome misclassifications. Additionally, the extensive array of diagnostic tests used during the pandemic has also contributed to misclassification biases. The urgency to publish quickly may have further influenced these biases or led to their oversight, affecting the validity of the findings. These biases in studies vary considerably depending on the setting, data sources, and analytical methods and are likely more pronounced in low- and middle-income country (LMIC) settings due to inadequate data infrastructure. Addressing and mitigating these biases is essential for accurate VE estimates, guiding public health strategies, and sustaining public trust in vaccination programs. Transparent communication about these biases and rigorous improvement in the design of future observational studies are essential.

1 Introduction

The COVID-19 pandemic has triggered numerous vaccine studies, with 1,263 vaccine trials (all phases) registered in the World Health Organization's International Clinical Trials Registry Platform (WHO ICTRP) for more than 250 vaccine candidates (1). These vaccine trials involved more than 80 countries within the first 2 years of the pandemic (99). Due to the urgency of the pandemic, authorities were searching for rapid and reliable data on vaccine efficacy and robust real-world data on vaccine effectiveness (VE) to develop evidence-based vaccination strategies (2).

While randomized, placebo-controlled clinical trials provide robust efficacy and safety data, their value is limited by stringent inclusion criteria, lack of representativeness of diverse populations, and controlled environments that do not reflect local epidemiology or programmatic challenges. Thus, observational studies became crucial due to their rapid implementation, cost-effectiveness, and flexibility in integrating pre-existing data sources (3). These studies can capture the performance of vaccines across diverse populations, address operational issues, and inform sustainable, contextually appropriate vaccination policies using real-world evidence (4).

The landscape of observational studies on the effectiveness of COVID-19 vaccination shows that more than 2200 observational studies were published directly or indirectly referring to VE by the end of 2022 (5). Despite their importance, observational studies are prone to different biases, some being exemplified by COVID-19 pandemic-related factors (6). Recognizing the importance of these errors in VE studies, the WHO prepared interim guidelines for evaluating COVID-19 VE in 2021 (7) and a revision in 2022 (8). These documents have comprehensively outlined studies, errors, and measures to overcome those possible challenges. Despite these guidelines, the biases in observational studies on VE prevailed. An evaluation of those biases and a closer look into how these predicted biases in VE were observed in COVID-19 offers a unique opportunity to design better vaccine effectiveness studies in future pandemics (9). Test-Negative Design (TND), a case-control approach for estimating VE, involves comparing the odds of vaccination among individuals who test positive or negative for the disease. It was widely used during the early pandemic and extensively discussed (10–16). Hence, this paper explores common biases in cohort studies (Table 1), the most common type of studies used for COVID-19 VE assessments (17), to guide future researchers in planning VE studies.

Table 1

Table 1. Biases in cohort studies investigating COVID-19 vaccine effectiveness (VE).

2 Biases due to differential vaccination in populations

In COVID-19 VE studies employing a cohort design, individuals vaccinated through mass immunization programs are enrolled as cohort participants. If the vaccinated population systematically differs from the unvaccinated population, the estimated VE does not reflect the true VE. Biased estimates of VE occur when the participants selected for the study do not represent the general population, leading to systematic differences in the association observed between exposure and outcome among those selected and those eligible. These disparities lead to a bias often called “selection bias.” However, the type of biases in VE studies using cohort design that occurs due to differential vaccinations are not due to conditioning on common effect but resulting from the existence of common causes of exposure and outcome, which could be classified as confounding (18). Despite the classification used (as confounding or bias), a thorough assessment of how these systematic differences occur is crucial to understanding biases in COVID-19 VE studies (Figure 1).

Figure 1

Figure 1. Differential COVID-19 vaccination leading to inaccurate estimates of vaccine effectiveness in cohort studies. The colored labels show the specific names given to the biases that resulted in specific differential vaccinations. Green, overestimation of VE; Red, underestimation of VE.

2.1 Coverage-dependent bias in COVID-19 VE estimates

During the initial phase of vaccination campaigns, when coverage was low, the vaccinated group systematically differed from the unvaccinated group. These differences can cause the VE estimate to be underestimated or overestimated, depending on underlying reasons such as country-specific priorities and selective vaccination practices. This bias is sometimes referred to as “early vaccine bias” (8). Similarly, when vaccine coverage is high, unvaccinated individuals systematically differ from the vaccinated or general populations. Personal beliefs, health promotion practices, contraindications, or access issues can all contribute to inaccurate VE estimates among unvaccinated people in high-coverage settings. The Weekly COVID-19 Vaccination Dashboard of the Center for Disease Control (CDC) clearly showed these differences since 2021, when coverage was low and high (19). Any VE study conducted during these phases may be biased. This bias likely influenced many early COVID-19 VE studies when vaccination coverage was low, and it continues to affect studies as coverage increases beyond a certain level. Many biases described under “selection” biases could be part of this broader bias.

2.2 Biased selection due to socioeconomic status and demographic factors

Despite the massive production of COVID-19 vaccines, data worldwide show socioeconomic and demographic disparities leading to unequal availability, access, and affordability of vaccination (20). These factors separate those vaccinated from those who are not. The observation of this disparity is universal (21–24) and operates at global as well as national and subnational levels (25). Early in COVID-19 immunization programs, US data reported that Black and Hispanic ethnic groups and those who live in rural areas were less likely to be vaccinated (26), and these categories also have lower access and affordability to general healthcare, resulting in worse health outcomes. Data from more than 35 million people in the UK shows that the lower socioeconomic groups were less likely to get vaccinated (19). These socioeconomic disparities and inequalities systematically differentiate the vaccinated group, which typically has a lower infection risk and better disease outcomes. This usually leads to inflation of VE estimates, which may not apply to the general population.

A similar bias may operate in the opposite direction. Certain demographics, especially the young population, reported higher levels of vaccine hesitancy and declining to get vaccinated in some settings (27, 28). Our World in Data shows that in many countries, the lowest proportion of vaccination is among those between 18 and 25 years of age (29). Even though not vaccinated, this group is generally young, healthy, and less susceptible to severe complications and deaths. Combined data from Europe and the US shows that among the adult population, the lowest case fatality rate is among this age group (30). If this occurs in a specific community where VE studies are conducted, the estimates could be lower than expected due to demographic differences in the vaccinated and unvaccinated groups. In contrast, having elderly and more vulnerable populations in vaccinated cohorts might reduce the VE estimate. Thus, this bias could affect the VE estimate in both directions.

2.3 Healthy user bias

Healthy user bias is a broader concept where individuals who engage in one health-promoting behavior (such as vaccination) are more likely to engage in other health-promoting behaviors. This bias can lead to an overestimation of VE. If vaccinated individuals are also more likely to follow other preventive measures such as mask-wearing and social distancing, the observed benefits may be due to these combined behaviors rather than the vaccine alone. Healthy user bias typically affects prospective cohort designs on VE, where the prospective vaccine campaign participants primarily consist of those with healthy behaviors. One of the early COVID-19 VE studies from Hungary showed that the crude mortality rate was 5.3 per 100,000 person days among the unvaccinated group compared to 4.0 among the vaccinated group even during the non-endemic period, showing that the two groups may be systematically different and the vaccinated group is generally healthier (31). A retrospective cohort study using 6,974,817 individuals from December to June 2020 shows that recipients of three COVID-19 vaccines had lower non-COVID-19 mortality risk than their comparator groups. For mRNA vaccines, the adjusted hazard ratios of dose 1 and dose 2 ranged from 0.38 to 0.48 (32). Another early analysis from Milwaukee showed that unvaccinated persons had over twice the risk of non-COVID-19 natural death than the vaccinated (33). Many subsequent studies show reduced hospitalizations and deaths due to other causes in the vaccinated group, confirming the presence of healthy user bias (34, 101). In studies where all-cause mortality is considered as the outcome for VE calculation, this could have a more profound effect, where healthy behaviors may affect the risk of deaths due to other causes as well.

2.4 Healthy vaccinee effect

The “healthy vaccinee effect” is a related but distinct concept from “healthy user bias” in observational studies on VE. In these studies, individuals who choose to get vaccinated are generally healthier than those who do not (35). This can lead to overestimating VE since healthier individuals are naturally less likely to experience severe disease outcomes independent of the VE. Part of this effect could be observed in COVID-19 studies where the overall mortality and hospital admission rates are lower during the first few weeks after vaccination (36). The age-standardized all-cause mortality rate from the UK in 2021 shows that within the first 3 weeks following the first dose of vaccination, the death rate was 795.2 per 100,000 people. This rate increased to 1,232.2 per 100,000 people after the initial 3 weeks. A similar pattern was observed with the second dose, where the death rate was 471 per 100,000 people within the first 3 weeks, rising to 850 per 100,000 people after 3 weeks (37). The lower all-cause death rate within the first 3 weeks of vaccination is likely due to the fact that those vaccinated are not acutely ill at the time of vaccination, suggesting the indirect presence of the Healthy Vaccine Effect. Xu et al. reported that age, sex, and race/ethnicity groups adjusted non-COVID-19 related mortality rates among COVID-19 vaccinees were lower than those among comparators for the first three COVID-19 vaccines licensed in the USA. After the first dose, the adjusted hazard ratios (aHRs) were 0.46 for BNT162b2, 0.41 for the mRNA-1273 vaccine, and 0.55 for Ad26.COV2.S (32). Ostropolets and Hripcsak extensively analyzed this effect using a retrospective cohort design based on electronic health records. It showed that even after adjustment for many health-related variables, the vaccinated group had low overall health-seeking and hospital admission within the first few weeks of vaccination (38).

2.5 Frailty bias

The opposite of the healthy vaccine effect is known as the “frailty bias” or “unhealthy vaccine effect.” The impact of frailty on the outcome of COVID-19 is well-known, and low vaccine effectiveness among frail vaccine recipients is reported in many studies. For example, Meeraus et al. analyzed over 4.5 million AZD1222 vaccine recipients and noted that VE against hospitalization was >90% in the lowest multimorbidity quartile compared to 80% in the highest quartile. Further, among the elderly who are 'fit,' the VE was 86.2%, whereas it was 72% among the frail. VE against hospitalization was lowest in immunosuppressed individuals (65%) (39). However, frailty could be a selection bias as well. This bias occurs when individuals who are more likely to get vaccinated are those with poorer health or more comorbidities, leading to an underestimation of VE. In this scenario, the vaccinated group appears to have worse health outcomes not due to the vaccine's ineffectiveness but because they were already in poorer health than the unvaccinated group.

2.6 Confounding by indication

Confounding by indication occurs when the reason for vaccination is related to the patient's health status or risk of the outcome being studied. Patients with a higher perceived or known risk of severe COVID-19 might be more likely to get vaccinated, such as older adults or those with comorbidities. An analysis of vaccine policies from 185 countries shows that, except in the Western Pacific region, all other regions prioritized clinically vulnerable and elderly populations in their vaccine rollout (39). These subgroups have a higher baseline risk for unfavorable outcomes, which can confound observed VE because their health outcomes might differ from those not vaccinated regardless of the vaccine's true effectiveness. This can lead to overestimating or underestimating VE due to differences in baseline risk between treated and untreated groups. The bias was well-known to researchers yet common in published literature (40).

While both frailty bias and confounding by indication involve differential selection for vaccination, confounding by indication is driven by the perceived risk of COVID-19 outcomes and general health status, while frailty bias is more about the inherent vulnerability of individuals being vaccinated.

2.7 Differential depletion of susceptible bias

Differential depletion of susceptible (DDS) bias occurs when the most susceptible individuals are disproportionately removed from the at-risk population over time through infection or vaccination. Initially, highly susceptible individuals are more likely to contract COVID-19, making VE appear high. Over time, as susceptible individuals gain immunity, both vaccinated and unvaccinated groups show lower infection rates. This results in an apparent waning of VE, not due to the vaccine losing effectiveness but because the overall population has become less susceptible. In COVID-19 VE studies, this bias can underestimate VE over time, falsely suggesting waning VE. This bias is a form of selection bias because the composition of the population changes, affecting the observed effectiveness. DDS could affect all observational study designs used for VE studies. During the COVID-19 pandemic, many studies have shown waning effectiveness (41). For example, a study by Chemaitelly et al. from Egypt estimated the long-term effectiveness of COVID-19 mRNA boosters and reported negative relative effectiveness 6 months after boosting, attributing it to “negative immune imprinting” (42). However, Noam argued that in this study, it is possible that the use of discrete-time hazards conditioned on survival for at least 6 months after vaccination results in selection bias due to the depletion of susceptible from the cohort that did not receive a booster (43). Using a simulation model, Kahn et al. demonstrated that if baseline VE is high, the effect of depletion of susceptible bias is low. However, for 'leaky vaccines' (low baseline VE), the impact of this bias is higher (44). It is essential to consider the differential depletion of susceptible in vaccine effectiveness studies to avoid misleading conclusions.

3 Attrition bias

Attrition bias occurs when there are systematic differences between individuals who remain in a study and those who drop out over time (45). For instance, individuals from lower socioeconomic backgrounds, who often have different health behaviors, poorer health outcomes, and limited access to healthcare, are more likely to leave the study or not visit health facilities for diagnosis. This selective dropout can result in a non-representative sample, as these individuals are frequently at a higher risk of infection due to their circumstances. If they drop out disproportionately, the remaining participants may exhibit a lower overall risk of infection, potentially leading to an overestimation of effect size.

Attrition bias in observational VE studies is unique because these studies often involve large cohorts using secondary data. In many countries, COVID-19 vaccination was offered universally, and vaccination records are uniquely maintained in immunization registries, making the data on vaccination status better than other health data (46). However, outcome data for the cohort studies come from different registries, including hospitals, insurance, or similar databases, which can be linked to vaccination registries (47–49, 103). Access, affordability, and coverage of diagnosis and treatment may differ from vaccination, leading to a 'loss to follow-up' from the original cohort. Unlike prospective cohort studies, where investigators are aware of loss to follow-up, data linkage studies have no definitive way to determine whether there was loss to follow-up. Consequently, lost follow-up is often not reported in cohort studies on VE. For example, in the systematic review of Law et al. none of the cohort studies investigating the VE of inactivated vaccines reported the attrition rate (50). Discussing the impact of attrition on VE is specifically important based on the context and policies, health-seeking behaviors, and vulnerable groups.

4 Information biases

In VE studies using cohort design, information biases can significantly distort the VE estimates. These biases arise from misclassification of exposure or outcome, often due to inconsistent or inaccurate data collection methods. Once the cohorts are identified, it is crucial to obtain accurate data; however, measurement errors can lead to bias in the analysis, known as information bias (51). For example, variations in diagnostic testing practices, inaccuracies in vaccination records, or differential recall between vaccinated and unvaccinated individuals can result in either overestimation or underestimation of VE. The direction and magnitude of this bias depend on whether the distribution of errors for a specific variable, such as vaccination status or disease incidence, is influenced by the true values of these variables or by errors in measuring other variables.

4.1 Immortal time bias

Immortal time bias occurs in observational studies when an “immortal” time period is incorrectly classified or excluded from the analysis (52). This period is immortal because, during this time, the outcome of interest (such as infection or death) cannot occur. In COVID-19 vaccine studies, this bias can arise if the time between the initiation of the study and when individuals receive the vaccine is not correctly accounted for. If this period is mistakenly included in the vaccinated group, it can falsely enhance the vaccine's apparent effectiveness because individuals cannot experience the outcome during this “immortal” period. As an example, Flacco et al. (53) reported that the monthly mean death rates were 0.97 per 1,000 for vaccinated individuals and 2.26 per 1,000 for unvaccinated individuals, showing a significant difference in overall deaths. Their data indicates that the mean follow-up time was 561 days for those never vaccinated, compared to 399 days for those vaccinated, due to the time gap between the start of the study and vaccination. The vaccinated group had a mean of 162 days of immortal time before vaccination (only those still alive received the vaccine), while all deaths during this period were allocated to the unvaccinated group. Since this was a single cohort from a province, the vaccinated group contributed to the denominator of the unvaccinated group before receiving their vaccines. Using the same data, Berrino et al. (54) recalculated the mortality rates and concluded that the rates were almost similar when accounting for the immortal time of the vaccinated group. Immortal time bias is not due to differential vaccination but rather an error in the denominator, which may happen in the analysis process or retrospective cohort studies.

4.2 Misclassification bias

Misclassification bias occurs when individuals or events are incorrectly categorized regarding exposure or outcome status. This bias can arise if vaccination status or disease outcome is inaccurately recorded. While the misclassification of exposure status (vaccination) might seem theoretical in prospective cohort studies, most VE studies use retrospective designs. Due to incomplete or inaccurate historical data and errors in data linkage, there is a risk of misclassifying vaccination status. This is particularly relevant if vaccination records are not up-to-date or come from different healthcare providers with varying record-keeping practices. One of the first systematic reviews on the VE of COVID-19 vaccines showed that of the 42 studies published within the first 6 months, 31 used vaccination registries, five included self-reporting, and 6 did not report the source of vaccination information, highlighting the risk of misclassification bias in these studies (17).

Misclassification of disease or outcomes is expected due to variability in diagnostic test accuracy, differences in test administration timing, and potential misinterpretation of results. The COVID-19 pandemic led to rapid advancement and large-scale production of various diagnostic test kits, none of which possess 100% sensitivity or specificity. This inevitably results in false positives and false negatives, contributing to misclassification. Several systematic reviews on COVID-19 diagnostic tests indicate significant variability in sensitivity and specificity across different platforms, tests, timing of testing relative to exposure, and result interpretation (55–57). A meta-analysis of 24 commercially available antigen kits demonstrated a pooled sensitivity of 77% and a pooled specificity of 98%, highlighting a substantial risk of misclassification (58). Despite WHO's recommendation to use RT-PCR for VE studies (8), the reliance on two-stage screening procedures increases the likelihood of misclassification. These variations can lead to incorrect disease status classification, further complicating VE estimates.

During the pandemic, changes in diagnostic criteria (59–61), referral procedures, breakthrough infections, and even COVID-19 death classifications occurred over time. Since VE studies rely on real-world data, these variations in disease diagnosis and death classifications can differ not only over time but also across different sites, hospitals, or provinces, influenced by policy changes or subjective human factors. Many studies across the globe show that a large number of COVID-19 deaths are unaccounted for and reported as “excess mortality,” and there are time and space variations of these numbers (62). Surveillance and reporting biases further complicate the scenario; inconsistent case reporting practices and variability in public health surveillance intensity can lead to uneven detection and reporting of cases. Understanding the changes and events in study areas, regions, and countries over the study period is essential to comprehending misclassification biases.

4.2.1 Non-differential misclassification of outcomes

Non-differential misclassification occurs when the misclassification rate is similar across groups. This can happen if the outcome measured has a low specificity, leading to false positive cases in both groups. It generally biases the results toward the null, diluting the observed effect size and making it harder to detect a true association between vaccination and outcomes (63). Sometimes, it can happen due to the way the “outcome” is recorded. This has been shown in studies where the allocation of outcomes (COVID-19-related hospital admission) uses data from the billing process. These include patients without symptoms admitted for other reasons but tested positive for COVID-19 (38). Because the outcome documentation is from billing data, those without clinical disease but positive test results were included as those with outcomes (COVID-19 hospital admissions). The misclassification is similar for both groups; thus, the VE estimates are diluted. Similarly, a study from the Netherlands showed that 42% of cases included in the hospital register on COVID-19 patients are missing the reason for admission, and whether the positive COVID-19 test results were associated with COVID-19 clinical disease is not known (103).

4.2.2 Differential misclassification of outcome

Differential misclassification occurs when the likelihood of misclassification differs between groups (vaccinated vs. unvaccinated). Depending on the direction of the misclassification, it can lead to overestimation or underestimation of VE. In studies using secondary data for VE studies, the allocated diagnosis could be systematically different in the two groups. Some studies show that vaccinated people usually attribute mild to moderate symptoms to vaccine side effects and do not seek care, thus less likely to be diagnosed as having COVID-19 (38). On the other hand, healthcare workers may suspect COVID-19 more among unvaccinated groups during hospital visits and perform testing, leading to higher detection or “diagnostic bias.” This could be partly due to policies where test results were mandated for many institutions and procedures if the individual is not vaccinated, thus leading to more cases of asymptomatic or mild cases of COVID-19 among unvaccinated groups. This will lead to differential misclassification of outcome status among vaccinated and unvaccinated groups, inflating VE estimates. However, it will not always overestimate VE. A study in Australia reported that fully vaccinated participants were twice as likely as those who were unvaccinated to report positive COVID testing intentions, which may lead to overdiagnosis of COVID-19 in the vaccinated arm, underestimating VE (64). The context-specific factors could play a major role in deciding in which direction the bias operates.

4.3 Case counting window bias

The case counting window bias occurs when the time frame for counting cases in a study is not properly aligned with the period during which a vaccine is expected to be effective. This can happen if the cases are counted from the start of follow-up for the unvaccinated group but only after vaccination for the vaccinated group; the difference in these windows can lead to biased VE estimates. So, the cases in the unvaccinated group are counted during a period when they were at higher risk. In contrast, cases in the vaccinated group are only counted after they might have already benefited from some protection. Fung et al. showed that even a vaccine with almost zero VE could be presented as having a VE of 48% if the case counting window bias is ignored (65).

4.4 Waning immunity bias

Waning immunity bias occurs when the observed effectiveness of a vaccine appears to decline over time due to a natural decrease in immune response, notably the reduction in neutralizing antibody levels. Initially, vaccines demonstrate high effectiveness due to a robust immune response. However, as antibody levels wane, individuals may become more susceptible to infection, creating the perception of declining vaccine effectiveness (VE). Numerous studies, including several systematic reviews (45, 66, 67), have reported this phenomenon, particularly those involving mRNA vaccines against COVID-19. A systematic review of 40 studies estimated that VE against omicron infection and symptomatic disease decreased by 20% at 6 months post-primary vaccination cycle and by 30% at 9 months post-booster dose administration (68). This waning immunity is a natural biological process; failing to account for it in VE studies properly can introduce bias, leading to inaccurate interpretations. Long-term VE studies must incorporate original or short-term VE data and clearly state the duration post-vaccination to avoid biased interpretation of VE estimates. Additionally, considering the date of vaccination in analysis and data presentation is essential. Addressing this bias is crucial, particularly in the ongoing “infodemic” (69) and vaccine hesitancy.

5 Confounding

Confounding occurs when an extraneous factor influences both the likelihood of vaccination and the health outcome, leading to a distorted estimate of vaccine effectiveness (18). This manuscript has described factors such as health-seeking behaviors, demographic differences, and underlying health conditions leading to differential vaccination. While these factors were labeled as biases—such as selection bias, healthy user bias, and frailty bias—these are, in fact, confounding factors. They influence both vaccination status and outcomes, distorting the true effect of the vaccine. For instance, individuals with prior exposure to SARS-CoV-2 or underlying comorbidities may have different risks of severe COVID-19, which could distort VE estimates if not properly adjusted for (70, 71, 100). In addition to these specific factors, researchers must consider other confounders that may further affect VE estimates, such as geographic differences, occupational exposures, and previous infection history. Failure to control for such confounders can lead to inflated or underestimated VE estimates, particularly when assessing protection against severe disease. To ensure accurate and reliable VE estimates, thorough identification and adjustment of these confounding factors are essential in observational studies.

6 Biases and challenges of VE studies in LMICs

The assessment of VE in low and lower-middle-income countries (LMICs) faces significant challenges due to various biases and unique factors. Disparities in vaccine deployment, reliance on less effective vaccines, supply constraints, and dosing challenges necessitate evaluations of mixed and suboptimal regimens. Additionally, distinct health and demographic profiles can lead to differential vaccination coverage, altering estimated vaccine protection (72). High seroprevalence of naturally acquired immunity and the risk of new variant emergence due to uncontrolled transmission further complicate VE estimates. The absence of robust systems for maintaining vaccination records, integrating data across platforms, and ensuring accurate digitalization, along with limited diagnostic capacity and barriers to accessing and affording healthcare, exacerbate the risk of misclassification of both exposure (vaccination status) and outcome (disease occurrence). Such misclassification is a critical source of information bias in these studies, significantly impacting the accuracy of VE estimates. Therefore, targeted VE studies are essential to develop accurate and effective vaccination strategies tailored to LMICs. However, the availability of vaccine studies in these settings is limited. A systematic review by Petráš et al. included 761 published VE studies, but only two were from low-income countries, both from Zambia (73). One was from a prison outbreak using a case-control design with self-reporting/rapid tests (74), and the other was a hospital-based study with significant missing data (75). Among lower-middle-income countries, Indian authors published 28 studies. Except for India, only Bangladesh (76), Egypt (77), Morocco (78), and Pakistan (79–83) had estimated VE in their settings. Real-world evidence and the effect of biases on those estimates from LMICs are missing in global literature. A comprehensive analysis of these available studies is critical to understanding how specific biases affect VE estimates in future academic preparations.

7 Effects of biased estimates

Biases significantly impact the internal validity of vaccine efficacy (VE) studies, with far-reaching societal and policy implications. Early studies often reported VE as high as 95% (84), fostering overconfidence in vaccine protection and prompting global policy changes. Health agencies described vaccines as “extremely protective,” creating a misconception that vaccines could prevent infection (85). This led to policy changes in some places prioritizing vaccination for transmission interruption, sometimes neglecting vulnerable populations at higher risk for severe COVID-19 outcomes. Overestimating VE altered public perception, with increased infection rates post-vaccination due to risk behaviors (86). Variations in VE estimates influence vaccination willingness (87); one study showed that 51.3% would accept a COVID-19 vaccine that is 50% effective, and 77.1% would accept a vaccine that is 95% effective (84). Differences in VE estimates can significantly fuel vaccine hesitancy and erode public trust in vaccination programs. Media coverage and anti-vaccine movements can exploit these inconsistencies, spreading misinformation and increasing hesitancy. Transparent communication about the limitations and strengths of VE studies is crucial to maintaining public trust and encouraging vaccine acceptance.

8 Minimizing biases in cohort studies on VE

Target trial emulation (TTE) is often used to minimize biases in observational studies on VE by applying design principles from randomized trials to the analysis of observational data, thereby explicitly tying the analysis to the trial it is emulating. The TTE approach was extensively used in COVID-19 VE studies when applying cohort design (14, 88–94, 102). Still, the challenges and errors in properly executing this approach can introduce the abovementioned biases, resulting in inaccuracies in estimating VE if the emulation is improperly planned and executed. Although referred to as “TTE bias” in literature, it is not technically a specific type of bias but could be due to any other type detailed above.

Biases, once introduced, are challenging to control in observational studies. Minimizing bias during the design stage is paramount. The WHO guidelines on observational studies for COVID-19 vaccine effectiveness (8) outlined various strategies to mitigate these biases. Bayesian modeling approaches could address specific biases, such as misclassification bias due to imperfect tests (95). A thorough evaluation of all potential biases is essential when reporting VE studies but reporting and assessing the direction of biases may be more challenging than it seems. Brookmeyer and Morrison (96) demonstrated this complexity using data published in a VE study (97) to simulate different biases occur in linked registry studies. They showed that if the bias is due to a single source, the direction of the bias is predictable. However, if multiple sources of biases are present, then the direction of the bias can be either way. Often, the biases are multiple; thus, predicting direction can be difficult. It is crucial to discuss these biases' probable impact and direction on VE study results and report them comprehensively in all observational studies.

9 Conclusion

These well-known epidemiological biases may occur more frequently during a pandemic. Rapid vaccine development and distribution can lead to differential vaccination practices, prioritizing high-risk populations and exacerbating frailty bias and confounding by indication. Socioeconomic disparities and diverse health behaviors become more pronounced, while the overburdened health systems increase the risk of data inaccuracies and outcome misclassification. The frequency of occurrence of these biases varies widely based on the setting, data sources, and analysis. These biases could be higher in VE in LMIC settings due to a lack of proper data sources. Understanding and mitigating these biases are crucial for accurate VE estimates, informing public health strategies, and maintaining public trust in vaccination programs. With the peak pandemic now behind us and less urgency for rapid publication, a comprehensive investigation into the long-term vaccine effectiveness is warranted, ensuring that all previously discussed biases are thoroughly addressed. Additionally, future studies must explicitly account for whether individuals who were infected with COVID-19 subsequently received vaccination, as this may affect long-term outcomes such as mortality and PASC. While studies like Cai et al. (98) demonstrate that the risk of death declines over time but remains elevated in previously hospitalized patients, they do not account for whether subsequent vaccination modifies this risk. Incorporating this factor in future research will provide more accurate assessments of vaccine effectiveness and long-term health risks, thereby guiding public health policies more effectively.

Author contributions

SA: Conceptualization, Data curation, Validation, Visualization, Writing – original draft, Writing – review & editing. BT: Conceptualization, Writing – review & editing. SS: Supervision, Writing – review & editing. J-LE: Conceptualization, Supervision, Writing – review & editing. JK: Conceptualization, Supervision, Writing – review & editing.

Funding

The author(s) declare that no financial support was received for the research, authorship, and/or publication of this article.

Acknowledgments

We would like to thank John D. Clemens for his support during the conceptualization of this manuscript. Chat GTP 40 was used for language improvement.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

The author(s) declared that they were an editorial board member of Frontiers, at the time of submission. This had no impact on the peer review process and the final decision.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

1. World Health Organization. International Clinical Trials Registry Platform. (2024). Available at: https://trialsearch.who.int/Default.aspx (accessed March 8,2024).

Google Scholar

2. Excler J-L, Saville M, Berkley S, Kim JH. Vaccine development for emerging infectious diseases. Nat Med. (2021) 27:591–600. doi: 10.1038/s41591-021-01301-0

PubMed Abstract | Crossref Full Text | Google Scholar

3. Sheldrick RC. Randomized trials vs. real-world evidence: how can both inform decision-making? J Am Med Assoc. (2023) 329:1352–3. doi: 10.1001/jama.2023.4855

PubMed Abstract | Crossref Full Text | Google Scholar

4. Bollaerts K, Wyndham-Thomas C, Miller E, Izurieta HS, Black S, Andrews N, et al. The role of real-world evidence for regulatory and public health decision-making for Accelerated Vaccine Deployment- a meeting report. Biologicals. (2024) 85:101750. doi: 10.1016/j.biologicals.2024.101750

PubMed Abstract | Crossref Full Text | Google Scholar

5. World Health Organization. Landscape of Observational Studies on the Effectiveness of COVID-19 Vaccination. (2023). Available at: https://www.who.int/publications/m/item/draft-landscape-of-observational-study-designs-on-the-effectiveness-of-covid-19-vaccination (accessed March 8,2024).

Google Scholar

6. Lipsitch M, Jha A, Simonsen L. Observational studies and the difficult quest for causality: lessons from vaccine effectiveness and impact studies. Int J Epidemiol. (2016) 45:2060–74. doi: 10.1093/ije/dyw124

PubMed Abstract | Crossref Full Text | Google Scholar

7. Patel MK, Bergeri I, Bresee JS, Cowling BJ, Crowcroft NS, Fahmy K, et al. Evaluation of post-introduction COVID-19 vaccine effectiveness: summary of interim guidance of the World Health Organization. Vaccine. (2021) 39:4013–24. doi: 10.1016/j.vaccine.2021.05.099

PubMed Abstract | Crossref Full Text | Google Scholar

8. World Health Organization. Evaluation of COVID-19 Vaccine Effectiveness in a Changing Landscape of COVID-19 Epidemiology and Vaccination: Interim Guidance, 1 October 2022: Second Addendum to Evaluation of COVID-19 Vaccine Effectiveness: Interim Guidance. Geneva: World Health Organization (2022).

Google Scholar

9. Ricotta EE, Rid A, Cohen IG, Evans NG. Observational studies must be reformed before the next pandemic. Nat Med. (2023) 29:1903–5. doi: 10.1038/s41591-023-02375-8

PubMed Abstract | Crossref Full Text | Google Scholar

10. Hitchings MDT, Lewnard JA, Dean NE, Ko AI, Ranzani OT, Andrews JR, et al. Use of recently vaccinated individuals to detect bias in test-negative case-control studies of COVID-19 vaccine effectiveness. Epidemiology. (2022) 33:450–6. doi: 10.1097/EDE.0000000000001484

PubMed Abstract | Crossref Full Text | Google Scholar

11. Schnitzer ME. Estimands and estimation of COVID-19 vaccine effectiveness under the test-negative design: connections to causal inference. Epidemiology. (2022) 33:325–33. doi: 10.1097/EDE.0000000000001470

PubMed Abstract | Crossref Full Text | Google Scholar

12. Graham S, Tessier E, Stowe J, Bernal JL, Parker EPK, Nitsch D, et al. Bias assessment of a test-negative design study of COVID-19 vaccine effectiveness used in national policymaking. Nat Commun. (2023) 14:3984. doi: 10.1038/s41467-023-39674-0

PubMed Abstract | Crossref Full Text | Google Scholar

13. Shi X, Li KQ, Mukherjee B. Current challenges with the use of test-negative designs for modeling COVID-19 vaccination and outcomes. Am J Epidemiol. (2023) 192:328–33. doi: 10.1093/aje/kwac203

PubMed Abstract | Crossref Full Text | Google Scholar

14. Li G, Gerlovin H, Figueroa Muñiz MJ, Wise JK, Madenci AL, Robins JM, et al. Comparison of the test-negative design and cohort design with explicit target trial emulation for evaluating COVID-19 vaccine effectiveness. Epidemiology. (2024) 35:137–49. doi: 10.1097/EDE.0000000000001709

PubMed Abstract | Crossref Full Text | Google Scholar

15. Mésidor M, Liu Y, Talbot D, Skowronski DM, De Serres G, Merckx J, et al. Test negative design for vaccine effectiveness estimation in the context of the COVID-19 pandemic: a systematic methodology review. Vaccine. (2024) 42:995–1003. doi: 10.1016/j.vaccine.2023.12.013

PubMed Abstract | Crossref Full Text | Google Scholar

16. Ortiz-Brizuela E, Carabali M, Jiang C, Merckx J, Talbot D, Schnitzer ME. Potential biases in test-negative design studies of COVID-19 vaccine effectiveness arising from the inclusion of asymptomatic individuals. Am J Epidemiol. (2024) 2024:kwae288. doi: 10.1093/aje/kwae288

PubMed Abstract | Crossref Full Text | Google Scholar

17. Teerawattananon Y, Anothaisintawee T, Pheerapanyawaranun C, Botwright S, Akksilp K, Sirichumroonwit N, et al. A systematic review of methodological approaches for evaluating real-world effectiveness of COVID-19 vaccines: advising resource-constrained settings. PLoS ONE. (2022) 17:e0261930. doi: 10.1371/journal.pone.0261930

PubMed Abstract | Crossref Full Text | Google Scholar

18. Hernán MA, Hernández-Díaz S, Robins JM. A structural approach to selection bias. Epidemiology. (2004) 15:615–25. doi: 10.1097/01.ede.0000135174.63482.43

PubMed Abstract | Crossref Full Text | Google Scholar

19. Center for Disease Control. Weekly COVID-19 Vaccination Dashboard (2021). Available at: https://www.cdc.gov/covidvaxview/weekly-dashboard/index.html

Google Scholar

20. Privor-Dumm L, Excler J-L, Gilbert S, Abdool Karim SS, Hotez PJ, Thompson D, et al. Vaccine access, equity and justice: COVID-19 vaccines and vaccination. Br Med J Glob Health. (2023) 8:e011881. doi: 10.1136/bmjgh-2023-011881

PubMed Abstract | Crossref Full Text | Google Scholar

21. Benoni R, Sartorello A, Moretti F, Marchiori F, Accordini L, Postiglione C, et al. Disparities in access to COVID-19 vaccine in Verona, Italy: a cohort study using local health immunization data. Front Public Health. (2023) 11:1167414. doi: 10.3389/fpubh.2023.1167414

PubMed Abstract | Crossref Full Text | Google Scholar

22. Cavillot L, Van Loenhout JaF, Devleesschauwer B, Wyndham-Thomas C, Van Oyen H, et al. Sociodemographic and socioeconomic disparities in COVID-19 vaccine uptake in Belgium: a nationwide record linkage study. J Epidemiol Community Health. (2023) 78:176–83. doi: 10.1136/jech-2023-220751

PubMed Abstract | Crossref Full Text | Google Scholar

23. Sabatini S, Kaufmann M, Fadda M, Tancredi S, Noor N, Van Der Linden BWA, et al. Factors associated with COVID-19 non-vaccination in Switzerland: a nationwide study. Int J Public Health. (2023) 68:1605852. doi: 10.3389/ijph.2023.1605852

PubMed Abstract | Crossref Full Text | Google Scholar

24. Welsh J, Biddle N, Butler DC, Korda RJ. Discretion in decision to receive COVID-19 vaccines and associated socio-economic inequalities in rates of uptake: a whole-of-population data linkage study from Australia. Publ Health. (2023) 224:82–9. doi: 10.1016/j.puhe.2023.08.020

PubMed Abstract | Crossref Full Text | Google Scholar

25. Bayati M, Noroozi R, Ghanbari-Jahromi M, Jalali FS. Inequality in the distribution of COVID-19 vaccine: a systematic review. Int J Equity Health. (2022) 21:122. doi: 10.1186/s12939-022-01729-x

PubMed Abstract | Crossref Full Text | Google Scholar

26. Murthy BP. COVID-19 vaccination coverage and demographic characteristics of infants and children aged 6 months-4 years-United States, June 20-December 31, 2022. Morbid Mortal Week Rep. (2023) 72:7a4. doi: 10.15585/mmwr.mm7207a4

PubMed Abstract | Crossref Full Text | Google Scholar

27. Fazel M, Puntis S, White SR, Townsend A, Mansfield KL, Viner R, et al. Willingness of children and adolescents to have a COVID-19 vaccination: results of a large whole schools survey in England. eClinicalMedicine. (2021) 40:101144. doi: 10.1016/j.eclinm.2021.101144

PubMed Abstract | Crossref Full Text | Google Scholar

28. Bergen N, Johns NE, Chang Blanc D, Hosseinpoor AR. Within-country inequality in COVID-19 vaccination coverage: a scoping review of academic literature. Vaccines. (2023) 11:517. doi: 10.3390/vaccines11030517

PubMed Abstract | Crossref Full Text | Google Scholar

29. Havers FP, Pham H, Taylor CA, Whitaker M, Patel K, Anglin O, et al. COVID-19-associated hospitalizations among vaccinated and unvaccinated adults 18 years or older in 13 US States, January 2021 to April 2022. J Am Med Assoc Intern Med. (2022) 182:1071–81. doi: 10.1001/jamainternmed.2022.4299

PubMed Abstract | Crossref Full Text | Google Scholar

30. Goldstein JR, Lee RD. Demographic perspectives on the mortality of COVID-19 and other epidemics. Proc Nat Acad Sci USA. (2020) 117:22035–41. doi: 10.1073/pnas.2006392117

PubMed Abstract | Crossref Full Text | Google Scholar

31. Pálinkás A, Sándor J. Effectiveness of COVID-19 vaccination in preventing all-cause mortality among adults during the third wave of the epidemic in Hungary: nationwide retrospective cohort study. Vaccines. (2022) 10:1009. doi: 10.3390/vaccines10071009

PubMed Abstract | Crossref Full Text | Google Scholar

32. Xu S, Huang R, Sy LS, Hong V, Glenn SC, Ryan DS, et al. A safety study evaluating non-COVID-19 mortality risk following COVID-19 vaccination. Vaccine. (2023) 41:844–54. doi: 10.1016/j.vaccine.2022.12.036

PubMed Abstract | Crossref Full Text | Google Scholar

33. Atanasov VA, Barreto Parra PN, Franchi L. Selection bias and COVID-19 vaccine effectiveness against death: evidence from linked mortality and vaccination records. SSRN Electron J. (2022) 2022:460. doi: 10.2139/ssrn.4250460

Crossref Full Text | Google Scholar

34. Arbel R, Peretz A, Sergienko R, Friger M, Beckenstein T, Duskin-Bitan H, et al. Effectiveness of a bivalent mRNA vaccine booster dose to prevent severe COVID-19 outcomes: a retrospective cohort study. Lancet Infect Dis. (2023) 23:914–21. doi: 10.1016/S1473-3099(23)00122-6

PubMed Abstract | Crossref Full Text | Google Scholar

35. Furst T, Straka R, Janosek J. Healthy vaccinee effect: a bias not to be forgotten in observational studies on COVID-19 vaccine effectiveness. Pol Arch Intern Med. (2024) 134:16634. doi: 10.20452/pamw.16634

PubMed Abstract | Crossref Full Text | Google Scholar

36. Prunas O, Weinberger DM, Pitzer VE, Gazit S, Patalon T. Waning effectiveness of the BNT162b2 vaccine against infection in adolescents in Israel. Clin Infect Dis. (2022) 76:113–8. doi: 10.1093/cid/ciac315

PubMed Abstract | Crossref Full Text | Google Scholar

37. Office for National Statistics. Deaths Involving COVID-19 By Vaccination Status, England: Deaths Occurring Between 1 January and 31 October 2021. O'ffice for National Statistics (2021). Available at: https://www.ons.gov.uk/peoplepopulationandcommunity/birthsdeathsandmarriages/deaths/bulletins/deathsinvolvingcovid19byvaccinationstatusengland/deathsoccurringbetween1januaryand31october2021 (accessed March 8,2024).

Google Scholar

38. Ostropolets A, Hripcsak G. COVID-19 vaccination effectiveness rates by week and sources of bias: a retrospective cohort study. Br Med J Open. (2022) 12:e061126. doi: 10.1136/bmjopen-2022-061126

PubMed Abstract | Crossref Full Text | Google Scholar

39. Meeraus W, Joy M, Ouwens M, Taylor KS, Venkatesan S, Dennis J, et al. AZD1222 effectiveness against severe COVID-19 in individuals with comorbidity or frailty: the RAVEN cohort study. J Infect. (2024) 88:106129. doi: 10.1016/j.jinf.2024.106129

PubMed Abstract | Crossref Full Text | Google Scholar

40. Nguyen VG, Yavlinsky A, Beale S, Hoskins S, Byrne TE, Lampos V, et al. Comparative effectiveness of different primary vaccination courses on mRNA-based booster vaccines against SARS-CoV-2 infections: a time-varying cohort analysis using trial emulation in the Virus Watch community cohort. Int J Epidemiol. (2023) 52:342–54. doi: 10.1093/ije/dyad002

PubMed Abstract | Crossref Full Text | Google Scholar

41. Bobrovitz N, Ware H, Ma X, Li Z, Hosseini R, Cao C, et al. Protective effectiveness of previous SARS-CoV-2 infection and hybrid immunity against the omicron variant and severe disease: a systematic review and meta-regression. Lancet Infect Dis. (2023) 23:556–67. doi: 10.1016/S1473-3099(22)00801-5

PubMed Abstract | Crossref Full Text | Google Scholar

42. Chemaitelly H, Ayoub HH, Tang P, Coyle P, Yassine HM, Al Thani AA, et al. Long-term COVID-19 booster effectiveness by infection history and clinical vulnerability and immune imprinting: a retrospective population-based cohort study. Lancet Infect Dis. (2023) 23:816–27. doi: 10.1016/S1473-3099(23)00058-0

PubMed Abstract | Crossref Full Text | Google Scholar

43. Barda N. The ups and downs of observational vaccine research. Lancet Infect Dis. (2023) 23:767–8. doi: 10.1016/S1473-3099(23)00119-6

PubMed Abstract | Crossref Full Text | Google Scholar

44. Kahn R, Schrag SJ, Verani JR, Lipsitch M. Identifying and alleviating bias due to differential depletion of susceptible people in postmarketing evaluations of COVID-19 vaccines. Am J Epidemiol. (2022) 191:800–11. doi: 10.1093/aje/kwac015

PubMed Abstract | Crossref Full Text | Google Scholar

45. Booker CL, Harding S, Benzeval M. A systematic review of the effect of retention methods in population-based cohort studies. BMC Publ Health. (2011) 11:249. doi: 10.1186/1471-2458-11-249

PubMed Abstract | Crossref Full Text | Google Scholar

46. Cameron-Blake E, Tatlow H, Andretti B, Boby T, Green K, Hale T, et al. A panel dataset of COVID-19 vaccination policies in 185 countries. Nat Hum Behav. (2023) 7:1402–13. doi: 10.1038/s41562-023-01615-8

PubMed Abstract | Crossref Full Text | Google Scholar

47. Meijerink H, Veneti L, Kristoffersen AB, Danielsen AS, Stecher M, Starrfelt J. Estimating vaccine effectiveness against COVID-19 using cause-specific sick leave as an indicator: a nationwide population-based cohort study, Norway, July 2021—December 2022. BMC Publ Health. (2024) 24:1861. doi: 10.1186/s12889-024-19374-0

PubMed Abstract | Crossref Full Text | Google Scholar

48. Uemura K, Ono S, Michihata N, Yamana H, Yasunaga H. Duration of effectiveness of the COVID-19 vaccine in Japan: a retrospective cohort study using large-scale population-based registry data. BMC Infect Dis. (2024) 24:648. doi: 10.1186/s12879-024-09488-6

PubMed Abstract | Crossref Full Text | Google Scholar

49. Urquidi C, Sepúlveda-Peñaloza A, Valenzuela MT, Ponce A, Menares V, Cortes CP, et al. Vaccine effectiveness in reducing COVID-19-related hospitalization after a risk-age-based mass vaccination program in a Chilean municipality: a comparison of observational study designs. Vaccine. (2024) 42:3851–6. doi: 10.1016/j.vaccine.2024.05.002

PubMed Abstract | Crossref Full Text | Google Scholar

50. Law M, Ho SSH, Tsang GKC, Ho CMY, Kwan CM, Yan VKC, et al. Efficacy and effectiveness of inactivated vaccines against symptomatic COVID-19, severe COVID-19, and COVID-19 clinical outcomes in the general population: a systematic review and meta-analysis. Lancet Region Health Western Pacific. (2023) 37:100788. doi: 10.1016/j.lanwpc.2023.100788

PubMed Abstract | Crossref Full Text | Google Scholar

51. Rothman KJ, Greenland S, Lash TL. Modern Epidemiology. Philadelphia, PA: Wolters Kluwer Health/Lippincott Williams & Wilkins Philadelphia (2008).

Google Scholar

52. Alessandria M, Malatesta GM, Berrino F, Donzelli A. A critical analysis of all-cause deaths during COVID-19 vaccination in an Italian Province. Microorganisms. (2024) 12:1343. doi: 10.3390/microorganisms12071343

PubMed Abstract | Crossref Full Text | Google Scholar

53. Flacco ME, Acuti Martellucci C, Soldato G, Di Martino G, Carota R, De Benedictis M, et al. COVID-19 vaccination did not increase the risk of potentially related serious adverse events: 18-month cohort study in an Italian Province. Vaccines. (2022) 11:10031. doi: 10.3390/vaccines11010031

PubMed Abstract | Crossref Full Text | Google Scholar

54. Berrino F, Donzelli A, Bellavite P, Malatesta G. COVID-19 vaccination and all-cause and non-COVID-19 mortality. A revaluation of a study carried out in an Italian Province. Epidemiol Prev. (2023) 47:374–8. doi: 10.19191/EP23.6.A643.075

PubMed Abstract | Crossref Full Text | Google Scholar

55. Borges LP, Martins AF, Silva BM, Dias BP, Gonçalves RL, Souza DRV, et al. Rapid diagnosis of COVID-19 in the first year of the pandemic: a systematic review. Int Immunopharmacol. (2021) 101:108144. doi: 10.1016/j.intimp.2021.108144

PubMed Abstract | Crossref Full Text | Google Scholar

56. Dinnes J, Sharma P, Berhane S, Van Wyk SS, Nyaaba N, Domen J, et al. Rapid, point-of-care antigen tests for diagnosis of SARS-CoV-2 infection. Cochr Database Syst Rev. (2022) 7:CD013705. doi: 10.1002/14651858.CD013705.pub3

PubMed Abstract | Crossref Full Text | Google Scholar

57. Fox T, Geppert J, Dinnes J, Scandrett K, Bigio J, Sulis G, et al. Antibody tests for identification of current and past infection with SARS-CoV-2. Cochr Database Syst Rev. (2022) 11:CD013652. doi: 10.1002/14651858.CD013652.pub2

PubMed Abstract | Crossref Full Text | Google Scholar

58. Manten K, Katzenschlager S, Brümmer LE, Schmitz S, Gaeddert M, Erdmann C, et al. Clinical accuracy of instrument-based SARS-CoV-2 antigen diagnostic tests: a systematic review and meta-analysis. Virol J. (2024) 21:99. doi: 10.1186/s12985-024-02371-5

PubMed Abstract | Crossref Full Text | Google Scholar

59. Wang YY, Jin YH, Ren XQ, Li YR, Zhang XC, Zeng XT, et al. Updating the diagnostic criteria of COVID-19 “suspected case” and “confirmed case” is necessary. Mil Med Res. (2020) 7:17. doi: 10.1186/s40779-020-00245-9

PubMed Abstract | Crossref Full Text | Google Scholar

60. Hong KH, Kim GJ, Roh KH, Sung H, Lee J, Kim SY, et al. Update of guidelines for laboratory diagnosis of COVID-19 in Korea. Ann Lab Med. (2022) 42:391–7. doi: 10.3343/alm.2022.42.4.391

PubMed Abstract | Crossref Full Text | Google Scholar

61. Wu Y, Feng X, Gong M, Han J, Jiao Y, Li S, et al. Evolution and major changes of the diagnosis and treatment protocol for COVID-19 patients in China 2020-2023. Health Care Sci. (2023) 2:135–52. doi: 10.1002/hcs2.45

PubMed Abstract | Crossref Full Text | Google Scholar

62. Giattino C, Ritchie H, Roser M, Ortiz-Ospina E, Hasell J. Excess Mortality During the Coronavirus Pandemic (COVID-19). Our World in Data (2021). Available at: https://ourworldindata.org/excess-mortality-covid

Google Scholar

63. Hansen CH. Bias in vaccine effectiveness studies of clinically severe outcomes that are measured with low specificity: the example of COVID-19-related hospitalisation. Euro Surveill. (2024) 29:259. doi: 10.2807/1560-7917.ES.2024.29.7.2300259

PubMed Abstract | Crossref Full Text | Google Scholar

64. Glasziou P, Mccaffery K, Cvejic E, Batcup C, Ayre J, Pickles K, et al. Testing behaviour may bias observational studies of vaccine effectiveness. J Assoc Med Microbiol Infect Dis Can. (2022) 7:242–6. doi: 10.3138/jammi-2022-0002

PubMed Abstract | Crossref Full Text | Google Scholar

65. Fung K, Jones M, Doshi P. Sources of bias in observational studies of COVID-19 vaccine effectiveness. J Eval Clin Pract. (2024) 30:30–6. doi: 10.1111/jep.13839

PubMed Abstract | Crossref Full Text | Google Scholar

66. Feikin DR, Higdon MM, Abu-Raddad LJ, Andrews N, Araos R, Goldberg Y, et al. Duration of effectiveness of vaccines against SARS-CoV-2 infection and COVID-19 disease: results of a systematic review and meta-regression. Lancet. (2022) 399:924–44. doi: 10.1016/S0140-6736(22)00152-0

PubMed Abstract | Crossref Full Text | Google Scholar

67. Higdon MM, Baidya A, Walter KK, Patel MK, Issa H, Espié E, et al. Duration of effectiveness of vaccination against COVID-19 caused by the omicron variant. Lancet Infect Dis. (2022) 22:1114–6. doi: 10.1016/S1473-3099(22)00409-1

PubMed Abstract | Crossref Full Text | Google Scholar

68. Menegale F, Manica M, Zardini A, Guzzetta G, Marziano V, D'andrea V, et al. Evaluation of waning of SARS-CoV-2 vaccine-induced immunity: a systematic review and meta-analysis. J Am Med Assoc Netw Open. (2023) 6:e2310650. doi: 10.1001/jamanetworkopen.2023.10650

PubMed Abstract | Crossref Full Text | Google Scholar

69. Gallotti R, Valle F, Castaldo N, Sacco P, De Domenico M. Assessing the risks of “infodemics” in response to COVID-19 epidemics. Nat Hum Behav. (2020) 4:1285–93. doi: 10.1038/s41562-020-00994-6

PubMed Abstract | Crossref Full Text | Google Scholar

70. Rennert L, Ma Z, Mcmahan CS, Dean D. Effectiveness and protection duration of COVID-19 vaccines and previous infection against any SARS-CoV-2 infection in young adults. Nat Commun. (2022) 13:3946. doi: 10.1038/s41467-022-31469-z

PubMed Abstract | Crossref Full Text | Google Scholar

71. Altarawneh HN, Chemaitelly H, Ayoub HH, Tang P, Hasan MR, Yassine HM, et al. Effects of previous infection, vaccination, and hybrid immunity against symptomatic Alpha, Beta, and Delta SARS-CoV-2 infections: an observational study. eBioMedicine. (2023) 95:104734. doi: 10.1016/j.ebiom.2023.104734

PubMed Abstract | Crossref Full Text | Google Scholar

72. Clemens J, Aziz AB, Tadesse BT, Kang S, Marks F, Kim J. Evaluation of protection by COVID-19 vaccines after deployment in low and lower-middle income countries. eClinicalMedicine. (2022) 43:101253. doi: 10.1016/j.eclinm.2021.101253

PubMed Abstract | Crossref Full Text | Google Scholar

73. Petráš M, Janovská D, Lomozová D, Franklová M, Dlouhý P, Rosina J, et al. Understanding the time-driven shifts of vaccine effectiveness against any and severe COVID-19 before and after the surge of Omicron variants within 2.5 years of vaccination: a meta-regression. Int J Infect Dis. (2024) 142:106986. doi: 10.1016/j.ijid.2024.106986

PubMed Abstract | Crossref Full Text | Google Scholar

74. Simwanza J, Hines JZ, Sinyange D, Sinyange N, Mulenga C, Hanyinza S, et al. COVID-19 vaccine effectiveness during a prison outbreak when omicron was the dominant circulating variant-Zambia, December 2021. Am J Trop Med Hyg. (2022) 107:1055–9. doi: 10.4269/ajtmh.22-0368

PubMed Abstract | Crossref Full Text | Google Scholar

75. Chanda D, Hines JZ, Itoh M, Fwoloshi S, Minchella PA, Zyambo KD, et al. COVID-19 vaccine effectiveness against progression to in-hospital mortality in Zambia, 2021–2022. Open For Infect Dis. (2022) 9:ofac469. doi: 10.1093/ofid/ofac469

PubMed Abstract | Crossref Full Text | Google Scholar

76. Khanam F, Islam MT, Ahmmed F, Ahmed SU, Hossen MI, Rajib MH, et al. Measuring the effectiveness of COVID-19 vaccines used during a surge of the delta variant of SARS-CoV-2 in Bangladesh: a test-negative design evaluation. Vaccines. (2022) 10:122069. doi: 10.3390/vaccines10122069

PubMed Abstract | Crossref Full Text | Google Scholar

77. Ashmawy R, Kamal E, Amin W, Sharaf S, Kabeel S, Albiheyri R, et al. Effectiveness and safety of inactivated SARS-CoV-2 vaccine (BBIBP-CorV) among healthcare workers: a seven-month follow-up study at fifteen central hospitals. Vaccines. (2023) 11:892. doi: 10.3390/vaccines11050892

PubMed Abstract | Crossref Full Text | Google Scholar

78. Belayachi J, Obtel M, Mhayi A, Razine R, Abouqal R. Long term effectiveness of inactivated vaccine BBIBP-CorV (Vero Cells) against COVID-19 associated severe and critical hospitalization in Morocco. PLoS ONE. (2022) 17:e0278546. doi: 10.1371/journal.pone.0278546

PubMed Abstract | Crossref Full Text | Google Scholar

79. Ul Munamm SA, Nadeem I, Mahdi N, Saqlain M, Rana ZK, Khatana UF, et al. Comparative analysis of mRNA and inactivated COVID-19 vaccines: a study from Faisalabad district of Pakistan. J R Coll Physicians Edinb. (2022) 52:240–6. doi: 10.1177/14782715221131409

PubMed Abstract | Crossref Full Text | Google Scholar

80. Khan UI, Niaz M, Azam I, Hasan Z, Hassan I, Mahmood SF, et al. Effectiveness of inactivated COVID-19 vaccines against SARS-CoV-2 infections among healthcare personnel in Pakistan: a test-negative case-control study. Br Med J Open. (2023) 13:e071789. doi: 10.1136/bmjopen-2023-071789

PubMed Abstract | Crossref Full Text | Google Scholar

81. Mushtaq MZ, Nasir N, Mahmood SF, Khan S, Kanji A, Nasir A, et al. Exploring the relationship between SARS-CoV-2 variants, illness severity at presentation, in-hospital mortality and COVID-19 vaccination in a low middle-income country: a retrospective cross-sectional study. Health Sci Rep. (2023) 6:e1703. doi: 10.1002/hsr2.1703

PubMed Abstract | Crossref Full Text | Google Scholar

82. Nadeem I, Ul Munamm SA, Ur Rasool M, Fatimah M, Abu Bakar M, Rana ZK, et al. Safety and efficacy of Sinopharm vaccine (BBIBP-CorV) in elderly population of Faisalabad district of Pakistan. Postgrad Med J. (2023) 99:463–9. doi: 10.1136/postgradmedj-2022-141649

PubMed Abstract | Crossref Full Text | Google Scholar

83. Nisar MI, Ansari N, Malik AA, Shahid S, Lalani KRA, Chandna MA, et al. Assessing the effectiveness of COVID-19 vaccines in Pakistan: a test-negative case-control study. J Infect. (2023) 86:e144–7. doi: 10.1016/j.jinf.2023.01.016

PubMed Abstract | Crossref Full Text | Google Scholar

84. Wagner AL, Sheinfeld Gorin S, Boulton ML, Glover BA, Morenoff JD. Effect of vaccine effectiveness and safety on COVID-19 vaccine acceptance in Detroit, Michigan, July 2020. Hum Vaccin Immunother. (2021) 17:2940–5. doi: 10.1080/21645515.2021.1917233

PubMed Abstract | Crossref Full Text | Google Scholar

85. World Health Organization. Episode #49—can I get infected after vaccination? In: World Health Organization, , editors. Science in 5. Geneva: World Health Organization (2021).

Google Scholar

86. Day M. COVID-19: stronger warnings are needed to curb socialising after vaccination, say doctors and behavioural scientists. Br Med J. (2021) 372:n783. doi: 10.1136/bmj.n783

PubMed Abstract | Crossref Full Text | Google Scholar

87. Wagner AL, Huang Z, Ren J, Laffoon M, Ji M, Pinckney LC, et al. Vaccine hesitancy and concerns about vaccine safety and effectiveness in Shanghai, China. Am J Prev Med. (2021) 60:S77–86. doi: 10.1016/j.amepre.2020.09.003

PubMed Abstract | Crossref Full Text | Google Scholar

88. Ioannou GN, Bohnert ASB, O'hare AM, Boyko EJ, Maciejewski ML, Smith VA, et al. Effectiveness of mRNA COVID-19 vaccine boosters against infection, hospitalization, and death: a target trial emulation in the omicron (B.1.1.529) variant era. Ann Intern Med. (2022) 175:1693–706. doi: 10.7326/M22-1856

PubMed Abstract | Crossref Full Text | Google Scholar

89. Andersson NW, Thiesson EM, Baum U, Pihlström N, Starrfelt J, Faksová K, et al. Comparative effectiveness of bivalent BA.4-5 and BA.1 mRNA booster vaccines among adults aged ≥50 years in Nordic countries: nationwide cohort study. Br Med J. (2023) 382:e075286. doi: 10.1136/bmj-2022-075286

PubMed Abstract | Crossref Full Text | Google Scholar

90. Bajema KL, Berry K, Streja E, Rajeevan N, Li Y, Mutalik P, et al. Effectiveness of COVID-19 treatment with nirmatrelvir-ritonavir or molnupiravir among U.S. veterans: target trial emulation studies with one-month and six-month outcomes. Ann Intern Med. (2023) 176:807–16. doi: 10.7326/M22-3565

PubMed Abstract | Crossref Full Text | Google Scholar

91. Gazit S, Saciuk Y, Perez G, Peretz A, Ben-Tov A, Stuart EA, et al. Hybrid immunity against reinfection with SARS-CoV-2 following a previous SARS-CoV-2 infection and single dose of the BNT162b2 vaccine in children and adolescents: a target trial emulation. Lancet Microbe. (2023) 4:e495–505. doi: 10.1016/S2666-5247(23)00103-9

PubMed Abstract | Crossref Full Text | Google Scholar

92. Monteiro HS, Lima Neto AS, Kahn R, Sousa GS, Carmona HA, Andrade JSJr, et al. Impact of CoronaVac on COVID-19 outcomes of elderly adults in a large and socially unequal Brazilian city: a target trial emulation study. Vaccine. (2023) 41:5742–51. doi: 10.1016/j.vaccine.2023.07.065

PubMed Abstract | Crossref Full Text | Google Scholar

93. Tran VT, Perrodeau E, Saldanha J, Pane I, Ravaud P. Efficacy of first dose of COVID-19 vaccine versus no vaccination on symptoms of patients with long covid: target trial emulation based on ComPaRe e-cohort. Br Med J Med. (2023) 2:e000229. doi: 10.1136/bmjmed-2022-000229

PubMed Abstract | Crossref Full Text | Google Scholar

94. Català M, Burn E, Rathod-Mistry T, Xie J, Delmestri A, Prieto-Alhambra D, et al. Observational methods for COVID-19 vaccine effectiveness research: an empirical evaluation and target trial emulation. Int J Epidemiol. (2024) 53:dyad138. doi: 10.1093/ije/dyad138

PubMed Abstract | Crossref Full Text | Google Scholar

95. Eusebi P, Speybroeck N, Hartnack S, Stærk-Østergaard J, Denwood MJ, Kostoulas P. Addressing misclassification bias in vaccine effectiveness studies with an application to COVID-19. BMC Med Res Methodol. (2023) 23:55. doi: 10.1186/s12874-023-01853-4

PubMed Abstract | Crossref Full Text | Google Scholar

96. Brookmeyer R, Morrison DE. Estimating vaccine effectiveness by linking population-based health registries: some sources of bias. Am J Epidemiol. (2022) 191:1975–80. doi: 10.1093/aje/kwac145

PubMed Abstract | Crossref Full Text | Google Scholar

97. Rosenberg ES. New COVID-19 cases and hospitalizations among adults, by vaccination status-New York, May 3-July 25, 2021. Morbid Mortal Week Rep. (2021) 70:37a6. doi: 10.15585/mmwr.mm7037a6

PubMed Abstract | Crossref Full Text | Google Scholar

98. Cai M, Xie Y, Topol EJ, Al-Aly Z. Three-year outcomes of post-acute sequelae of COVID-19. Nat Med. (2024) 30:1564–73. doi: 10.1038/s41591-024-02987-8

PubMed Abstract | Crossref Full Text | Google Scholar

99. Viper Group. COVID-19 Vaccine Tracker. (2022). Available at: https://covid19.trackvaccines.org/vaccines/ (accessed November 6, 2023).

Google Scholar

100. De Gier B, Huiberts AJ, Hoeve CE, Den Hartog G, Van Werkhoven H, Van Binnendijk R, et al. Effects of COVID-19 vaccination and previous infection on Omicron SARS-CoV-2 infection and relation with serology. Nat Commun. (2023) 14:4793. doi: 10.1038/s41467-023-40195-z

PubMed Abstract | Crossref Full Text | Google Scholar

101. De Gier B, Van Asten L, Boere TM, Van Roon A, Van Roekel C, Pijpers J, et al. Effect of COVID-19 vaccination on mortality by COVID-19 and on mortality by other causes, the Netherlands, January 2021-January 2022. Vaccine. (2023) 41:4488–96. doi: 10.1016/j.vaccine.2023.06.005

PubMed Abstract | Crossref Full Text | Google Scholar

102. Mcconeghy KW, Hur K, Dahabreh IJ, Jiang R, Pandey L, Gellad WF, et al. Early mortality after the first dose of COVID-19 vaccination: a target trial emulation. Clin Infect Dis. (2024) 78:625–32. doi: 10.1093/cid/ciad604

PubMed Abstract | Crossref Full Text | Google Scholar

103. Van Werkhoven CH, Valk AW, Smagge B, De Melker HE, Knol MJ, Hahné SJ, et al. Early COVID-19 vaccine effectiveness of XBB.1.5 vaccine against hospitalisation and admission to intensive care, the Netherlands, 9 October to 5 December 2023. Euro Surveill. (2024) 29:703. doi: 10.2807/1560-7917.ES.2024.29.1.2300703

PubMed Abstract | Crossref Full Text | Google Scholar