Diagnostic performance of rapid antigen testing for SARS-CoV-2: the COVid-19 AntiGen (COVAG) extension study

Background: This study is the extension of the COVAG study. We compared two RATs, the Panbio COVID-19 Ag Rapid Test (Abbott) and the SD Biosensor Q SARS-CoV-2 Rapid Antigen Test (Roche), against RT-PCR on the foil of new variants.

Methods: We included 888 all-comers at a diagnostic center between October 20, 2021, and March 18, 2022. RT-PCR-positive samples with a Ct value ≤32 were examined for SARS-CoV-2 variants.

Findings: The sensitivity of the Abbott-RAT and Roche-RAT were 65 and 67%, respectively. For both RATs, lower Ct values were significantly correlated with higher sensitivity. For samples with Ct values ≤25, the sensitivities of the Roche-RAT and of the Abbott-RAT were 96 and 95%, for Ct values 25–30 both were 19%, and for Ct values ≥30 they were 6 and 2%, respectively. The RATs had substantially higher sensitivities in symptomatic than asymptomatic participants (76, 77%, vs. 29, 31%, for Abbott-RAT, Roche-RAT, respectively) and in participants referred to testing by their primary care physician (84, 85%) compared to participants who sought testing due to referral by the health department (55, 58%) or a warning by the Corona-Warn-App (49, 49%). In persons with self-reported previous COVID-19 sensitivities were markedly lower than in patients without previous COVID-19: 27% vs. 75% for Roche-RAT and 27% vs. 73% for Abbott-RAT. We did not find significant correlation between vaccination status and sensitivity. The Omicron variant was detected with a sensitivity of 94 and 92%, the delta variant with a sensitivity of 80 and 80% for Abbott-RAT and Roche-RAT, respectively. This difference is attributable to the lower Ct values of the Omicron samples compared to the Delta samples. When adjusted for the Ct value, a multivariate logistic regression did not show a significant difference between Omicron and Delta. In terms of sensitivity, we found no significant difference between the wild-type and the Omicron and Delta variants, but a significantly lower sensitivity to the alpha variant compared to the other variants.

The specificities were > 99% overall.

1 Introduction

Severe acute respiratory syndrome Coronavirus type 2 (SARS-CoV-2) is the causative agent of Coronavirus disease 19 (COVID-19). COVID-19 emerged in late 2019, quickly spread around the world and was declared a global pandemic on March 11, 2021, by the World Health Organization (WHO) (1). Since its emergence multiple SARS-CoV-2 variants developed which mostly were characterized by mutations in the Spike protein but also within the Nucleocapsid protein (2–5). Variants showing a decrease in the effectiveness of available diagnostic tests among other criteria are termed Variants of Concern (VOC) by the WHO (6). To date the WHO has listed 5 VOCs, namely: B.1.1.7 (alpha), B.1.351 (beta), P.1 (gamma), B.1.617.2 (delta) and the currently prevailing B.1.1.529 (Omicron) (6). For Omicron several sub-lineages have been identified with BA.5 being the currently dominant one in Europe (3).

The clinical presentation of COVID-19 ranges from asymptomatic to prolonged illness requiring intensive care treatment and death (7, 8). As SARS-CoV-2 can be transmitted by symptomatic as well as asymptomatic persons the identification of infectious carriers is crucial to contain COVID-19 by means of contact tracing and isolation of infectious patients (8). This requires effective testing and an early diagnosis of COVID-19. Detection of acute SARS-CoV-2 infection can be achieved by direct testing including nucleic acid amplification tests (NAATs) or through rapid antigen tests (RATs). NAATs identify viral RNA in specimens from the respiratory tract while RATs recognize viral proteins, mostly the Nucleocapsid protein (9). To date NAAT-based assays such as reverse transcription-polymerase chain reaction (RT-PCR) are the gold standard in detecting acute SARS-CoV-2 infection. RATs are widely employed as well as they can be conducted at the point of care, provide fast results within 15–30 min, and can be used for self-testing. Positive RAT results need to be verified by RT-PCR testing (10). Indirect tests including assays detecting antibodies against the Spike-or the Nucleocapsid protein are not useful in the diagnosis of acute infection as they only become positive after 3 days and more or may be already positive from an earlier infection (Nucleocapsid-and Spike antibodies) or vaccination (Spike-Antibodies) (11, 12).

This study is the extension of the COVAG study originally performed from February 1, 2021, to March 31, 2021. During the first data collection period we saw that the alpha variant decreased the effectiveness of the RATs compared to the wild-type (13). As new SARS-CoV-2 variants emerged afterwards, the COVAG study was continued to comprehensively examine two of the most sensitive RATs in a real-world, prospective, head-to-head study, placing specific emphasis on clinical characteristics and the presence of SARS-CoV-2 variant genotypes (9).

2 Methods

2.1 Setting and participants

This prospective study was conducted at the Corona Test Centre Cannstatter Wasen in Stuttgart, Germany as an extension of the COVAG study (13). Individuals scheduled for RT-PCR testing of nasopharyngeal swabs were advised of the study orally and in writing. Participants had to be aged ≥18 years and capable of understanding the nature, significance, and implications of the study. Children and adolescents <18 years of age and patients obviously suffering from clinical conditions requiring emergency hospitalization were excluded. All participants provided written and informed consent. The study was approved by Ethics Committee II (Mannheim) of the University of Heidelberg (reference number 2020-417MF) and the German Institute for Drugs and Medical Devices.

We recorded demographic characteristics, reasons for testing, medical history including SARS-CoV-2 vaccination history, clinical symptoms, and vital signs (heart rate, blood pressure, body temperature, and oxygen saturation) and we stratified the reasons for testing into four major categories: participants referred by their primary care physicians, by the Health Department, participants seeking RT-PCR testing to confirm a positive antigen test and participants who received a warning in their digital contact-tracing app (Corona-Warn-App). We grouped the participants based on their COVID-19 vaccination status into non-vaccinated (0 or 1 vaccination), vaccinated (2 vaccinations), boostered (3 or more vaccinations) and with unknown vaccination status.

In addition to collecting the oro-and nasopharyngeal swabs for RT-PCR testing, we collected two completely independent nasopharyngeal swab specimens to run two commercially available and widely used RATs. The swabs were collected by medically educated personnel of the test center by rotating teams with strict adherence to the instructions issued by the manufacturers. We used the Abbott Panbio™ COVID-19 Ag Rapid Test (Abbott Rapid Diagnostics Jena GmbH, Jena, Germany www.abbott.com/poct) and the Roche-SD Biosensor SARS-CoV-2 Rapid Antigen Test (identical to SD BIOSENSOR Standard Q COVID-19 Ag www.sdbiosensor.com; Roche Diagnostics; Mannheim, Germany www.roche.com). We chose those two tests in continuation of our first data collection period and because they were among the most sensitive tests according to a Cochrane analysis (13, 14).

Hereafter, we refer to the tests as Abbott-RAT and Roche-RAT, respectively. We randomly assigned the participants to three sampling groups according to the sequence of collecting the nasopharyngeal swabs (group 1: RT-PCR, RAT-Roche, RAT-Abbott; group 2: RAT-Roche, RAT-Abbott, RT-PCR; and group 3: RAT-Abbott, RT-PCR, RAT-Roche) to reduce bias based on the order of test performance.

2.2 Analytical procedures

Both the Abbott-RAT and the Roche-RAT were carried out by medically educated staff according to the manufacturers’ instructions on-site at the Corona Test Centre, immediately after sampling the nasopharyngeal swabs. The nasopharyngeal swabs for real-time RT-PCR (rRT-PCR) testing were placed in 2 mL of a phosphate-buffered saline solution (ISOTON™ II Diluent, Becton Dickinson, Galway, Ireland) and delivered to the SYNLAB Medical Care Centre Leinfelden-Echterdingen. PCR was always performed after onsite interpretation of the RATs, excluding that the interpretation of RATs was affected by the PCR results.

SARS-CoV-2 RNA was extracted from the nasopharyngeal swab samples and purified using the PurePrep Pathogens kit and a PurePrep 96 instrument (Molgen, Veenendaal, the Netherlands) within 6 h after sampling to limit degradation. The integrity of the RNA was monitored by co-amplification of a control RNA included in the solution for the lysis of the swabs. In cases in which neither SARS-CoV-2 RNA nor the control RNA were detected, the RNA preparation was repeated. The rRT-PCR assay was performed using either the RIDA®GENE SARS-CoV-2 test kit (R-Biopharm, Darmstadt, Germany) or the Allplex SARS-CoV2 (Seegene, Seoul, Korea) or the Virella SARS-CoV-2 seqc (Gerbion, Kornwestheim, Germany) on the CFX96 Touch Real-Time PCR detection device (Bio-Rad, Feldkirchen, Germany) or the CFX-96 IVD Real-Time PCR detection device (Bio-Rad, Feldkirchen, Germany) according to the manufacturers’ instructions. The RIDA®GENE SARS-CoV-2 test kit targets the SARS-CoV-2 envelope (E) gene, the Allplex SARS-CoV2 targets the N-gene, S-gene/RdRP and the E-gene (pan Sarbecovirus) and the Virella Seqc SARS-CoV2 targets the RdRp/S-gene and the E-gene (pan Sarbecovirus). Samples producing a cycle threshold (Ct) ≤ 35 were considered positive by RT-PCR.

We screened RT-PCR-positive samples with a Ct ≤ 32 for SARS-CoV-2 variants of concern (VOC). Until November 8^th 2021 this analysis was performed at SYNLAB Medical Care Center Leinfelden-Echterdingen using the Kits Seegene Allplex Variant I (Seegene, Seoul, Korea) and Virella SARS-CoV2 Mut 3 (Gerbion, Kornwestheim, Germany) according to the supplier’s instructions. Afterwards the VOC analysis was performed at SYNLAB Medical Care Center Weiden using the Novaplex SARS-CoV-2 Variants I Assay, Novaplex SARS-CoV-2 Variants IV Assay and Novaplex SARS-CoV-2 Variants VII Assay (Seegene, Seoul, Korea) according to the supplier’s instructions.

Samples were screened for B.1.617.2 (delta), B.1.617.2.1 (delta plus), B.1.1.529/BA.1 (omicron) and BA.2 (omicron stealth). Samples with positive results for L452R and P681R and absence of K417N were assigned to the delta variant. Positive results for L452R, P681R and K417N were considered as Delta plus. Presence of N501Y, E484A and HV69/70del were considered as Omikron BA.1 and occurrence of N501Y, E484A with absence of HV69/70del as Omikron BA.2.

2.3 Statistical analysis

Continuous data are presented as means, standard deviations (SD), medians, and 25th and 75th percentiles. Categorical data are presented as absolute numbers and percentages (Table 1).

Table 1

Table 1. Clinical characteristics of participants of the COVAG Extension study with reasons for testing other than to confirm a positive RAT.

In our analysis, the performance indicators for the two RATs in relation to RT-PCR (chosen as the gold standard for having COVID-19) are given by sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and diagnostic efficacy (number of correct test results divided by the total number of test results).

In Table 2, the p-values apply to two-sided testing of the null hypothesis that the difference between the Abbott-RAT and the Roche-RAT performance indicators is equal to zero. The probability densities underlying the two-sided testing are estimated by means of 5,000 bootstrap iterations.

Table 2

Table 2. Diagnostic performance of 2 commercial RATs for SARS-Cov-2 antigens in participants of the COVAG extension study with reasons for testing other than to confirm a positive RAT.

The risk of having COVID-19 according to baseline anthropometric and anamnestic characteristics was expressed in terms of crude odds ratios (ORs) and ORs adjusted for age and sex as calculated by logistic regression (Supplementary Table S1).

We also visualized the sensitivities of both RDTs relative to the rRT-PCR-derived Ct values (Figure 1) and the PPVs and NPVs according to hypothetical disease prevalence rates in the range of 0–0·05 (Figure 1). To compare the PPV and NPV of the RDTs with standardized criteria on performance, we also used the following hypothetical sensitivity and specificity levels (tiers 1–3) recommended by Kost et al. (15): tier 1, 90, 95%; tier 2, 95, 97.5%; and tier 3, 100%, ≥99% (Figure 1).

Figure 1

Figure 1. Framing of the COVAG extension study (October 20, 2021 to March 18, 2022) into the time course of the COVID-19 pandemic in Germany. Abszissa: calendar week within 2021 and 2022; bars: Germany-wide weekly proportions of variants of concern (VOC) in percent. Blue solid line: estimated proportion of variant B.1.617.2 (Delta) in the COVAG extension study to Germany (based on logistic regression with the categories ‘Delta’ vs. ‘Omikron’).

Finally, we investigated whether the sensitivities of the two RDTs were related to the reason for testing, comorbidities, clinical symptoms, vital signs, or SARS-CoV-2 genotypes using univariate (Table 2) and multivariate logistic regression (Table 3).

Table 3

Table 3. Predictors of positive RATs among SARS-CoV-2 positive samples in a multivariate model (CT ≤ 32, N = 155).

The statistical tests were two-sided and p < 0·05 was considered significant. The analyses were carried out using R v4.0.2.¹

3 Results

3.1 Clinical characteristics of participants

The extension of the COVAG study was conducted from October 20, 2021 to March 18, 2022. Figure 1 shows the data collection period and the emergence of variants framed within the course of the COVID-19 pandemic in Germany. A total of 1,508 persons agreed to participate in this study. 21 persons were disregarded from further evaluation because at least one of the three tests was not available. This resulted in 1487 persons enrolled in the COVAG Extension study (Figure 3; Supplementary Table S1) including 801 (53.9%) women, 685 (46.1%) men and one diverse person (0.1%). Adverse effects from performing any of the tests were not experienced.

Figure 2

Figure 2. Flowchart illustrating the inclusion of participants into the COVAG extension study and data analysis.

Figure 3

Figure 3. Relationships between sensitivities of RDTs vs. rRT-PCR cycle threshold (Ct) values. The solid lines indicate sensitivities, the dotted lines represent the upper, and the lower bounds the corresponding 95% confidence intervals. (A) Left: Roche-RDT; right: Abbott-RDT. (B) Sensitivities according to SARS-CoV-2 genotypes. Left: Roche-RDT; right: Abbott-RDT; red: Delta variant; blue: Omikron variant.

Within the period of data collection, self-testing with RATs and RT-PCR confirmation in the case of a positive RAT was performed very frequently in Germany which in many participants who already had a positive self-test beforehand. To reduce selection bias, we excluded these 591 (39.7%) participants from the further analyses (Figure 2). 888 participants were tested for other reasons. Those included a warning by the Corona-Warn-App in 419 (28.2%) participants, a referral from the health department in 236 (15.9%) participants, and a referral from the primary care physician in 233 (15.7%) participants. For 8 (0.5%) participants no information regarding the reason for testing was available (Table 1). The anthropometric and anamnestic characteristics of all 1,487 participants can be found in Supplementary Table S1. Further data analysis was performed for the 888 participants with reasons for testing other than to confirm a positive RAT.

Of 888 participants, 497 (56%) were women and 390 (43.9%) were men, one person (0.1%) is assigned neither to women nor to men. 665 (74.9%) participants self-reported having no comorbidities, while 223 (25.1%) reported having any comorbidities. The most common comorbidities were hypertension (9.5%) and dyslipoproteinemia (4.7%). Other comorbidities were low in frequency. 101 (11.4%) participants self-reported having had a previous COVID-19 infection (Table 1).

98 (11.0%) participants are non-vaccinated (0 or 1 vaccination against COVID-19), 321 (36.2%) participants are ‘vaccinated’ (2 vaccinations against COVID-19), 463 participants (52.1%) have received a booster vaccination (3 or more vaccinations against COVID-19). For six persons (0.7%), the vaccination status is unknown.

447 (50.3%) participants reported having clinical symptoms while 441 (49.7%) reported none. The most common symptoms were malaise, cough, headache, and musculoskeletal pain at frequencies of 36.6, 30.3, 30.0, and 16.3%, respectively (Table 1).

188 (21.2%) participants were tested positive for SARS-CoV-2 by RT-PCR. 126 (14.2%) were tested positive by the Abbott-RAT and 128 (14.4%) by the Roche-RAT. 125 (14.1%) samples had a Ct value ≤25, 16 (1.8%) a Ct value 25–30, 47 (5.3%) ≥ 30. 155 (17.5%) RT-PCR samples had a Ct value ≤32. 52 RT-PCR positive samples with a Ct value ≤32 could not examined be for variants. Of the remaining 103 samples, the Omicron variant was found in 41 and the Delta variant was found in 62.

3.2 Diagnostic performance of RATs

Sensitivity. The Abbott-RAT and the Roche-RAT had overall sensitivities of 65.4% (95% CI 60.7–70.2%) and 67.0% (95% CI 62.4–71.8%) respectively (Table 2). The sensitivities of both RATs were significantly associated with the Ct-value derived from RT-PCR (Figure 1A).

The Abbott-RAT and Roche-RAT did not show a significant difference in sensitivity (p = 0.2091; Table 2). Due to higher power in the total study cohort (n = 1,487) the Roche-RAT had a significantly higher sensitivity than the Abbott-RAT (p = 0.0093, Supplementary Table S2). Among participants seeking testing due to a referral by their primary care physician, the sensitivities for the Abbot-RAT and Roche-RAT were 83.5 and 84.8%, for participants with a referral by the health department they were 54.6 and 57.6% and following a warning by their Corona-Warn-App the sensitivities were 48.8% for both tests (Table 2), respectively. In the participants excluded because they were tested to confirm a positive antigen test the sensitivities of the Abbott-RAT and the Roche-RAT were 93.0 and 94.5%.

Participants with previous COVID-19 showed significantly lower sensitivities of only 26.7% for both RATs (OR 0.12 (95%CI: 0.05,0.3), p < 0.0001). This finding is attributable to Ct values being markedly higher (Median 31.2) in patients with previous COVID-19 and not consistent anymore when adjusted for the Ct value (Table 3).

For participants without previous COVID-19, significantly higher sensitivities (72.8 and 74.7%, Abbott-RAT and Roche-RAT, respectively) were found in line with markedly lower Ct values (Median 19.2).

In symptomatic participants, the sensitivities were significantly higher (76.0 and 77.4%%, Abbott-RAT and Roche-RAT, respectively) than in asymptomatic participants (28.6 and 31.0%, Abbott-RAT and Roche-RAT, respectively). This finding is in line with Ct-values being lower in symptomatic patients than in asymptomatic patients (Ct Median 18.7 vs. 30.8, Table 2).

We further analyzed the diagnostic performance of RATs according to the vaccination status. The sensitivities of the RATs in non-vaccinated participants (0 or 1 vaccination) were 64.9 and 67.7% for Abbott-RAT and Roche-RAT, respectively. For participants with two vaccinations, the sensitivities were 59.4 and 60.9%. For participants with at least one booster vaccination, we find sensitivities equal to 70.6% for both RATs.

We also investigated whether the SARS-CoV-2 variants Delta and Omicron affected the sensitivity of the RATs. Both variants had similar sensitivities compared to the wild-type from the first wave of the Covag study. Compared to the alpha variant the alpha variant had significantly lower sensitivities than the wild-type, delta and omicron (Figure 4) (13).

Figure 4

Figure 4. rRT-PCR cycle thresholds (Ct) values on rRT-PCR for SARS-Cov-2 RNA of different variants versus sensitivities of the Roche-RDT. The solid lines indicate sensitivities, the dotted lines represent the upper and the lower bounds the corresponding 95% confidence intervals. Magenta: Delta; red: Omikron; green: Alpha; blue: WT.

To firmly establish independent predictors of sensitivity, we calculated ORs for having a positive RAT according to subgroups by multivariate logistic regression (Table 3). Covariables were age, sex, Ct value, reason for testing, presence or absence of any comorbidity and previous COVID-19, COVID-19 vaccination status, presence or absence of any clinical symptom, and the SARS-CoV-2 genotype. As expected, Ct values were strongly associated with the sensitivities of both tests. The sensitivities of the Abbott-RAT and Roche-RAT were lower in participants who sought testing due to a warning in the Corona Warn App.

When excluding the Ct value from the multivariate logistic regression symptomatic participants were detected with a significantly higher sensitivity than asymptomatic participants (Abbott-RAT: OR 4.35, p = 0.0081; Roche-RAT: OR 3.46, p = 0.0216). However, when adjusting for the Ct value this finding was not significant anymore (Table 3). The vaccination status was not associated with a change in sensitivity of the RATs.

As the Ct value is the strongest predictor for the sensitivity of the RATs, we calculated the sensitivity of the RATs separately for different Ct values. For a Ct value ≤25 the sensitivities were 95.2 and 96.0% for the Abbott-RAT and the Roche-RAT, respectively (Table 4). For a Ct value of 25–30 both RATs had a sensitivity of 18.8%. For a Ct value of 30–32, the sensitivities were 0.0 and 7.1% respectively, for Ct values ≥32 the sensitivities were 3.0 and 6.0% for Abbott-RAT and Roche-RAT, respectively.

Table 4

Table 4. Sensitivities of Abbott-RAT and Roche-RAT versus Ct values.

Specificity. The specificity exceeded 99% overall and in mostly all participant strata (Table 2; Supplementary Table S2).

PPV, NPV, and diagnostic performance. The rate of true negatives in our study cohort (n = 888) was 697 of 700 (99.6%) and 698 of 700 (99.7%), the rate of false negatives was 65 of 188 (34.6%) and 62 of 188 (33.0%) for the Abbott-RAT and the Roche-RAT, respectively. The rate of true positives was 123 of 188 (65.4%) and 126 of 700 (67%). The rate for false positives was 3 of 700 (0.4%) and 2 of 700 (0.3%) for Abbott-RAT and Roche-RAT, respectively.

When also including the participants who already had a positive self-test beforehand (total of n = 1,487) the rate of false negatives decreased to 101 of 704 (14.4%) and 90 of 704 (12.8%) for the Abbott-RAT and the Roche-RAT, respectively. The rate of false positives was also overall very low with 4 of 783 (0.5%) and 2 of 783 (0.3%) for the Abbott-RAT and the Roche-RAT. Of the 591 participants who sought RT-PCR testing to confirm a positive self-test, 511 (86.5%) were confirmed positive by RT-PCR while 80 (13.5%) were tested negative by RT-PCR.

The SARS-CoV-2 prevalence in our study cohort was 78.8% (n = 888). At this prevalence the PPV was at 97.6 and 98.4% for Abbott-RAT and Roche-RAT (n = 888, Table 2). For symptomatic participants the PPV was higher (98.2 and 98.3%, Abbott-RAT and Roche-RAT, respectively) than for asymptomatic participants (92.3 and 96.3%, Abbott-RAT and Roche-RAT, respectively). The NPV was 91.5 and 91.8% for Abbott-RAT and Roche-RAT. The NPV was higher for asymptomatic (93.0 and 93.2%, Abbott-RAT and Roche-RAT, respectively) than for symptomatic participants (89.5 and 90.1%, Abbott-RAT and Roche-RAT, respectively).

Because patients with SARS-CoV-2 infections were enriched in our study population compared to the general population, we examined the PPVs and NPVs at assumed prevalence rates up to 0.05 (Figure 5). To compare the PPV and NPV of the RATs with standardized performance criteria, we also used the following hypothetical sensitivity and specificity levels (tiers 1–3) recommended by Kost et al. (15): tier 1, 90, 95%; tier 2, 95, 97.5%; and tier 3, 100%, ≥99% (Figure 5). At this prevalence rate, our results suggest a PPV and NPV of 88.9 and 98.2% for Abbott-RAT, and 92.5 and 98.3% for the Roche-RAT, the Roche-RAT displaying a higher PPV than the Abbott-RAT and both scoring higher than the hypothetical tiers 1 through 3, reflecting increases in NPV in the order of Abbott-RAT < Roche-RAT < tier 1 < tier 2 < tier 3. The NPVs ranged in the order of tier 3 > tier 2 > tier 1 > Roche-RAT > Abbott-RAT.

Figure 5

Figure 5. (A) Prevalence versus PPV for tier 1–3 [for hypothetical sensitivities and specificities (15)] and both rapid tests. (B) Prevalence versus NPV for tier 1–3 [for hypothetical sensitivities and specificities (15)] and both rapid tests.

4 Discussion

This study is an extension of the COVAG study which is one of the largest prospective, real-world evaluations of RATs to date (13). We compared two of the most sensitive RATs provided by Abbott Diagnostics and Roche Diagnostics, especially in the light of newly emerged variants (9). We found that the sensitivities of RATs for asymptomatic patients was as low as 30%. We found that the Omicron and Delta variant were detected with not significantly different sensitivities compared to the wild-type at Ct values >25 (13).

In contrast to the first wave of our study, there was no significant difference in sensitivity between the Abbott-RAT and Roche-RAT (13). However, with an extended sample size (n = 1,487) after including participants seeking RT-PCR testing to confirm a positive antigen test, the Roche-RAT had a significantly higher sensitivity than the Abbott-RAT (p = 0.0093; Supplementary Table S2), attributable to the participants with an age ≤ median. This finding is in good agreement with the results of the first wave of our study.

The sensitivities were substantially higher among participants referred by their primary care physician (84–85%, Table 2). As primary care physicians refer patients to RT-PCR testing based on their clinical presentation and history, the pretest probability is higher and patients with higher symptom burden sent for testing, also reflected by lower Ct values in these participants. This shows that the sensitivity of the RATs can be increased by considering the clinical background. The PPVs of RATs was overall very good (88–92%). Compared to the tiers recommend by Kost et al. the NPV occur to be lower than the recommend values of tier 1–3. However, due to the smallness of the discrepancies between the measured NPVs and the recommended NPV ranges (<2%) and the small number of false positives, the last statement made about the NPVs should be taken with caution.

4.1 Diagnostic performance of the RATs

The WHO formulated minimum performance requirements of ≥80% sensitivity and ≥ 97% specificity for RATs (16). The European Centre for Disease Prevention and Control (ECDC) agreed to the performance requirements set by the WHO (17). In our study both RATs did not meet the sensitivity performance requirements while meeting the specificity requirements (Abbott-RAT: sensitivity 65.4%, specificity 99.6%; Roche-RAT: sensitivity 67.0%, specificity 99.7%). Similar results were reported by a Cochrane Analysis which reported sensitivities of 56.7% (95% CI 44.3–68.3%) and 64.4% (95% CI 52.2–75.0%) for the Abbott-RAT and the Roche-RAT, respectively (9). In a large comparative in vitro evaluation of 122 RATs reported the Paul-Ehrlich-Institut (PEI), the overall sensitivity of the Abbott-RAT and the Roche-RAT were 64.0 and 46.0%, respectively (18). While the Abbott-RAT showed a comparable sensitivity of 65.4% in our study, the Roche-RAT yielded a better sensitivity of 67.0%. However, also in the study by the PEI both RATs failed to meet the sensitivity requirement set by the WHO. This is in large contrast to the sensitivities of 97.6 and 95.5%, respectively, reported by the providers Abbott and Roche. for samples with Ct values ≤30 (19, 20).

During our study comparable sensitivities (95.2 and 96% for Abbott-RAT and Roche-RAT, respectively) were reported only for Ct values ≤25. For Ct values of 25–30 the sensitivities were only 18.8% for both tests.

The RATs´ performance strongly relates to Ct values. The study by the Paul-Ehrlich-Institut showed sensitivities for the Abbott-RAT of 100% for Ct-values ≤25, 60.9% for Ct values between 25–30 and 0% for Ct values ≥30 (18). The Roche-RAT in comparison yielded a sensitivity of 88.9% for Ct values ≤25, 30.4% for Ct values between 25–30 and also 0% for Ct values ≥30 (18). Evidently thus, the performance of the RATs in our study is worse than in the in vitro study by the Paul-Ehrlich-Institut, suggesting that challengeable information will only be obtained under real world conditions. This notwithstanding the common denominator of the results from Paul-Ehrlich-Institut and of ours is that the performance requirements are only met for samples with a Ct ≤ 25. Hence, patients with a high viral load are well detected while patients with a lower viral load are missed (21).

An important clinical distinction is whether symptoms are present or not. The sensitivity of the RATs is markedly lower for asymptomatic than for symptomatic patients. With a sensitivity of around 30%, asymptomatic and infected patients were detected at very low sensitivity in our study. Symptomatic patients on the other hand are detected with a sensitivity of around 77%. A Cochrane analysis by Dinnes et al. reported similar results for symptomatic (Abbott-RAT: 74.8%; Roche-RAT: 78.8%) and higher results for asymptomatic (Abbott-RAT: 56.9%; Roche-RAT: 59.4%) patients compared to our study (9). Although slightly below the performance requirements of the WHO RATs may be considered useful in symptomatic patients while they are not in asymptomatic patients. These differences in sensitivity are clearly attributable to the lower Ct values of symptomatic patients. In Germany RATs have been used for screening of asymptomatic persons (10). Yet, in these patients RATs are clearly insufficient for screening.

The RNA viral load determined by RT-PCR is only a proxy for the infectiousness of patients as also non-infectious viral RNA is detected by RT-PCR. To reliably determine the infectiousness of a patient, viral growth can be examined in culture. In a study from the UK, contacts of SARS-CoV-2 infected patients were recruited, and RT-PCR and virus culture were performed daily. Additionally, a RAT different from the ones used in our study was performed in RT-PCR positive samples as well as in samples one day before and after a positive RT-PCR. The sensitivity of the RATs was higher for samples with positive viral cultures (79%) than for samples with only positive RT-PCR (47%). Positive viral cultures were detected for a median of 5 days (IQR 3–7 days) and the peak viral load determined by viral cultures and RT-PCR was at a median of 3 days after symptom onset (IQR 3–5/6 days). Interestingly the sensitivity of the RATs was lower before and during the peak viral load (sensitivity: 67%) than after the peak viral load (sensitivity: 92%) (22). This shows that RATs have reduced sensitivity during the beginning of infection possibly leading to delayed diagnosis (22). In a study from Germany the Roche-RAT was compared to RT-PCR and viral culture. Although the Roche-RAT reached a sensitivity of only 42.8%, none of the samples with positive viral cultures was missed (23). Hence and accordance to the current study, RATs appear to have a low overall sensitivity, while highly infectious participants may reliably be detected.

The specificity of the RATs was overall very good and met the specificity requirements of the WHO and ECDC (16, 17, 24).

True positivity of the RATs approaches 100 percent and false positive results do virtually not occur. This places into question whether the general recommendation to confirm a positive RAT by PCR should always be followed. However, copy numbers determined upon disease onset may be useful to monitor the progression of and recovery from COVID-19.

4.2 Influence of the SARS-CoV-2 genotype on the diagnostic performance of RATs

During the first data collection period from February 1 to March 31, 2021, the dominant variants were the wild-type and the alpha variant. The sensitivities of the RATs for the alpha variant were significantly lower than for the wild-type also when adjusted for the Ct-value (13). In the current wave of our study (October 20, 2021 to March 18, 2022), the prevailing variants were Delta followed by Omicron. Omicron was detected with a high sensitivity of 92–94%, while Delta was detected with a lower sensitivity of 80%. This difference can solely be explained by the lower Ct values of Omicron compared to Delta (Median 17.6, IQR 15.7–19.8 vs. Median 19.6, IQR 16.3–23). Consistently, in a multivariate logistic regression adjusted for the Ct values there was no significant difference between Omicron and Delta anymore. Also, when compared at set Ct values of ≤25, 25–30, ≥30 there was no significant difference in sensitivity for Delta and Omicron, respectively. While it has been argued that that Omicron produces a higher viral load leading to better detection by RATs in general, recent findings do not confirm this assumption (25, 26). Another study from the USA also found that the sensitivities for Omicron compared to the Delta variant are not significantly different (27).

We further examined the sensitivities for Omicron and Delta compared to the wild-type data coming from the first data collection period.

4.3 Influence of the COVID-19 vaccination and previous infection on the diagnostic performance of RATs

For patients with previous COVID-19 the sensitivities for Abbott-RAT and Roche-RAT were very low (26.7%). These low sensitivities are attributable to the high Ct values in these patients (Median 30, IQR 25–33). This is plausible because patients with a previous COVID-19 infection may have lower viral loads due to mucous IgA built in response to the previous infection (28). As in the first data collection period, we also found in the COVAG Extension study that both the sensitivities and the viral load of patients with comorbidities are low. This is unexpected and may reflect a referral bias in the sense that the indication for testing is more frequent and earlier in patients at high risk for severe COVID-19. There was no significant difference in the sensitivities between vaccinated and unvaccinated participants (65–66% vs. 66–70%, Table 2), perhaps since mucous IgA is formed to a lesser extent after vaccination (29). This could explain why the vaccination status does not seem to influence the sensitivity of RATs, while a previous COVID-19 infection could. Another explanation would be that after vaccination antibodies are formed only against the Spike protein whereas after a previous infection antibodies against the Spike protein and the Nucleocapsid protein are formed (30). As RATs detect the Nucleocapsid antigen Nucleocapsid antibodies could reduce available antigens for detection.

4.4 Limitations

Among the limitations of this study is that the reference method RT-PCR does not indicate the infectiousness of patients, because RT-PCR can also detect non-viable virus particles, also there is a certain correlation between the Ct value and infectivity (31). Another limitation is that we used three different PCR kits targeting different genes. This may have produced small differences in the PCR performance characteristics.

Furthermore, we performed RATs once only and not in series. Serial testing for SARS-CoV-2 with RATs may substantially increase their diagnostic performance (32).

4.5 Conclusion

The diagnostic performance of RATs is highly associated with the viral load. The sensitivity of RATs is substantially higher in symptomatic than in asymptomatic patients and in patients referred by primary care physicians compared to other reasons for testing. Hence, RATs are significantly more useful in a clinical setting than for screening purposes. Our study does not suggest that the vaccination status influences the sensitivity of RATs.

Data availability statement

Data will be made available to researchers upon justified request and formal agreement to make sure that rules of good scientific practice are obeyed and that credit is given to the people who have been in charge of the design and the organization of the study. Interested researchers are invited to address their request or proposal to WM (winfried.maerz@synlab.com). The authors confirm that they accessed and validated these data and that all other researchers can access the data in the same manner the authors did.

Ethics statement

The studies involving humans were approved by Ethik-Kommission II der Universität Heidelberg, Medizinische Fakultät Mannheim. The studies were conducted in accordance with the local legislation and institutional requirements. The participants provided their written informed consent to participate in this study.

Author contributions

CW: Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Project administration, Supervision, Visualization, Writing – original draft, Writing – review & editing. AD: Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Supervision, Validation, Visualization, Writing – original draft, Writing – review & editing. EW: Formal analysis, Investigation, Project administration, Writing – review & editing. H-JW: Writing – review & editing, Project administration. HB: Investigation, Writing – review & editing. AS: Investigation, Writing – review & editing. SH: Writing – review & editing. NL: Investigation, Methodology, Writing – review & editing. WM: Conceptualization, Data curation, Formal analysis, Funding acquisition, Investigation, Methodology, Project administration, Resources, Supervision, Validation, Visualization, Writing – original draft, Writing – review & editing.

Funding

The author(s) declare financial support was received for the research, authorship, and/or publication of this article. The costs of the study were defrayed by SYNLAB Holding Deutschland GmbH. The management had no role in writing of the report or the decision to submit for publication. There was no financial support to SYNLAB Holding Deutschland GmbH from the manufacturers of the assays used in this evaluation and there has been no other financial support for this work that could have influenced its outcome.

Acknowledgments

We thank the participants for joining in free of remuneration. We thank Katja Pöhl, CEO, SYNLAB MVZ Leinfelden-Echterdingen and Christoph Mahnke, CEO SYNLAB Holding Deutschland GmbH, who supported the study. We thank all study personnel, especially Manuel Kraft, Ali Hussein, Alexander Ignatenko, Brigitte Schwandt, Orhan Bunjaku, and Rosa Oberhauser for carrying out the RATs and documenting the participants at the Corona Test Center Cannstatter Wasen, Stuttgart.

Conflict of interest

CW, EW, HB, AS, NL, SH, and WM were employed by SYNLAB Holding Germany GmbH or its regional subsidiaries. AD is the owner of Company Dr. Dressel Consulting.

The remaining author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fmed.2024.1352633/full#supplementary-material

Footnotes

1. ^http://www.r-project.org

References

1. World Health Organization. (2020). WHO characterizes COVID-19 as a pandemic. Available at: https://www.who.int/emergencies/diseases/novel-coronavirus-2019/events-as-they-happen (Accessed September 24, 2022)

Google Scholar

2. Zhang, Y, Zhang, H, and Zhang, W. SARS-CoV-2 variants, immune escape, and countermeasures. Front Med. (2022) 16:196–207. doi: 10.1007/s11684-021-0906-x

PubMed Abstract | Crossref Full Text | Google Scholar

3. European Centre for Disease Prevention and Control (ECDC). (2022). SARS-CoV-2 variants of concern as of 22 September 2022. Available at: https://www.ecdc.europa.eu/en/covid-19/variants-concern (Accessed September 26, 2022)

Google Scholar

4. Wu, H, Xing, N, Meng, K, Fu, B, Xue, W, Dong, P, et al. Nucleocapsid mutations R203K/G204R increase the infectivity, fitness, and virulence of SARS-CoV-2. Cell Host Microbe. (2021) 29:1788–1801.e6. doi: 10.1016/j.chom.2021.11.005

PubMed Abstract | Crossref Full Text | Google Scholar

5. Osterman, A, Badell, I, Basara, E, Stern, M, Kriesel, F, Eletreby, M, et al. Impaired detection of omicron by SARS-CoV-2 rapid antigen tests. Med Microbiol Immunol. (2022) 211:105–17. doi: 10.1007/s00430-022-00730-z

PubMed Abstract | Crossref Full Text | Google Scholar

6. World Health Organization. (2022) Tracking SARS-CoV-2 variants. Available at: https://www.who.int/activities/tracking-SARS-CoV-2-variants (Accessed September 26, 2022)

Google Scholar

7. Zhou, F, Yu, T, Du, R, Fan, G, Liu, Y, Liu, Z, et al. Clinical course and risk factors for mortality of adult inpatients with COVID-19 in Wuhan, China: a retrospective cohort study. Lancet. (2020) 395:1054–62. doi: 10.1016/s0140-6736(20)30566-3

PubMed Abstract | Crossref Full Text | Google Scholar

8. Glenet, M, Lebreil, AL, Heng, L, N’Guyen, Y, Meyer, I, and Andreoletti, L. Asymptomatic COVID-19 adult outpatients identified as significant viable SARS-CoV-2 shedders. Sci Rep. (2021) 11:20615. doi: 10.1038/s41598-021-00142-8

PubMed Abstract | Crossref Full Text | Google Scholar

9. Dinnes, J, Sharma, P, Berhane, S, van Wyk, SS, Nyaaba, N, Domen, J, et al. Rapid, point-of-care antigen tests for diagnosis of SARS-CoV-2 infection. Cochrane Database Syst Rev. (2022) 2022:CD013705. doi: 10.1002/14651858.cd013705.pub3

PubMed Abstract | Crossref Full Text | Google Scholar

10. Bundesministerium für Gesundheit. (2022) Fragen und Antworten zu COVID-19 Tests. Bundesministerium für Gesundheit. Available at: https://www.bundesgesundheitsministerium.de/coronavirus/nationale-teststrategie/faq-covid-19-tests.html (Accessed December 14, 2022)

Google Scholar

11. Deeks, JJ, Dinnes, J, Takwoingi, Y, Davenport, C, Spijker, R, Taylor-Phillips, S, et al. Antibody tests for identification of current and past infection with SARS-CoV-2. Cochrane Libr. (2020) 2020:cd013652. doi: 10.1002/14651858.cd013652

PubMed Abstract | Crossref Full Text | Google Scholar

12. Stocking, C, de Miguel, L, Suteu, G, Dressel, A, Soricelli, A, Roskos, M, et al. Evaluation of five widely used serologic assays for antibodies to SARS-CoV-2. Diagn Microbiol Infect Dis. (2022) 102:115587. doi: 10.1016/j.diagmicrobio.2021.115587

PubMed Abstract | Crossref Full Text | Google Scholar

13. Wertenauer, C, Brenner Michael, G, Dressel, A, Pfeifer, C, Hauser, U, Wieland, E, et al. Diagnostic performance of rapid antigen testing for SARS-CoV-2: the COVid-19 anti gen (COVAG) study. Front Med. (2022) 9:9. doi: 10.3389/fmed.2022.774550

PubMed Abstract | Crossref Full Text | Google Scholar

14. Dinnes, J, Deeks, JJ, Berhane, S, Taylor, M, Adriano, A, Davenport, C, et al. Rapid, point-of-care antigen tests for diagnosis of SARS-CoV-2 infection. Cochrane Database Syst Rev. (2021) 2022:CD013705. doi: 10.1002/14651858.cd013705.pub2

PubMed Abstract | Crossref Full Text | Google Scholar

15. Kost, GJ. The impact of increasing disease prevalence, false omissions, and diagnostic uncertainty on coronavirus disease 2019 (COVID-19) test performance. Arch Pathol Lab Med. (2021) 145:797–813. doi: 10.5858/arpa.2020-0716-sa

PubMed Abstract | Crossref Full Text | Google Scholar

16. World Health Organization. (2021). Antigen-detection in the diagnosis of SARS-CoV-2 infection: Interim guidance, 6 October 2021. Available at: https://apps.who.int/iris/handle/10665/345948

Google Scholar

17. European Centre for Disease Prevention and Control. (2021). Options for the use of rapid antigen detection tests for COVID-19 in the EU/EEA – first update. Available at: https://www.ecdc.europa.eu/en/publications-data/options-use-rapid-antigen-tests-covid-19-eueea-first-update

Google Scholar

18. Scheiblauer, H, Filomena, A, Nitsche, A, Puyskens, A, Corman, VM, Drosten, C, et al. Comparative sensitivity evaluation for 122 CE-marked rapid diagnostic tests for SARS-CoV-2 antigen, Germany, September 2020 to April 2021. Euro Surveill. (2021) 26:2100441. doi: 10.2807/1560-7917.es.2021.26.44.2100441

Crossref Full Text | Google Scholar

19. Abbott. (2022) PANBIO™ COVID-19 Ag rapid test device. Available at: https://www.globalpointofcare.abbott/de/product-details/panbio-covid-19-ag-antigen-test.html (Accessed October 26, 2022)

Google Scholar

20. Roche. (2022) SARS-CoV-2 rapid antigen test instructions to use. Available at: https://assets.cwp.roche.com/f/94122/x/074742adcc/packungsbeilage_sars-cov-2rapidantigentest_023_de_202104.pdf (Accessed October 26, 2022)

Google Scholar

21. Singanayagam, A, Patel, M, Charlett, A, Bernal, JL, Saliba, V, Ellis, J, et al. Duration of infectiousness and correlation with RT-PCR cycle threshold values in cases of COVID-19, England, January to May 2020. Euro Surveill. (2020) 25:2001483. doi: 10.2807/1560-7917.es.2020.25.32.2001483

Crossref Full Text | Google Scholar

22. Hakki, S, Zhou, J, Jonnerby, J, Singanayagam, A, Barnett, JL, Madon, KJ, et al. Onset and window of SARS-CoV-2 infectiousness and temporal correlation with symptom onset: a prospective, longitudinal, community cohort study. Lancet Respir Med. (2022) 10:1061–73. doi: 10.1016/s2213-2600(22)00226-0

PubMed Abstract | Crossref Full Text | Google Scholar

23. Korenkov, M, Poopalasingam, N, Madler, M, Vanshylla, K, Eggeling, R, Wirtz, M, et al. Evaluation of a rapid antigen test to detect SARS-CoV-2 infection and identify potentially infectious individuals. J Clin Microbiol. (2021) 59:e0089621. doi: 10.1128/jcm.00896-21

PubMed Abstract | Crossref Full Text | Google Scholar

24. World Health Organization. Antigen-detection in the diagnosis of SARS-CoV-2 infection. Interim Guidance. (2021) 6:2021.

Google Scholar

25. Loacker, L, Dlaska, M, Griesmacher, A, and Anliker, M. Ct values of different SARS CoV2 variants: a single center observational study from Innsbruck. Austria Clin Chem Lab Med. (2022) 60:e225–7. doi: 10.1515/cclm-2022-0542

PubMed Abstract | Crossref Full Text | Google Scholar

26. Yuasa, S, Nakajima, J, Takatsuki, Y, Takahashi, Y, Tani-Sassa, C, Iwasaki, Y, et al. Viral load of SARS-CoV-2 omicron is not high despite its high infectivity. J Med Virol. (2022) 94:5543–6. doi: 10.1002/jmv.27974

PubMed Abstract | Crossref Full Text | Google Scholar

27. Soni, A, Herbert, C, Filippaios, A, Broach, J, Colubri, A, Fahey, N, et al. Comparison of rapid antigen tests’ performance between delta and omicron variants of SARS-CoV-2: a secondary analysis from a serial home self-testing study. Ann Intern Med. (2022) 175:1685–92. doi: 10.7326/m22-0760

PubMed Abstract | Crossref Full Text | Google Scholar

28. Sterlin, D, Mathian, A, Miyara, M, Mohr, A, Anna, F, Claër, L, et al. IgA dominates the early neutralizing antibody response to SARS-CoV-2. Sci Transl Med. (2021) 13:2223. doi: 10.1126/scitranslmed.abd2223

PubMed Abstract | Crossref Full Text | Google Scholar

29. Sheikh-Mohamed, S, Isho, B, Chao, GYC, Zuo, M, Cohen, C, Lustig, Y, et al. Systemic and mucosal IgA responses are variably induced in response to SARS-CoV-2 mRNA vaccination and are associated with protection against subsequent infection. Mucosal Immunol. (2022) 15:799–808. doi: 10.1038/s41385-022-00511-0

PubMed Abstract | Crossref Full Text | Google Scholar

30. Montague, BT, Wipperman, MF, Chio, E, Crow, R, Hooper, AT, O’Brien, MP, et al. Elevated serum IgA following vaccination against SARS-CoV-2 in a cohort of high-risk first responders. Sci Rep. (2022) 12:14932. doi: 10.1038/s41598-022-19095-7

PubMed Abstract | Crossref Full Text | Google Scholar

31. Platten, M, Hoffmann, D, Grosser, R, Wisplinghoff, F, Wisplinghoff, H, Wiesmüller, G, et al. SARS-CoV-2, CT-values, and infectivity—conclusions to be drawn from side observations. Viruses. (2021) 13:1459. doi: 10.3390/v13081459

PubMed Abstract | Crossref Full Text | Google Scholar

32. Wu, S, Archuleta, S, Ming, LS, Somani, J, Chye, QS, and Fisher, D. Serial antigen rapid testing in staff of a large acute hospital. Lancet Infect Dis. (2022) 22:14–5. doi: 10.1016/s1473-3099(21)00723-4

PubMed Abstract | Crossref Full Text | Google Scholar