Prognostic Accuracy of Early Warning Scores for Clinical Deterioration in Patients With COVID-19

Background: Early Warning Scores (EWS), including the National Early Warning Score 2 (NEWS2) and Modified NEWS (NEWS-C), have been recommended for triage decision in patients with COVID-19. However, the effectiveness of these EWS in COVID-19 has not been fully validated. The study aimed to investigate the predictive value of EWS to detect clinical deterioration in patients with COVID-19.

Methods: Between February 7, 2020 and February 17, 2020, patients confirmed with COVID-19 were screened for this study. The outcomes were early deterioration of respiratory function (EDRF) and need for intensive respiratory support (IRS) during the treatment process. The EDRF was defined as changes in the respiratory component of the sequential organ failure assessment (SOFA) score at day 3 (ΔSOFA_resp = SOFA _resp at day 3–SOFA_resp on admission), in which the positive value reflects clinical deterioration. The IRS was defined as the use of high flow nasal cannula oxygen therapy, noninvasive or invasive mechanical ventilation. The performances of EWS including NEWS, NEWS 2, NEWS-C, Modified Early Warning Scores (MEWS), Hamilton Early Warning Scores (HEWS), and quick sepsis-related organ failure assessment (qSOFA) for predicting EDRF and IRS were compared using the area under the receiver operating characteristic curve (AUROC).

Results: A total of 116 patients were included in this study. Of them, 27 patients (23.3%) developed EDRF and 24 patients (20.7%) required IRS. Among these EWS, NEWS-C was the most accurate scoring system for predicting EDRF [AUROC 0.79 (95% CI, 0.69–0.89)] and IRS [AUROC 0.89 (95% CI, 0.82–0.96)], while NEWS 2 had the lowest accuracy in predicting EDRF [AUROC 0.59 (95% CI, 0.46–0.720)] and IRS [AUROC 0.69 (95% CI, 0.57–0.81)]. A NEWS-C ≥ 9 had a sensitivity of 59.3% and a specificity of 85.4% for predicting EDRF. For predicting IRS, a NEWS-C ≥ 9 had a sensitivity of 75% and a specificity of 88%.

Conclusions: The NEWS-C was the most accurate scoring system among common EWS to identify patients with COVID-19 at risk for EDRF and need for IRS. The NEWS-C could be recommended as an early triage tool for patients with COVID-19.

Introduction

The outbreak of coronavirus disease 2019 (COVID-19) has recently become a public health emergency of international concern (1). A novel coronavirus, termed as severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), was isolated as the pathogen of COVID-19 (2). As of October 18, 2020, there have been more than 40 million confirmed COVID-19 cases and 1.1 million deaths globally from World Health Organization reports.

With a sharp increase in the number of cases and limited medical resources, healthcare systems worldwide are facing unprecedented challenges (3). Although the majority of patients with COVID-19 have mild symptoms, patients with advanced age and chronic comorbidities such as hypertension tend to have poor outcomes (4, 5). Patients infected with SARS-CoV-2 tend to get worse from illness onset with a median duration of 7 days, in which severe type may deteriorate to acute respiratory distress syndrome (ARDS) or multiple organ failure (6, 7). During the outbreak of the COVID-19 crisis, early and quick recognition of patients who are at high risk of clinical deterioration would therefore be significantly important (8). A severity-based approach is urgently needed for triaging high risk patients with COVID-19 (9).

The Early Warning Scores (EWS) are a variety of physiologic scoring systems widely used in the world. These systems are based on bedside indices that can be obtained easily and rapidly such as heart rate, respiratory rate, systolic blood pressure, and peripheral oxygen saturation (SpO₂), allowing quick and accurate identify patients at high risk of clinical deterioration. Now, various EWS were developed for early recognition of clinical deterioration. The National Early Warning Score (NEWS), the most common EWS, was initially recommended by the Royal College of Physicians (RCP) (10). It has been proved that NEWS was associated with ICU admission and death outside ICU (11, 12). Its updated version, the National Early Warning Score 2 (NEWS2), with a new SpO₂ scoring scale, was published by the RCP in 2017 to improve prediction for clinical deterioration in patients with a hypercapnic respiratory failure (13). Other versions of EWS such as the Modified Early Warning Score (MEWS) (14, 15) and Hamilton Early Warning Score (HEWS) (16) have also been recently developed to improve the early recognition of hospitalized patients at risk for deterioration, with a significant degree of variation in the clinical variables and the weightings assigned.

Currently, guidelines from the RCP recommends the use of the NEWS2 for initial assessment in patients with COVID-19 (17). Moreover, NEWS-C, a new version of modified NEWS, has also been recommended for triage decisions in patients with COVID-19 (18, 19). However, the recommendations were only based on expert opinions and have not been fully validated in COVID-19 patients.

In this study, we aimed to compare the performance of EWS including NEWS, NEWS2, NEWS-C, HEWS, MEWS, and quick sepsis-related organ failure assessment (qSOFA) to predict early deterioration of respiratory function (EDRF) and the need for intensive respiratory support (IRS) in patients with COVID-19.

Materials and Methods

Study Design and Participants

The study was approved by the Ethics Commission of Renmin Hospital of Wuhan University (WDRY2020-K048) and was conducted in accordance with the amended Declaration of Helsinki. Written informed consent was waived by the Ethics Commission in the setting of COVID-19 crisis in Wuhan.

Patients with age ≥18 years and confirmed COVID-19 admitted between February 7, 2020 and February 17, 2020 were screened in our study. We excluded patients for pregnancy, death within 48 h of admission, or having a Do Not Resuscitate order. COVID-19 was diagnosed by the real-time RT-PCR method on nasal or pharyngeal swab specimens.

Date Collection

Baseline demographics, clinical characteristics on hospital arrival including symptoms, vital signs, and oxygen therapy, laboratory findings, treatments, and outcomes were prospectively collected by two trained reviewers. The NEWS, NEWS-C, NEWS2, HEWS, MEWS, and qSOFA were calculated based on the demographic and clinical characteristics of each patient (Supplementary Table 1).

Outcome Assessment

Respiratory function was assessed according to the respiratory component of the sequential organ failure assessment score (SOFA_resp). The EDRF was defined as a positive change in respiratory function at day 3 (ΔSOFA_resp = SOFA_resp at day 3–SOFA_resp on admission). The positive value of ΔSOFA_resp reflects clinical deterioration. The IRS was defined as the use of high flow nasal cannula oxygen therapy, noninvasive or invasive mechanical ventilation. IRS was considered if the patients met the following criteria: a respiratory rate of ≥30 breaths per minute and arterial oxygen saturation (SaO₂) ≤ 93% or a ratio of the partial pressure of arterial oxygen (PaO₂) to inspired oxygen (FiO₂) of 300 mmHg or less while the patient was receiving oxygen therapy of ≥10 L/min for at least 15 min. The choices of respiratory support method (high flow nasal cannula oxygen therapy, noninvasive or invasive mechanical ventilation) were at the discretion of the attending clinicians.

Statistical Analysis

Data distribution was evaluated using the Kolmogorov-Smirnov test. Continuous variables were expressed as mean ± standard deviation or median interquartile range as appropriate. Categorical variables were expressed as frequencies and percentages. Baseline data were compared using the Student's t-test or the Mann-Whitney U test for continuous variables and the chi-square test or Fisher's exact test for categorical variables. Receiver operating characteristic (ROC) curves were constructed to assess the performance of EWS, and the optimal cut-off values were calculated by the Youden index. The area under the receiver operating characteristic curve (AUROC) were compared by the method described by Hanley and McNeil (20). All statistical analyses were performed using the SPSS software package, version 13.0 (SPSS, Inc., Chicago, IL, USA) and MedCalc software 15.0 (MedCalc Software Ltd, Ostend, Belgium). A two-tailed P value of <0.05 was considered statistically significant.

Results

Patient Characteristics

Between February 7, 2020 and February 17, 2020, a total of 123 patients with COVID-19 were screened for inclusion. Of these patients, seven patients were excluded, including one patient with pregnancy, three patients who died within 48 h after admission, and three patients with DNR order. Finally, 116 patients were included for this study.

The baseline characteristics were shown in Table 1. Of 116 patients, the median age was 63 [IQR 51, 72] years and 47.4% were men. Fever was the most common symptom (86.2%), followed by fatigue (85.3%), cough (69.0%), and dyspnea (56.9%). The baseline NEWS, NEWS-C, NEWS2, HEWS, MEWS, and qSOFA at admission were 5 [3, 7], 6 [5, 9], 6 [5, 8], 3 [2, 5], 2 [2, 3], and 1 [0, 1], respectively. The distributions of all patients by NEWS, NEWS-C, NEWS2, HEWS, MEWS, and qSOFA at admission were presented in Figure 1.

TABLE 1

Table 1. Clinical characteristics of COVID-19 patients.

FIGURE 1

Figure 1. Distribution of patients by NEWS (A), NEWS-C (B), NEWS2 (C), HEWS (D), MEWS (E), and qSOFA (F). HEWS, Hamilton Early Warning Score; MEWS, Modified Early Warning Score; NEWS, National Early Warning Score; NEWS-C, modified NEWS; NEWS 2, National Early Warning Score 2; qSOFA, quick Sepsis-related Organ Failure Assessment.

The baseline PaO₂/FiO₂ on admission was 292 [245, 326] mmHg. At day 3, the median PaO₂/FiO₂ was 314 [241, 330] mmHg. A total of 27 (23.3%) patients developed EDRF according to the ΔSOFA_resp (SOFA_resp at day 3–baseline SOFA_resp), in which a positive value reflected clinical deterioration. Patients with EDRF tended to be older and had a higher rate of hypertension than those without EDRF (all P < 0.01). Compared with the patients without EDRF, the patients with EDRF have higher proportions of dyspnea (88.9 vs. 47.2%; P < 0.001) and higher respiratory rate [28 (21, 22) vs. 22 (20, 23) breaths/minute; P < 0.01] but lower baseline PaO₂/FiO₂ value [245 (167, 303) vs. 305 (272, 328) mmHg; P < 0.001]. On admission, patients with EDRF had higher NEWS, NEWS-C, HEWS, MEWS, and qSOFA than non-EDRF patients (all P < 0.05; Table 1). However, the NEWS2 between patients with EDRF and non-EDRF was comparable (P = 0.16; Table 1).

A total of 24 patients (20.7%) needed IRS during the period of hospital stay. Patients with IRS also tended to be older and had a higher rate of hypertension than those without IRS (all P < 0.05). Compared with the patients without IRS, the patients requiring IRS have higher proportions of dyspnea (91.7 vs. 47.8%; P < 0.001) and a higher respiratory rate [32 (24, 25) vs. 22 (20, 26); P < 0.001] but a lower baseline PaO₂/FiO₂ value [196 (150, 260) vs. 305 (274, 329); P < 0.001]. Patients with IRS had a higher NEWS, NEWS-C, NEWS2, HEWS, MEWS, and qSOFA than non-IRS patients (all P < 0.001; Table 1). A total of 20 patients (17.2%) developed both IRS and EDRF in this cohort. The hospital mortality rate was 7.8%. The mortality in patients with EDRF or IRS was higher than those without EDRF or IRS (all P < 0.001).

Performance of EWS for Clinical Deterioration

To assess the utility of EWS to predict EDRF and need for IRS, the ROC curves were constructed and the AUROCs were calculated (Figure 2). Table 2 listed AUROC, optimal cutoff value, sensitivity, specificity, and positive and negative predictive values of EWS.

FIGURE 2

Figure 2. The receiver operating characteristic curves of early warning scores for clinical deterioration. (A) EDRF; (B) IRS. AUC, area under the curve; HEWS, Hamilton Early Warning Score; MEWS, Modified Early Warning Score; NEWS, National Early Warning Score; NEWS-C, modified NEWS; NEWS 2, National Early Warning Score 2; qSOFA, quick Sepsis-related Organ Failure Assessment; EDRF, early deterioration of respiratory function; IRS, intensive respiratory support.

TABLE 2

Table 2. Performance of early warning scores in predicting clinical deterioration.

Among these EWS, NEWS-C was the most accurate scoring system for predicting EDRF {AUROC 0.79 [95% confidence interval (CI), 0.69–0.89]}. The AUROC of the NEWS-C for predicting EDRF was much higher than that for NEWS2 (0.59, 95% CI 0.46–0.72; P < 0.001) and qSOFA (0.62, 95% CI 0.51–0.74; P < 0.001). The AUROC of NEWS-C for predicting EDRF was also larger than NEWS (0.73, 95% CI 0.62–0.84), HEWS (0.75, 95% CI 0.63–0.86), and MEWS (0.71, 95% CI 0.59–0.83), although the difference is not statistically significant (Table 3). A NEWS-C ≥ 9 had a sensitivity of 59.3% and a specificity of 85.4% for predicting EDRF.

TABLE 3

Table 3. The cross-comparisons between AUROCs of early warning scores for predicting EDRF.

Among these EWS, NEWS-C was the most accurate scoring system for predicting IRS [AUROC 0.89 (95% CI, 0.82–0.96)]. The AUROC of the NEWS-C for predicting IRS was much higher than that for NEWS2 (0.69, 95% CI 0.57–0.81; P < 0.001), MEWS (0.80, 95% CI 0.70–0.90; P = 0.03), and qSOFA (0.72, 95% CI 0.61–0.82; P < 0.001). The AUROC of NEWS-C for predicting IRS was also higher than NEWS (0.86, 95% CI 0.78–0.95) and HEWS (0.87, 95% CI 0.77–0.97), although the difference is not statistically significant (Table 4). For predicting IRS, a NEWS-C ≥ 9 had a sensitivity of 75% and a specificity of 88%.

TABLE 4

Table 4. The cross-comparisons between AUROCs of Early Warning Scores for predicting IRS.

Discussion

Until now, there are limited studies to evaluate the predictive value of EWS in patients with COVID-19. In the current study, we found that the NEWS-C was the most accurate scoring system among common EWS for predicting EDRF and IRS in patients with COVID-19. On the contrary, NEWS 2 had the lowest accuracy for predicting both outcomes.

EWS have been developed and widely used around the world for early recognition of clinical deterioration (21). The NEWS, endorsed by RCP, is already used for predicting deterioration in many hospitals across the United Kingdom (10). It is reported that the NEWS has better performance than other EWS to identify patients at risk of ICU admission and mortality (12). Moreover, the NEWS was more accurate in predicting clinical deterioration than qSOFA in infected patients outside the ICU (24).

The NEWS 2, updated version of NEWS, was recommended by RCP in 2017. The new SpO₂ scoring scale in NEWS 2, with a lower SpO₂ threshold than NEWS, was implemented to avoid over-use of supplemental oxygen and facilitate management in hypercapnic patients (27). Recently, the NEWS 2 has been recommended for predicting clinical deterioration in patients with COVID-19 (17). In our study, NEWS 2 had a lower performance than NEWS in predicting EDRF and IRS. This is in accordance with previous study, which demonstrated that NEWS 2 did not predict clinical outcome in elderly patients with COVID-19 (28). The possible reasons were as follows: (1) the incidence of type II respiratory failure in patients with COVID-19 was low in this study; and (2) the NEWS2 modifications to NEWS may not improve discrimination of poor outcome in hospital patients including those with type II respiratory failure (26). Therefore, NEWS 2 may be inappropriate for triage decision in patients with COVID-19.

A modified NEWS, termed NEWS-C, has also been recommended for triage decision in patients with COVID-19 (18, 19). The MEWS (14) and HEWS (16) have been developed to early identify clinical deterioration in generally hospitalized patients, with a significant degree of variation in the clinical variables and the weightings assigned. In this study, the NEWS-C had largest AUROC for predicting EDRF and IRS in these EWS. NEWS-C modifications to EWS added an age ≥65 years as an independent component. Several studies have showed that old aging was independently associated with mortality in patients with COVID-19 (4, 23). Therefore, it may offer better predictive performance than other EWS.

qSOFA, consisting of three clinical variables (mental status, respiratory rate, and blood pressure), has been proposed as a rapid screening tool for infected patients (29). The effectiveness of the qSOFA has been validated in various heterogeneous sepsis patients (30, 31). Recently, several reports have demonstrated that the qSOFA can accurately assess the severity of community-acquired pneumonia (32–34). However, qSOFA had a lower performance in predicting clinical deterioration compared with other EWS in our study. This may be partially explained by the low percentage of hypotension and alter mental status in this cohort. The finding was also consistent with previous studies, in which qSOFA may not be appropriate to identify critically ill patients with COVID-19 (22, 35).

This study had several limitations. First, this was a retrospective study with relatively small sample size. A larger cohort validation is still required. Second, the changes in EWS during the treatment process was not recorded in this study. Third, as the intubation rate and mortality in this population were lower than that in critically ill patients (25, 35, 36), caution must be taken in extrapolating the results of the study for critically ill patients. Additional assessments of organ dysfunction should be required in critically ill patients with COVID-19.

Conclusion

The NEWS-C was the most accurate scoring system among common EWS to identify patients with COVID-19 at risk for EDRF and need for IRS. The NEWS-C could be recommended as an early triage tool for COVID-19.

Data Availability Statement

The original contributions presented in the study are included in the article/Supplementary Materials, further inquiries can be directed to the corresponding author/s.

Ethics Statement

The studies involving human participants were reviewed and approved by Ethics Commission of Renmin Hospital of Wuhan University. The ethics committee waived the requirement of written informed consent for participation.

Author Contributions

YS, G-wT, and ZL: conception and design. M-jJ, R-cX, and ZL: administrative support. YS, M-jJ, S-jY, J-lZ, and G-gM: provision of study materials or patients. YS, S-jY, J-lZ, KL, R-cX, and G-gM: collection and assembly of data. YS, G-wT, R-cX, M-jJ, and ZL: data analysis and interpretation. All authors: manuscript writing and final approval of manuscript.

Funding

This article was supported by grants from the Research Funds of Zhongshan Hospital (2019ZSQN13 and XYYX201922), the Research Funds of Shanghai Municipal Health Commission (2019ZB0105), Natural Science Foundation of Shanghai (20ZR1411100), Program of Shanghai Academic/Technology Research Leader (20XD1421000), National Natural Science Foundation of China (82070085), Clinical Research Funds of Zhongshan Hospital (2020ZSLC38 and 2020ZSLC27), and Smart Medical Care of Zhongshan Hospital (2020ZHZS01).

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Acknowledgments

The authors appreciate the great sacrifice and contribution of the people from Wuhan, Hubei province, China.

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fmed.2020.624255/full#supplementary-material

References

1. Sahu KK, Mishra AK, Lal A. Trajectory of the COVID-19 pandemic: chasing a moving target. Ann Transl Med. (2020) 8:694. doi: 10.21037/atm-20-2793

CrossRef Full Text | Google Scholar

2. Zhu N, Zhang D, Wang W, Li X, Yang B, Song J, et al. A novel coronavirus from patients with pneumonia in China, 2019. N Engl J Med. (2020) 382:727–33. doi: 10.1056/NEJMoa2001017

CrossRef Full Text | Google Scholar

3. Tonetti T, Grasselli G, Zanella A, Pizzilli G, Fumagalli R, Piva S, et al. Use of critical care resources during the first 2 weeks (February 24-March 8, 2020) of the Covid-19 outbreak in Italy. Ann Intensive Care. (2020) 10:133. doi: 10.1186/s13613-020-00750-z

CrossRef Full Text | Google Scholar

4. Wu C, Chen X, Cai Y, Xia J, Zhou X, Xu S, et al. Risk factors associated with acute respiratory distress syndrome and death in patients with coronavirus disease 2019 pneumonia in Wuhan, China. JAMA Intern Med. (2020) 180:934–43. doi: 10.1001/jamainternmed.2020.0994

PubMed Abstract | CrossRef Full Text | Google Scholar

5. Yang S, Cao P, Du P, Wu Z, Zhuang Z, Yang L, et al. Early estimation of the case fatality rate of COVID-19 in mainland China: a data-driven analysis. Ann Transl Med. (2020) 8:128. doi: 10.21037/atm.2020.02.66

PubMed Abstract | CrossRef Full Text | Google Scholar

6. Chen N, Zhou M, Dong X, Qu J, Gong F, Han Y, et al. Epidemiological and clinical characteristics of 99 cases of 2019 novel coronavirus pneumonia in Wuhan, China: a descriptive study. Lancet. (2020) 395:507–13. doi: 10.1016/S0140-6736(20)30211-7

PubMed Abstract | CrossRef Full Text | Google Scholar

7. Zhou F, Yu T, Du R, Fan G, Liu Y, Liu Z, et al. Clinical course and risk factors for mortality of adult inpatients with COVID-19 in Wuhan, China: a retrospective cohort study. Lancet. (2020) 395:1054–62. doi: 10.1016/S0140-6736(20)30566-3

CrossRef Full Text | Google Scholar

8. Sun Q, Qiu H, Huang M, Yang Y. Lower mortality of COVID-19 by early recognition and intervention: experience from Jiangsu Province. Ann Intensive Care. (2020) 10:33. doi: 10.1186/s13613-020-00650-2

PubMed Abstract | CrossRef Full Text

9. Shang Y, Pan C, Yang X, Zhong M, Shang X, Wu Z, et al. Management of critically ill patients with COVID-19 in ICU: statement from front-line intensive care experts in Wuhan, China. Ann Intensive Care. (2020) 10:73. doi: 10.1186/s13613-020-00689-1

PubMed Abstract | CrossRef Full Text | Google Scholar

10. McGinley A, Pearse RM. A national early warning score for acutely ill patients. BMJ. (2012) 345:e5310. doi: 10.1136/bmj.e5310

PubMed Abstract | CrossRef Full Text | Google Scholar

11. Nannan PR, Minderhoud TC, Alam N, Nanayakkara P. Prognostic value of early warning scores in the emergency department (ED) and acute medical unit (AMU): a narrative review. Eur J Intern Med. (2017) 45:20–31. doi: 10.1016/j.ejim.2017.09.027

PubMed Abstract | CrossRef Full Text | Google Scholar

12. Smith GB, Prytherch DR, Meredith P, Schmidt PE, Featherstone PI. The ability of the national early warning score (NEWS) to discriminate patients at risk of early cardiac arrest, unanticipated intensive care unit admission, and death. Resuscitation. (2013) 84:465–70. doi: 10.1016/j.resuscitation.2012.12.016

PubMed Abstract | CrossRef Full Text | Google Scholar

13. Martin-Rodriguez F, Lopez-Izquierdo R, Del PVC, Sanchez-Soberon I, Delgado-Benito JF, Martin-Conty JL, et al. Can the prehospital national early warning score 2 identify patients at risk of in-hospital early mortality? A prospective, multicenter cohort study. Heart Lung. (2020) 49:585–91. doi: 10.1016/j.hrtlng.2020.02.047

PubMed Abstract | CrossRef Full Text | Google Scholar

14. Xie X, Huang W, Liu Q, Tan W, Pan L, Wang L, et al. Prognostic value of modified early warning score generated in a Chinese emergency department: a prospective cohort study. BMJ Open. (2018) 8:e24120. doi: 10.1136/bmjopen-2018-024120

PubMed Abstract | CrossRef Full Text | Google Scholar

15. Yu M, Huang B, Liu P, Wang A, Ding W, Zhai Y, et al. Detection of deteriorating patients after Whipple surgery by a modified early warning score (MEWS). Ann Transl Med. (2019) 7:574. doi: 10.21037/atm.2019.09.24

PubMed Abstract | CrossRef Full Text | Google Scholar

16. Skitch S, Tam B, Xu M, McInnis L, Vu A, Fox-Robichaud A. Examining the utility of the Hamilton early warning scores (HEWS) at triage: retrospective pilot study in a Canadian emergency department. CJEM. (2018) 20:266–74. doi: 10.1017/cem.2017.21

PubMed Abstract | CrossRef Full Text | Google Scholar

17. Royal College of Physicians. NEWS2 and Deterioration in COVID-19. Available online at: https://www.rcplondon.ac.uk/news/news2-and-deterioration-covid-19

18. Liao X, Wang B, Kang Y. Novel coronavirus infection during the 2019-2020 epidemic: preparing intensive care units-the experience in Sichuan Province, China. Intensive Care Med. (2020) 46:357–60. doi: 10.1007/s00134-020-05954-2

PubMed Abstract | CrossRef Full Text | Google Scholar

19. Swiss Society of Intensive Care Medicine. Recommendations for the admission of patients with COVID-19 to intensive care and intermediate care units (ICUs and IMCUs). Swiss Med Wkly. (2020) 150:w20227. doi: 10.4414/smw.2020.20227

PubMed Abstract | CrossRef Full Text | Google Scholar

20. Hanley JA, McNeil BJ. A method of comparing the areas under receiver operating characteristic curves derived from the same cases. Radiology. (1983) 148:839–43. doi: 10.1148/radiology.148.3.6878708

PubMed Abstract | CrossRef Full Text | Google Scholar

21. Churpek MM, Yuen TC, Edelson DP. Risk stratification of hospitalized patients on the wards. Chest. (2013) 143:1758–65. doi: 10.1378/chest.12-1605

PubMed Abstract | CrossRef Full Text | Google Scholar

22. Jang JG, Hur J, Hong KS, Lee W, Ahn JH. Prognostic accuracy of the SIRS, qSOFA, and NEWS for early detection of clinical deterioration in SARS-CoV-2 infected patients. J Korean Med Sci. (2020) 35:e234. doi: 10.3346/jkms.2020.35.e234

CrossRef Full Text | Google Scholar

23. Feng Y, Ling Y, Bai T, Xie Y, Huang J, Li J, et al. COVID-19 with different severity: a multi-center study of clinical features. Am J Respir Crit Care Med. (2020) 201:1380–8. doi: 10.1164/rccm.202002-0445OC

PubMed Abstract | CrossRef Full Text | Google Scholar

24. Churpek MM, Snyder A, Han X, Sokol S, Pettit N, Howell MD, et al. Quick sepsis-related organ failure assessment, systemic inflammatory response syndrome, and early warning scores for detecting clinical deterioration in infected patients outside the intensive care unit. Am J Respir Crit Care Med. (2017) 195:906–11. doi: 10.1164/rccm.201604-0854OC

PubMed Abstract | CrossRef Full Text | Google Scholar

25. Richardson S, Hirsch JS, Narasimhan M, Crawford JM, McGinn T, Davidson KW, et al. Presenting characteristics, comorbidities, and outcomes among 5700 patients hospitalized with COVID-19 in the New York City Area. JAMA. (2020) 23:2052–9. doi: 10.1001/jama.2020.6775

CrossRef Full Text | Google Scholar

26. Pimentel M, Redfern OC, Gerry S, Collins GS, Malycha J, Prytherch D, et al. A comparison of the ability of the national early warning score and the national early warning score 2 to identify patients at risk of in-hospital mortality: a multi-centre database study. Resuscitation. (2019) 134:147–56. doi: 10.1016/j.resuscitation.2018.09.026

PubMed Abstract | CrossRef Full Text | Google Scholar

27. Smith GB, Redfern OC, Pimentel MA, Gerry S, Collins GS, Malycha J, et al. The national early warning score 2 (NEWS2). Clin Med. (2019) 19:260. doi: 10.7861/clinmedicine.19-3-260

PubMed Abstract | CrossRef Full Text | Google Scholar

28. Sze S, Pan D, Williams C, Wong N, Sahota A, Bell D, et al. Letter to the editor: variability but not admission or trends in NEWS2 score predicts clinical outcome in elderly hospitalised patients with COVID-19. J Infect. (2020) S0163–4453:30341–8. doi: 10.1016/j.jinf.2020.05.063

CrossRef Full Text | Google Scholar

29. Singer M, Deutschman CS, Seymour CW, Shankar-Hari M, Annane D, Bauer M, et al. The third international consensus definitions for sepsis and septic shock (sepsis-3). JAMA. (2016) 315:801–10. doi: 10.1001/jama.2016.0287

CrossRef Full Text | Google Scholar

30. Shankar-Hari M, Phillips GS, Levy ML, Seymour CW, Liu VX, Deutschman CS, et al. Developing a new definition and assessing new clinical criteria for septic shock: for the third international consensus definitions for sepsis and septic shock (Sepsis-3). JAMA. (2016) 315:775–87. doi: 10.1001/jama.2016.0289

CrossRef Full Text | Google Scholar

31. Raith EP, Udy AA, Bailey M, McGloughlin S, MacIsaac C, Bellomo R, et al. Prognostic accuracy of the SOFA score, SIRS criteria, and qSOFA score for in-hospital mortality among adults with suspected infection admitted to the intensive care unit. JAMA. (2017) 317:290–300. doi: 10.1001/jama.2016.20328

PubMed Abstract | CrossRef Full Text | Google Scholar

32. Asai N, Watanabe H, Shiota A, Kato H, Sakanashi D, Hagihara M, et al. Efficacy and accuracy of qSOFA and SOFA scores as prognostic tools for community-acquired and healthcare-associated pneumonia. Int J Infect Dis. (2019) 84:89–96. doi: 10.1016/j.ijid.2019.04.020

PubMed Abstract | CrossRef Full Text | Google Scholar

33. Kolditz M, Scherag A, Rohde G, Ewig S, Welte T, Pletz M. Comparison of the qSOFA and CRB-65 for risk prediction in patients with community-acquired pneumonia. Intensive Care Med. (2016) 42:2108–10. doi: 10.1007/s00134-016-4517-y

PubMed Abstract | CrossRef Full Text | Google Scholar

34. Chen YX, Wang JY, Guo SB. Use of CRB-65 and quick sepsis-related organ failure assessment to predict site of care and mortality in pneumonia patients in the emergency department: a retrospective study. Crit Care. (2016) 20:167. doi: 10.1186/s13054-016-1351-0

PubMed Abstract | CrossRef Full Text | Google Scholar

35. Ferreira M, Blin T, Collercandy N, Szychowiak P, Dequin PF, Jouan Y, et al. Critically ill SARS-CoV-2-infected patients are not stratified as sepsis by the qSOFA. Ann Intensive Care. (2020) 10:43. doi: 10.1186/s13613-020-00664-w

PubMed Abstract | CrossRef Full Text | Google Scholar

36. Yang X, Yu Y, Xu J, Shu H, Xia J, Liu H, et al. Clinical course and outcomes of critically ill patients with SARS-CoV-2 pneumonia in Wuhan, China: a single-centered, retrospective, observational study. Lancet Respir Med. (2020) 8:475–81. doi: 10.1016/S2213-2600(20)30079-5

PubMed Abstract | CrossRef Full Text | Google Scholar