Time-to-Death Longitudinal Characterization of Clinical Variables and Longitudinal Prediction of Mortality in COVID-19 Patients: A Two-Center Study

Objectives: To characterize the temporal characteristics of clinical variables with time lock to mortality and build a predictive model of mortality associated with COVID-19 using clinical variables.

Design: Retrospective cohort study of the temporal characteristics of clinical variables with time lock to mortality.

Setting: Stony Brook University Hospital (New York) and Tongji Hospital.

Patients: Patients with confirmed positive for severe acute respiratory syndrome coronavirus-2 using polymerase chain reaction testing. Patients from the Stony Brook University Hospital data were used for training (80%, N = 1,002) and testing (20%, N = 250), and 375 patients from the Tongji Hospital (Wuhan, China) data were used for testing.

Intervention: None.

Measurements and Main Results: Longitudinal clinical variables were analyzed as a function of days from outcome with time-lock-to-day of death (non-survivors) or discharge (survivors). A predictive model using the significant earliest predictors was constructed. Performance was evaluated using receiver operating characteristics area under the curve (AUC). The predictive model found lactate dehydrogenase, lymphocytes, procalcitonin, D-dimer, C-reactive protein, respiratory rate, and white-blood cells to be early predictors of mortality. The AUC for the zero to 9 days prior to outcome were: 0.99, 0.96, 0.94, 0.90, 0.82, 0.75, 0.73, 0.77, 0.79, and 0.73, respectively (Stony Brook Hospital), and 1.0, 0.86, 0.88, 0.96, 0.91, 0.62, 0.67, 0.50, 0.63, and 0.57, respectively (Tongji Hospital). In comparison, prediction performance using hospital admission data was poor (AUC = 0.59). Temporal fluctuations of most clinical variables, indicative of physiological and biochemical instability, were markedly higher in non-survivors compared to survivors (p < 0.001).

Conclusion: This study identified several clinical markers that demonstrated a temporal progression associated with mortality. These variables accurately predicted death within a few days prior to outcome, which provides objective indication that closer monitoring and interventions may be needed to prevent deterioration.

Introduction

Coronavirus disease 2019 (COVID-19) has infected over 39 million people and killed more than 1 million people worldwide (October 18, 2020) (1–3). The widespread outbreaks with recent spikes across the states and the number of recurrences have strained and will continue to strain healthcare resources. There is an urgent need for effective tools for frontline physicians to effectively manage COVID-19 patients and anticipate resource needs under time-sensitive, stressful, and potentially resource-constrained circumstances in this pandemic.

Over a hundred commonly measured clinical variables associated with COVID-19 infection have been identified [see reviews (4–6)] including demographics, clinical signs and symptoms, comorbidities, serial imaging data, serial vital sign data, and serial laboratory blood tests, amongst others. A few studies have used some clinical variables at admission to the emergency department to predict likelihood of critical illness or mortality (7–17). However, patients presenting to hospitals are in various stages of disease severity. Prediction of mortality and other outcomes far downstream using only clinical variables at admission is likely inadequate. There is no consensus as to which clinical variables are the earliest predictors or most predictive of mortality (7–17). This is in part because patients presenting to hospitals are of variable disease severity and treatment of COVID-19 is still evolving. To our knowledge, there has been no systematic evaluation of the temporal characteristics of these clinical variables leading to mortality in COVID-19 and how these temporal characteristics are judiciously used to inform clinical decision making.

The goal of this study was thus to characterize the temporal progression of clinical and laboratory variables in COVID-19 patients with time lock to the day of death or discharge. We compared survivors and non-survivors to determine the earliest predictors of mortality in the disease progression. Based on these data, we then developed a mathematical model to predict mortality at each day prior to outcome using individual and combinations of these clinical predictors. This predictive model was developed and independently tested using data from Stony Brook University Hospital in New York. To extend its generalizability, we further tested this predictive model on an independent COVID-19 patient cohort from Tongji Hospital, Wuhan, China. To our knowledge, this is one of the first longitudinal models to monitor the progression and mortality in COVID-19.

Materials and Methods

Study Population

Our study followed the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD). Data came from two sites: The Stony Brook University Hospital (New York) data were used for training (80%) and testing (20%), while the Tongji Hospital (Wuhan, China) data were used for just testing purposes. This was a retrospective study approved by the Stony Brook University Institution Review Board Office of Research Compliance, approval number IRB2020-00207, and was exempt from informed consent and Health Insurance Portability & Accountability Act (HIPAA) waiver. The inclusion criteria were: (i) patients who were diagnosed by positive tests of real-time polymerase chain reaction (RT-PCR) for severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), and (ii) patients who were hospitalized. Exclusion criteria were: (i) COVID-19 patients who were under 18 years of age, and (ii) patients who were still in the hospital at the time of analysis. The COVID-19 Persons Under Investigation (PUI) registry from the Emergency Department consisted of 5,766 patients from February 7, 2020, and May 4, 2020. Of these patients, 2,594 were confirmed COVID-19 positive cases, of which 1252 were hospitalized. Primary analysis was performed on all hospitalized patients (N = 1,252, 14.5% mortality rate), and secondary analysis was performed on general floor (N = 1001, 8.09% mortality rate) and ICU (N = 251, 40.2% mortality rate) patient groups.

The Tongji Hospital (Wuhan, China) data were obtained from Jan 10, 2020 to Feb 24, 2020 (N = 485, of which 375 had the needed clinical variables) with approval of their institutional review board and waiver of informed consent (18). Of the 375 patients, 201 survived and 174 died (46.4% mortality rate). Similar inclusion and exclusion criteria were applied to these de-identified data. This dataset was used for “testing” only.

Data Collection

The clinical outcome was mortality at discharge. The input variables included demographic information (age, gender, ethnicity, and race), chronic comorbidities (smoking, diabetes, hypertension, asthma, chronic obstructive pulmonary disease, coronary artery disease, heart failure, cancer, immunosuppression, and chronic kidney disease), serial vital signs (heart rate, respiratory rate, pulse oxygen saturation [SpO₂], systolic blood pressure and temperature), and serial laboratory tests (C-reactive protein [CRP], D-dimer, ferritin, lactate dehydrogenase [LDH], lymphocytes, procalcitonin, alanine aminotransferase [ALT], brain natriuretic peptide [BNP], and troponin).

Statistical Analysis and Predictive Modeling

Statistical analysis was performed using SPSS v26 (IBM, Armonk, NY) and SAS v9.4 (SAS Institute, Cary, NC). Group comparisons of categorical variables in frequencies and percentages were performed using the Chi-squared test or Fisher exact test. Group comparison of continuous variables in medians and interquartile ranges (IQR) used the Mann-Whitney U test. For all analyses, a p < 0.05 was considered to be statistically significant with correction for multiple comparisons with the false discovery rate where appropriate.

Clinical variables were analyzed as a function of days from outcome with time-locked to day of death (non-survivors) or discharge (survivors). Clinical variables were compared between groups at each time point with linear mixed models that included demographic information such as sex, age, ethnicity, and comorbidity as covariates. Within-subject correlation was adjusted in the linear mixed models using covariance (i.e., compound symmetric, autoregressive, or unstructured) matrices.

The temporal fluctuation of each clinical variable between groups was calculated by taking within-subject standard deviation across time normalized by mean, excluding the three time points closest to death or discharge to avoid possible spikes closer to the day of death. The medians of within-subject standard deviations were compared between the non-survivors and survivors using the Mann-Whitney U test.

Univariable logistic regression models were first built using individual clinical variables to predict outcomes on each day separately. Prediction performance was evaluated by area under the curve (AUC) of the receiver operating characteristic (ROC) curve. The Stony Brook University Hospital data were split into 80% for training and 20% for testing. The average ROC curve and AUC were obtained with five runs. Using ROC analysis, the top earliest predictors were identified. Instead of calibration measures (e.g., calibration slope), we demonstrated consistency through internal and external validation and systematically selected top variables for prediction. We started with univariable models (single predictor) and evaluated different combinations of 12 variables. We further constructed models using combinations of top predictors that included top three, top five, and top seven clinical variables. AUCs with the top 3, 5, or 7 variables for all three analysis cohorts were analyzed to verify consistency of top predictors across models. These were done for different days prior to outcome separately. For comparison, prediction performance using clinical variables at admission of the same dataset was also computed. In addition, Tongji Hospital data were also used as a “testing” dataset for external validation.

To avoid the potential of overfitting, for training, we performed univariable analysis first and identified 12 laboratory measures and vital signs to build the predictive model to predict mortality. Then we performed variable selection and used top 3, 5, or 7 variables to present results. No more than 10 variables were included in one model at any given time for a cohort of 1,252 patients.

Results

Clinical Variables

Of the 1,252 hospitalized patients (Stony Brook Hospital), 1,070 survived and 182 did not (14.5% mortality rate). Table 1 summarizes the demographics, comorbidities, vital signs, and laboratory data of the survivors and non-survivors. The non-survivor group was older than the survivor group (73 ± 15 vs. 60 ± 17 years of age, p < 0.0001), with more males dying than females (p = 0.021). Ethnicity and race were statistically different between groups (p < 0.05). History of smoking, hypertension, chronic obstructive pulmonary disease (COPD), coronary artery disease, and heart failure were significantly different between groups (p < 0.05, after correction for multiple comparisons). Signs and symptoms such as fever, cough, fatigue, myalgia, nausea or vomiting, and chest discomfort were significantly different between groups (p < 0.05, after correction for multiple comparisons). The non-surviving group had reported greater co-morbidities, it was surprising to find that the surviving cohort reported more signs and symptoms.

TABLE 1

Table 1. All hospitalized patients.

The time course of the clinical variables as a function of days to outcome are shown in Figure 1. LDH, procalcitonin, ferritin, ALT, and SpO₂ of non-survivors changed sharply on the day of or a day prior to death, relative to those of survivors. By contrast, lymphocyte count, CRP, respiratory rate, WBCs, and heart rate showed gradually increasing differences early on prior to death. Unexpectedly though, in the surviving group, ferritin and procalcitonin remained relatively elevated but stable throughout the entire hospitalization until discharge, despite being acute phase reactants. In addition, when looking at CRP and respiratory rate, it was noted that in the surviving cohort the values continued to decrease toward normal values rather than stay elevated like in the mortality cohort. This kind of temporal progression change was not evident in other temporal variables. Overall, there were more temporal fluctuations in the group that passed away as compared to the group that survived. We also separated the temporal characteristics of clinical variables of patients in the ICU cohort (N = 251, 40.2% mortality rate) and the general floor cohort (N = 1,001, 8.09% mortality rate) (Supplementary Figure 1). While there were some differences, the majority of the temporal characteristics leading up to mortality or discharge were overall similar amongst the ICU cohort, general floor cohort and all hospitalized patients, except that LDH, ferritin, and ALT spiked in the ICU cohort but not in the general floor cohort. As was supported by the trend of variables for all hospitalized patients, lymphopenia continued to be worse in the non-surviving cohort in both the ICU and the general floor. CRP and respiratory rate also continued to down-trend throughout hospital stay in the surviving cohort in the ICU and general floor.

FIGURE 1

Figure 1. The time courses of the clinical variables of all hospitalized patients as a function of days to outcome, time lock to the day of death (“Non”: non-survivors) or the day of discharge (“Sur”: survivors). Error bars are SEM. Two rows of numbers are sample sizes. * indicates significant difference with correction of multiple comparison and covariate with sex, age, ethnicity, and comorbidities. BNP and troponin were not analyzed because their sample sizes were small and highly scattered.

The within-subject standard deviations of the clinical variables across time were computed to evaluate temporal fluctuation. For all hospitalized patients, the ratio of temporal fluctuation of non-survivors to survivors for D-dimer, procalcitonin, ferritin, WBCs, LDH, respiratory rate, CRP, SpO₂, heart rate, and systolic blood pressure were 4.78, 3.75, 2.26, 2.15, 1.77, 1.63, 1.58, 1.55, 1.43, and 1.37, respectively (p < 0.001), indicating higher physiological instability amongst non-survivors compared to survivors. Similar results were found when data were separated into general floor and ICU patient groups.

For the Stony Brook Hospital data, the AUC predicting mortality in all hospitalized patient cohort for individual clinical variables at each day are illustrated in Figure 2. Overall, we noticed the trend that the prediction performance, as determined by a higher AUC value, increased the closer we were to the outcome of death, with an AUC of 80–99% in days 0–4 prior to death, and AUC >70% from days 5–10 prior to death, with specificity higher than sensitivity. The earliest predictors that showed high prediction performance by AUC were LDH, lymphocytes, procalcitonin, D-dimer, CRP, respiratory rate, and WBCs. The AUC of combined top 7 predictors from zero to 9 days prior to outcome were: 0.99, 0.96, 0.94, 0.90, 0.82, 0.75, 0.73, 0.77, 0.79, and 0.73, respectively. The corresponding specificity were >0.96 for all zero to 9 days prior to outcome. The corresponding sensitivity was 0.80, 0.73, 0.68, 0.46, 0.34, 0.23, 0.12, 0.37, 0.33, and 0.29 respectively. The moderate to low sensitivity was due to data asymmetry, namely, low mortality rate (14.5%, all hospitalized COVID-19 patients), as expected. The sensitivity of ICU group with 40% mortality was excellent. With more multi-institutional datasets, the model should yield better sensitivity and generalizability.

FIGURE 2

Figure 2. AUC for all hospitalized COVID-19 patients as a function of days to outcome, time lock to the day of death (non-survivors) or the day of discharge (survivors) for individual and top earliest predictors of mortality (“testing” data).

Similar predictive models were built for the general floor cohort and the ICU cohort (Figure 3). The AUCs of top 7 predictors for all hospitalized cohort were better than that of ICU group, which in turn were better than general floor cohort. Prediction performance was consistent for the ICU group because data were more balanced (mortality rate of 40.2%) compared to the general floor group (mortality rate of 8.09%).

FIGURE 3

Figure 3. AUC comparisons of all hospitalized (N = 1252), ICU (N = 251), and general floor (N = 1,001) as a function of days to outcome, time lock to the day of death (non-survivors) or the day of discharge (survivors) for the seven earliest predictors of mortality (20% “testing” data).

To determine the best predictive model, we built and tested predictive models with different combination of top clinical variables. AUC for predicting mortality from top 3 clinical variables (LDH, lymphocytes, and procalcitonin), top 5 clinical variables (LDH, lymphocytes, procalcitonin, D-dimer, and CRP) and top 7 clinical variables (LDH, lymphocytes, procalcitonin, D-dimer, CRP, respiratory rate, and WBCs) are listed in Table 2.

TABLE 2

Table 2. Predictive performance of top 3, 5, and 7 clinical variables for all hospitalization, general floor, and ICU cohorts.

These models were designed by a statistical logistic regression model utilizing ROC analysis to identify which of the clinical variables or combination of clinical variables were the most predictive. It is not a scoring system. As a result, having missing variables like procalcitonin would likely reduce the prediction performance, but does not invalidate the use of these models. Prediction using the top 7 variables performed better than models using the top 5, which performed better than models using the top 3 variables. As a result, if a hospital is not able to collect certain lab values on patients, they could use the top 3 or top 5 variable prediction models, with the knowledge that it would not be as accurate if all seven variables were collected. This model requires further validation using large and multi-institutional dataset to achieve generalizability. With more data, the model should become more accurate.

By comparison, the prediction performance of the top individual clinical variables at admission to the emergency department yielded an AUC ranging from 0.50 to 0.61, and the combined earliest predictors at admission yielded an AUC of 0.59.

Temporal characteristics of clinical variables were also described and compared between non-survivors and survivors for data from the Tongji Hospital, Wuhan, China (N = 375) (Figure 4). The mortality rate of this cohort was 46.4%. Some temporal characteristics were similar, and others were different from the Stony Brook data. In particular, LDH, lymphocytes, CRP and D-dimer showed significant differences between non-survivors and survivors from the onset and these differences were time-invariant. For external validation, a logistic regression model trained on the Stony Brook hospital data and tested on the Tongji Hospital data using the significant variables LDH, lymphocytes, CRP and D-dimer demonstrated the AUC for the zero to 9 days prior to death as follows: 1.0, 0.86, 0.88, 0.96, 0.91, 0.62, 0.67, 0.50, 0.63, and 0.57, respectively.

FIGURE 4

Figure 4. Tongji data: The time courses of the clinical variables as a function of days to outcome, time lock to the day of death (“Non”: non-survivors) or the day of discharge (“Sur”: survivors). Error bars are SEM (N = 375). Two rows of numbers are sample sizes. * indicates significant difference with correction of multiple comparison and covariate with sex and age.

Discussion

With widespread COVID-19 outbreaks, improved understanding of the temporal disease progression can guide prognosis and treatment, as well as anticipate resource needs. This study characterized the temporal progression of clinical variables in COVID-19 patients with time lock to the day of death or discharge. The major findings were: (i) the earliest predictors of mortality were lactate dehydrogenase, lymphocyte count, procalcitonin, D-dimer, C-reactive protein, respiratory rate, and white-blood cells, (ii) there is a down-trending CRP (normal limit <10 mg/L) and respiratory rate (normal limit 12–16 breaths/minute) to normal values in the survival cohort that is not evident in other temporal variables, (iii) the temporal fluctuations of most clinical variables, indicative of physiological and biochemical instability, were markedly higher in non-survivors compared to survivors, (iv) the overall performance of predictive models was better in the days leading up to the day of death, (v) the best predictive models were those using the top 7 variables, followed by the top 5 variables, and then by the top 3 variables, (vi) these predictive models were further tested on data from another hospital, and showed similar performance accuracy, and (vii) by comparison, prediction performance of the top individual clinical variables at hospital admission was poor.

While there are many COVID-19 prediction models (7–17), this study is novel because: (i) our predictive model assesses predictions at multiple time points with time locked to death, and (ii) it evaluated models using all hospitalized, general floor and ICU patients as well as data from another hospital to improve generalizability. This is in contrast to most previous COVID-19 prediction models that lacked external validation and used only admission data (one time point) which is less informative because patients come to hospitals at different degrees of severity. To our knowledge, this is the first study that systematically characterizes the longitudinal progression of commonly measured clinical variables over the course of hospitalization of COVID-19 patients.

Temporal Progression of Clinical Variables

We characterized commonly measured clinical parameters into three categories: the variables that changed early on during the hospital course then gradually worsened (CRP, lymphocyte count, WBCs, respiratory rate, and heart rate), the variables that spiked before death (LDH, procalcitonin, ferritin, alanine aminotransferase, and SpO₂ levels), and others that were time invariant.

CRP and lymphocyte count were found to be early warning signs of COVID-19 mortality. CRP, an acute inflammatory marker, was significantly elevated early in the hospitalization in the non-survivor group than the survivor group. In addition, patients who survived had declining levels of CRP throughout the hospital stay. These findings are consistent with previous reports on the predictive value of CRP in hospitalized COVID-19 patients (19, 20). However, our results suggest that the dynamic trend of CRP over time, rather than a single value, is predictive of outcome. One other study also correlated upward trending CRP with the eventual need of intubation and suggests that early rise in CRP predicts worse prognosis (21). As trending CRP is a widely accessible clinical tool, early CRP trends can assist physicians to stratify patients and determine the need for further medical intervention vs. symptomatic management alone.

Lymphopenia was also found to be an early predictor of mortality, consistent with literature review (22, 23). We found that in non-survivors, lymphocyte count is lower early in admission, and continues to trend downwards during the hospitalization. As lymphocytes play a significant role in the immune defense to viral infection, lymphopenia may reveal disease mechanisms of COVID-19 and suggest therapeutic targets. Presumed theories of lymphopenia include: (1) the virus directly damaging lymphocytes through coronavirus receptors, or lymphatic organs such as the lymph nodes and the spleen; (2) lymphocyte apoptosis induced by either inflammatory cytokine, metabolic derangements, or both (22, 24). Persistent lymphopenia causing mortality from COVID-19 infection could be a result of any one of these possible mechanisms and deserves further research. Nevertheless, our results confirmed the clinical utility of trending CRP and lymphocyte count in monitoring COVID-19 severity and risk stratification.

In contrast to the early indicators, LDH, procalcitonin, ferritin, alanine aminotransferase spiked by a few orders of magnitude prior to death but largely remained temporally stable and elevated during hospitalization. Elevation of these markers indicate significant oxidative stress and systemic inflammation, particularly prior to death. LDH, in particular, has been found to be associated with mortality in respiratory epidemics of MERS-CoV, H7N9, and H5N1 (25). Together with procalcitonin and ferritin, the spikes of these physiological parameters demonstrate that the dynamic inflammatory response elicited by COVID-19 has a key role affecting disease severity and outcome. Ferritin and procalcitonin are acute phase reactants, so it is logical that they would spike with the deterioration of patients. Unexpectedly, in the surviving group, ferritin and procalcitonin remained relatively elevated but stable throughout the entire hospitalization until discharge, despite being acute phase reactants. This suggests that patients with COVID-19 remain at a hyper-inflammatory state even upon discharge, which has implications for post-discharge follow-up and treatments. Multiple studies have also demonstrated elevation of additional inflammatory markers such as IL-6, IL-8, IL-10, and TNF-α in COVID-19 patients (26, 27). Our results suggest that severe COVID-19 infection is more inflammatory than milder disease, and that mortality from COVID-19 is associated with an overwhelming inflammatory response.

Variables Predictive of Outcome

Understanding the temporal progression of these clinical markers allowed us to construct a prediction model with remarkable performance. Our model identified the earliest predictors of death to be LDH, lymphocytes, procalcitonin, D-dimer, CRP, respiratory rate, and WBCs. The predictive model using these combined predictors yielded a remarkable prediction performance: 80–99% AUC 0 to 4 days prior to death, and >70% AUC from 5 to 10 days prior to death, with specificity higher than sensitivity.

Compared to existing prediction models of COVID-19, our model has several strengths. This is the first time that temporal progression of clinical variables is considered into a prediction model. Most published models used clinical data at admission (7–9, 9–17, 28). However, we found that prediction using the admission timepoint has relatively poor accuracy compared to a few days prior to outcome. While this finding is intuitively logical, we provided evidence that roughly 4 days prior to outcome, our current model can yield a highly accurate prediction. Therefore, our prediction model may aid clinicians to anticipate patient's care escalation with a concrete timeline. From the top 7 predictors, we also derived prediction using the top 3, and top 5 predictors. Depending on the availability of laboratory tests, the number of input clinical variables can be customized at different resource settings for wider applicability.

We also sought external validation on a dataset from Tongji hospital in Wuhan, China. Some temporal characteristics were similar, and others were different from the Stony Brook data. LDH, lymphocytes, CRP and D-dimer showed differences between non-survivors and survivors from the onset and these differences were time-invariant, which may be indicative of a more severely ill cohort. This is also consistent with a higher mortality rate of 46.4% in Tongji cohort, compared to 14.5% in the Stony Brook cohort. Nevertheless, using our predictive model, the AUCs for the zero to 9 days prior to death were better than AUCs of Stony Brook data. This may be attributive to the high mortality rate. Despite the different populations of COVID-19 patients, external validation of our model supports the notion that these top earliest predictors of mortality are likely generalizable.

Clinical Implications

While the focus of our paper was to analyze the temporal progression of clinical variables to find which ones were predictive of mortality, we believe our findings have clinical relevance. Our prediction model can assist physicians to make decisions based on common laboratory values in as early as 10 days prior to death. As a result, these variables may serve as an early warning of the poor prognosis later on and the need to intervene now, whether by initiating dexamethasone treatment or starting prone positioning, if not already done so, as early intervention has been associated with lower mortality (29, 30). Currently, as there is no curative drug yet, symptomatic treatment through Supplementary oxygen or anticoagulation has been the mainstay methodology of treatment in hospitals. With the knowledge that COVID-19 induces a hyper-coagulable state, anticoagulation (i.e., low molecular weight heparin) has become an important treatment in the acute and long-term setting (31). However, because the use of anticoagulation comes with side effects such as bleeding, the decision of when to initiate anticoagulation has been contentious (31). With our predictive model, physicians may objectively weigh the risk and benefit of initiating anticoagulation with the predicted outcome. In addition, our findings suggests that the trend of key clinical variables such as CRP and lymphocytes can be used as treatment response to monitor treatment progress.

While our model and other similar models to date are not yet be able to predict mortality of an individual patient at this time, it is nonetheless important to objectively determine which set of variables are most predictive of outcomes. These variables were determined from group data analysis instead of depending on the variable experience of each individual physician. With further testing and validation as more standardized COVID-19 datasets or predictive models are shared publicly, predicting mortality and other outcomes on an individual patient may be possible.

Our study has several novelties: This is the first study that systematically characteristic the temporal progression of clinical variables with time-lock to the day of death or discharge. Most previous studies (7–17) reported similar laboratory variables to be prediction of mortality but based on only laboratory variables at hospital admission, which we believe to be less accurate because they were far downstream.

Using these temporal characteristics, we determined that the earliest predictors of mortality were lactate dehydrogenase, lymphocyte count, procalcitonin, D-dimer, C-reactive protein, respiratory rate, and white-blood cells, and that they could accurately predicted a few days prior to death. Another novelty is that we have designed prediction models using the top 7 variables, the top 5 variables, and the top 3 variables. We found that accuracy, as determined by a higher AUC value, was highest when using the top 7 variables, and lowest when using just the top 3 variables to predict outcome. Another interesting finding is that the high temporal fluctuations of many clinical variables, indicative of physiological and biochemical instability, were associated with higher likelihood of mortality. These findings were replicated on data from another hospital.

Limitations

Our study has several limitations. A major limitation of the current study is its retrospective nature which could have unintentional patient selection bias. This study design is also subject to residual confounding factors that were unaccounted for. Our predictive model is based on data from two medical centers and requires further, multi-center prospective validation. We did not compare clinical variables to normal ranges in order to focus the analysis on differences between survivors and non-survivors. Our use of the logistic regression model instead of a proportional hazard model did not allow for evaluating survival time and comparison of variables across time points. In a future model, we will look at multiple time points across time and survival time. Radiological imaging data are not included (32–37). AUROC is not actionable for clinicians. We believe that given the large complexity and multiple clinical parameters, there is unlikely to be a single (or two) variable that can provide actionable insights for the bedside clinician. A model of collection of variables will likely be needed to predict mortality and we will provide a user-friendly Excel to provide insights that are potentially actionable. This model may have more utility in the general floor patients than ICU patients because unlikely ICU patients who are more closely monitored so their physiologic derangements are more evident rapidly, general floor patients are not monitored as closely. On the general floor, our predictive model could provide early warning signs for escalate care, and our knowledge of the temporal change in variables could be used to trend efficacy of treatment.

Conclusion

This study characterized the temporal progression of readily available clinical and laboratory variables associated with COVID-19 infection, providing important insights in disease pathogenesis. The earliest clinical predictors of mortality were identified, and they accurately predicted death a few days prior to outcome. The indicators that change early on and gradually worsen can serve as early warning signs because they allow physicians to intervene and, thus, should be closely monitored in COVID-19 patients. This approach may prove useful for management of COVID-19 patients and allocation of hospital resources in time-sensitive, stressful, and resource-constrained circumstances.

Data Availability Statement

De-identified dataset available to be shared upon request.

Ethics Statement

The studies involving human participants were reviewed and approved by Stony Brook University Institution Review Board Office of Research Compliance. Written informed consent for participation was not required for this study in accordance with the national legislation and the institutional requirements.

Author Contributions

AC, ZZ, and WH contributed equally to the collection of the data, analysis of the data, and the writing of this manuscript. AS, HL, and TD contributed equally to the collection and interpretation of the data, and in writing the manuscript. All authors contributed to the article and approved the submitted version.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Acknowledgments

We would like to thank front-line healthcare workers for their dedication during this pandemic. We would also like to thank Dr. Henry Thode for his contribution in helping us collect the preliminary data.

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fmed.2021.661940/full#supplementary-material

Supplementary Figure 1. The time courses of the clinical variables for (A) ICU group and (B) general floor group as a function of days to outcome, time lock to the day of death (“Non”: non-survivors) or the day of discharge (“Sur”: survivors). Error bars are SEM. Two rows of numbers are sample sizes. * indicates significant difference with correction of multiple comparison and covariate with sex, age, ethnicity and comorbidities. Patients upgraded to ICU from a general floor were included in the ICU group.

References

1. Johns Hopkin University. Available online at: https://coronavirus.jhu.edu/map.html

2. Zhu N, Zhang D, Wang W, Li X, Yang B, Song J, et al. A novel coronavirus from patients with pneumonia in China, 2019. N Engl J Med. (2020) 382:727–33. doi: 10.1056/NEJMoa2001017

PubMed Abstract | CrossRef Full Text | Google Scholar

3. Huang C, Wang Y, Li X, Ren L, Zhao J, Hu Y, et al. Clinical features of patients infected with 2019 novel coronavirus in Wuhan, China. Lancet. (2020) 395:497–506. doi: 10.1016/S0140-6736(20)30183-5

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Rodriguez-Morales AJ, Cardona-Ospina JA, Gutiérrez-Ocampo E, Villamizar-Peña R, Holguin-Rivera Y, Escalera-Antezana JP, et al. Clinical, laboratory and imaging features of COVID-19: a systematic review and meta-analysis. Travel Med Infect Dis. (2020) 34:101623. doi: 10.1016/j.tmaid.2020.101623

PubMed Abstract | CrossRef Full Text | Google Scholar

5. Brown RAC, Barnard J, Harris-Skillman E, Harbinson B, Dunne B, Drake J, et al. Lymphocytopaenia is associated with severe SARS-CoV-2 disease: a systematic review and meta-analysis of clinical data. medRxiv [preprint]. (2020) 2020.2004.2014.20064659. doi: 10.1101/2020.04.14.20064659

CrossRef Full Text

6. Cao Y, Liu X, Xiong L, Cai K. Imaging and clinical features of patients with 2019 novel coronavirus SARS-CoV-2: a systematic review and meta-analysis. J Med Virol. (2020) 03:03. doi: 10.1002/jmv.25822

PubMed Abstract | CrossRef Full Text | Google Scholar

7. Lu J, Hu S, Fan R, Liu Z, Yin X, Wang Q, et al. ACP risk grade: a simple mortality index for patients with confirmed or suspected severe acute respiratory syndrome coronavirus 2 disease (COVID-19) during the early stage of outbreak in Wuhan, China. medRxiv [Preprint]. (2020). doi: 10.1101/2020.02.20.20025510

PubMed Abstract | CrossRef Full Text | Google Scholar

8. Zhang P, Zhu L, Cai J, Lei F, Qin JJ, Xie J, et al. Association of inpatient use of angiotensin converting enzyme inhibitors and angiotensin ii receptor blockers with mortality among patients with hypertension hospitalized with COVID-19. Circ Res. (2020) 26:1671–81. doi: 10.1161/CIRCRESAHA.120.317242

CrossRef Full Text | Google Scholar

9. Xie J, Hungerford D, Chen H, Abrams ST, Li S, Li X, et al. Development and external validation of a prognostic multivariable model on admission for hospitalized patients with COVID-19. medRxiv [preprint]. (2020). doi: 10.1101/2020.03.28.20045997

CrossRef Full Text | Google Scholar

10. Jiang X, Coffee M, Bari A, Wang J, Jiang X, Huang J, et al. Towards an artificial intelligence framework for data-driven prediction of coronavirus clinical severity. Comput Mater Cont. (2020) 63:537–51. doi: 10.32604/cmc.2020.010691

CrossRef Full Text | Google Scholar

11. Ji D, Zhang D, Xu J, Chen Z, Yang T, Zhao P, et al. Prediction for Progression Risk in Patients with COVID-19 Pneumonia: the CALL score. Clin Infect Dis. (2020) 7:1393–9. doi: 10.1093/cid/ciaa414

PubMed Abstract | CrossRef Full Text | Google Scholar

12. Hu H, Yao N, Qiu Y. Comparing Rapid Scoring Systems in Mortality Prediction of Critically Ill Patients With Novel Coronavirus Disease. Acad Emerg Med. (2020) 27:461–8. doi: 10.1111/acem.13992

PubMed Abstract | CrossRef Full Text | Google Scholar

13. Zhu JS, Ge P, Jiang C, Zhang Y, Li X, Zhao Z, et al. Deep-learning artificial intelligence analysis of clinical variables predicts mortality in COVID-19 patients. J Am Coll Emerg Physicians Open. (2020) 1:1364–73. doi: 10.1002/emp2.12205

PubMed Abstract | CrossRef Full Text | Google Scholar

14. Zhao Z, Chen A, Hou W, Graham JM, Li H, Richman PS, et al. Prediction model and risk scores of ICU admission and mortality in COVID-19. PLoS ONE. (2020) 15:e0236618. doi: 10.1371/journal.pone.0236618

PubMed Abstract | CrossRef Full Text | Google Scholar

15. Lam KW, Chow KW, Vo J, Hou W, Li H, Richman PS, et al. Continued in-hospital ACE inhibitor and ARB Use in hypertensive COVID-19 patients is associated with positive clinical outcomes. J Infect Dis. (2020) 222:1256–64. doi: 10.1093/infdis/jiaa447

CrossRef Full Text | Google Scholar

16. Hou W, Zhao Z, Chen A, Li H, Duong TQ. Machining learning predicts the need for escalated care and mortality in COVID-19 patients from clinical variables. Int J Med Sci. (2021) 18:1739–1745. doi: 10.7150/ijms.51235

PubMed Abstract | CrossRef Full Text | Google Scholar

17. Li X, Ge P, Zhu J, Li H, Graham J, Singer A, et al. Deep learning prediction of likelihood of ICU admission and mortality in COVID-19 patients using clinical variables. PeerJ. (2020) 8:e10337. doi: 10.7717/peerj.10337

PubMed Abstract | CrossRef Full Text | Google Scholar

18. Yan L, Zhang HT, Goncalves J, Xiao Y, Wang M, Guo Y, et al. An interpretable mortality prediction model for COVID-19 patients. Nat Machine Intelligence. (2020) 2:283–8. doi: 10.1038/s42256-020-0180-7

CrossRef Full Text | Google Scholar

19. Zhou F, Yu T, Du R, Fan G, Liu Y, Liu Z, et al. Clinical course and risk factors for mortality of adult inpatients with COVID-19 in Wuhan, China: a retrospective cohort study. Lancet. (2020) 395:1054–62. doi: 10.1016/S0140-6736(20)30566-3

PubMed Abstract | CrossRef Full Text | Google Scholar

20. Ruan Q, Yang K, Wang W, Jiang L, Song J. Clinical predictors of mortality due to COVID-19 based on an analysis of data of 150 patients from Wuhan, China. Intensive Care Med. (2020) 46:846–8. doi: 10.1007/s00134-020-05991-x

PubMed Abstract | CrossRef Full Text | Google Scholar

21. Mueller AA, Tamura T, Crowley CP, DeGrado JR, Haider H, Jezmir JL, et al. Inflammatory biomarker trends predict respiratory decline in COVID-19 patients. Cell Rep Med. (2020) 1:100144. doi: 10.1016/j.xcrm.2020.100144

PubMed Abstract | CrossRef Full Text | Google Scholar

22. Khourssaji M, Chapelle V, Evenepoel A, Belkhir L, Yombi JC, van Dievoet M-A, et al. A biological profile for diagnosis and outcome of COVID-19 patients. Clin Chem Lab Med (CCLM). (2020) 58:2141–50. doi: 10.1515/cclm-2020-0626

PubMed Abstract | CrossRef Full Text | Google Scholar

23. Wagner J, DuPont A, Larson S, Cash B, Farooq A. Absolute lymphocyte count is a prognostic marker in Covid-19: a retrospective cohort review. Int J Lab Hematol. (2020) 42:761–5. doi: 10.1111/ijlh.13288

PubMed Abstract | CrossRef Full Text | Google Scholar

24. Tan L, Wang Q, Zhang D, Ding J, Huang Q, Tang Y-Q, et al. Lymphopenia predicts disease severity of COVID-19: a descriptive and predictive study. Signal Transduction Targeted Therapy. (2020) 5:33. doi: 10.1038/s41392-020-0148-4

PubMed Abstract | CrossRef Full Text | Google Scholar

25. Shi J, Li Y, Zhou X, Zhang Q, Ye X, Wu Z, et al. Lactate dehydrogenase and susceptibility to deterioration of mild COVID-19 patients: a multicenter nested case-control study. BMC Med. (2020) 18:168. doi: 10.1186/s12916-020-01633-7

PubMed Abstract | CrossRef Full Text | Google Scholar

26. Zeng Z, Yu H, Chen H, Qi W, Chen L, Chen G, et al. Longitudinal changes of inflammatory parameters and their correlation with disease severity and outcomes in patients with COVID-19 from Wuhan, China. Critical Care. (2020) 24:525. doi: 10.1186/s13054-020-03255-0

PubMed Abstract | CrossRef Full Text | Google Scholar

27. Upadhyay J, Tiwari N, Ansari MN. Role of inflammatory markers in corona virus disease (COVID-19) patients: a review. Exp Biol Med. (2020) 245:1368–75. doi: 10.1177/1535370220939477

PubMed Abstract | CrossRef Full Text | Google Scholar

28. Sperrin M, Grant SW, Peek N. Prediction models for diagnosis and prognosis in Covid-19. BMJ. (2020) 369:m1464. doi: 10.1136/bmj.m1464

PubMed Abstract | CrossRef Full Text | Google Scholar

29. Sun Q, Qiu H, Huang M, Yang Y. Lower mortality of COVID-19 by early recognition and intervention: experience from Jiangsu Province. Ann Inten Care. (2020) 10:33. doi: 10.1186/s13613-020-00650-2

PubMed Abstract | CrossRef Full Text | Google Scholar

30. Group RC. Dexamethasone in hospitalized patients with Covid-19—preliminary report. N Engl J Med. (2020) 384:693–704. doi: 10.1056/NEJMoa2021436

CrossRef Full Text | Google Scholar

31. Atallah B, Mallah SI, AlMahmeed W. Anticoagulation in COVID-19. Eur Heart J Cardiovasc Pharmacother. (2020) 6:260–1. doi: 10.1093/ehjcvp/pvaa036

CrossRef Full Text | Google Scholar

32. Zhu J, Shen B, Abbasi A, Hoshmand-Kochi M, Li H, Duong TQ. Deep transfer learning artificial intelligence accurately stages COVID-19 lung disease severity on portable chest radiographs. PLoS ONE. (2020) 15:e0236621. doi: 10.1371/journal.pone.0236621

PubMed Abstract | CrossRef Full Text | Google Scholar

33. Kikkisetti S, Zhu J, Shen B, Li H, Duong TQ. Deep-learning convolutional neural networks with transfer learning accurately classify COVID-19 lung infection on portable chest radiographs. PeerJ. (2020) 8:e10309. doi: 10.7717/peerj.10309

PubMed Abstract | CrossRef Full Text | Google Scholar

34. Hussain L, Nguyen T, Li H, Abbasi AA, Lone KJ, Zhao Z, et al. Machine-learning classification of texture features of portable chest X-ray accurately classifies COVID-19 lung infection. Biomed Eng Online. (2020) 19:88. doi: 10.1186/s12938-020-00831-x

PubMed Abstract | CrossRef Full Text | Google Scholar

35. Cohen JP, Dao L, Roth K, Morrison P, Bengio Y, Abbasi AF, et al. Predicting COVID-19 pneumonia severity on chest X-ray with deep learning. Cureus. (2020) 12:e9448. doi: 10.7759/cureus.9448

PubMed Abstract | CrossRef Full Text | Google Scholar

36. Toussie D, Voutsinas N, Finkelstein M, Cedillo MA, Manna S, Maron SZ, et al. Clinical and chest radiography features determine patient outcomes in young and middle age adults with COVID-19. Radiology. (2020) 297:201754. doi: 10.1148/radiol.2020201754

CrossRef Full Text | Google Scholar

37. Li MD, Arun NT, Gidwani M, Chang K, Deng F, Little BP, et al. Automated assessment and tracking of COVID-19 pulmonary disease severity on chest radiographs using convolutional siamese neural networks. Radiol Artif Intelligence. (2020) 2. doi: 10.1148/ryai.2020200079

CrossRef Full Text | Google Scholar