Risk Stratification Model for Predicting Coronary Care Unit Readmission

Background: Use of statistical models for assessing the clinical risk of readmission to medical and surgical intensive care units is well established. However, models for predicting risk of coronary care unit (CCU) readmission are rarely reported. Therefore, this study investigated the characteristics and outcomes of patients readmitted to CCU to identify risk factors for CCU readmission and to establish a scoring system for identifying patients at high risk for CCU readmission.

Methods: Medical data were collected for 27,841 patients with a history of readmission to the CCU of a single multi-center healthcare provider in Taiwan during 2001-2019. Characteristics and outcomes were compared between a readmission group and a non-readmission group. Data were segmented at a 9:1 ratio for model building and validation.

Results: The number of patients with a CCU readmission history after transfer to a standard care ward was 1,790 (6.4%). The eleven factors that had the strongest associations with CCU readmission were used to develop and validate a CCU readmission risk scoring and prediction model. When the model was used to predict CCU readmission, the receiver-operating curve characteristic was 0.7038 for risk score model group and 0.7181 for the validation group. A CCU readmission risk score was assigned to each patient. The patients were then stratified by risk score into low risk (0–12), moderate risk (13–31) and high risk (32–40) cohorts check scores, which showed that CCU readmission risk significantly differed among the three groups.

Conclusions: This study developed a model for estimating CCU readmission risk. By using the proposed model, clinicians can improve CCU patient outcomes and medical care quality.

Background

An intensive care unit (ICU) readmission is associated with undesirable outcomes, including increases in mortality, duration of hospital stay, and medical costs (1, 2).

ICU readmissions is an essential consideration for three reasons. First, readmissions consume substantial financial and medical resources, regardless of whether the patient is readmitted for the same illness or an unrelated illness. Second, ICU readmissions is often used as a convenient indicator of care quality. Third, reported ICU readmission rates remain high, ranging from 4 to 14% (2). When determining whether a patient is ready for discharge, ICU personnel often rely solely on their own clinical experience and judgment. Thus, the decision is usually highly subjective (3). Insufficient hospital beds may result in premature discharge of ICU patients, some of whom inevitably require readmission. Therefore, identifying patients with a high risk of ICU readmission would not only prevent discharges of patients who are not ready for transfer to standard care, it would also reduce morbidity and mortality after discharge.

Many previous studies have addressed this problem by investigating risk factors for ICU readmission. Thus, various risk factors have been identified, and many different solutions have been proposed for estimating and reducing ICU readmission risk (4–8). However, no studies have validated a scoring system for predicting coronary care unit (CCU) readmission risk. Clinical characteristics substantially differ between CCU and ICU patients. For example, CCU patients typically receive care for an acute episode of congestive heart failure or acute myocardial infarction whereas medical ICU patients typically receive care for sepsis or acute respiratory failure.

The objectives of this study were to identify characteristics and outcomes in patients readmitted to CCU in order to identify factors that increase CCU readmission risk and to establish a scoring system for predicting patients who have a high risk of CCU readmission.

Materials and Methods

Data Source

This retrospective observational study analyzed data from the Chang Gung Research Database (CGRD) provided by the Chang Gung Medical Foundation, Taiwan. The CGRD contains complete electronic medical records (EMR) data from seven Chang Gung Memorial Hospital branches in seven different counties/cities in Taiwan, including three medical centers and four regional/district hospitals, which have 10,050 total beds (9, 10). The CGRD, which is the largest EMR database in Taiwan, contains records for approximately 6.1% of outpatients and 10.2% of inpatients in Taiwan (9, 10).

The identification number of each patient in the CGRD is encrypted and de-identified to protect privacy. Therefore, informed consent was waived for this study. The diagnosis and laboratory data could be linked and continuously monitored using consistent data encryption. This study was approved by the Institutional Review Committee on Human Research at our institution (IRB: 201900829B0C501). This study complied with the Declaration of Helsinki. The interpretation and conclusions contained herein do not represent the views of Chang Gung Memorial Hospital.

Study Patients and Setting

The CGRD search included EMRs for patients treated January 1, 2001, to July 1, 2019. Data were retrieved for all patients with a history of at least one CCU discharge (N = 40,187). After excluding patients without left ventricle ejection fraction (LVEF) data, the final number of participants in model building was 27,841. The flowchart of the study design and patient enrollment is shown in Figure 1. These patients were then separated into readmission and non-readmission groups, where readmission was defined as readmission to CCU after initial CCU discharge and non-readmission was defined as absence of CCU readmission history after initial CCU discharge, death after CCU discharge, or transfer from CCU to general ward.

FIGURE 1

Figure 1. Study design and screening criteria flow chart. CCU, coronary care unit.

This study was performed in two steps. First, the study data were segmented at a ratio of 9 to 1 for model building and model validation, respectively. That is, 90% of data were used to develop the prediction model, including variable selection, scoring, and model building. The remaining 10% of data were then used to validate the developed prediction model. The capability of the model was evaluated in terms of specificity, sensitivity, and positive predictive value.

Measurements

The primary outcome of interest was defined as any CCU readmission after CCU discharge from the index admission. Secondary outcomes of interest were CCU length of stay, in-hospital mortality and hospital costs during admission. Criteria for CCU admission included, but were not limited to, (1) a need for mechanical ventilation or airway protection, invasive hemodynamic monitoring, or circulatory assistive device such as extracorporeal membrane oxygenation (ECMO) or intra-aortic balloon pump (IABP), (2) medication requiring continuous hemodynamic monitoring, and (3) acute myocardial infarction, acute coronary syndrome, or post-intervention procedure such as left main stent placement or carotid artery stenting.

Covariate analysis included demographic, physiological and laboratory data on the day of discharge. Specifically, the analysis included age, gender, body weight, and smoking history. Underlying comorbidities included in the covariate analysis were hyperlipidemia, congestive heart failure (CHF), stroke, peripheral artery occlusive disease, atrial fibrillation, chronic kidney disease (eGFR <60), end stage renal disease under dialysis, diabetes mellitus, hypertension, coronary artery disease, chronic obstructive pulmonary disease, liver cirrhosis, percutaneous coronary intervention, myocardial infarction, coronary artery bypass graft (CABG) and Charlson comorbidity index. Since using continuous variables increases the complexity of calculations when establishing the risk score model, we convert the selected continuous variables into categorical variables according to clinical practice or references. Covariate analysis also included cause of ICU admission; source of initial ICU admission (emergency department vs. general ward); treatment with ventilator, ECMO or IABP during CCU stay; sepsis during CCU stay (pneumonia, urinary tract infection, intraabdominal infection or other bacteremia); and bleeding during CCU stay (intracranial bleeding, intra-abdominal bleeding, gastrointestinal bleeding and other bleeding).

Statistical Analysis

Data were presented as means and standard deviations for continuous variables and as proportions for categorical variables. Analysis of variance was performed to analyze differences in continuous variables. The χ² test was used to analyze differences in nominal variables. To identify independent predictors of CCU readmission, explanatory covariates with a p < 0.01 in univariate analysis were entered into a multivariable regression model in a forward/backward fashion. In order to facilitating clinical application, we eliminated variables which are less contributory based on literature, clinical knowledge and completeness of the data to assemble an easy model to assist clinicians. The results of the multivariable regression analysis were then used to derive the clinical prediction model with CCU readmission as the outcome variable. Point values were assigned according to a rough multiple of the odds ratio (OR) for each variable. An area under the receiver operator curve (AUROC) was drawn to estimate the accuracy of the model in predicting CCU readmission risk at the time of CCU discharge. Statistical significance was defined as a P < 0.05. All analyses were performed using SAS 9.2 (SAS Institute Inc., Cary, NC, USA).

Results

From January 1, 2001 to July 1, 2019, 27,841 patients were discharged from our CCU. Of these, 1,790(6.4%) were “bounce back” patients, i.e., patients readmitted to CCU after discharge to a general ward.

Table 1 presents the characteristics of the groups at the time of admission, i.e., before transfer to a ward. Patient characteristics statistically associated with readmission included older age, female gender, low body weight, no tobacco use, and any history of the following: hyperlipidemia, CHF, stroke, peripheral artery occlusive disease, atrial fibrillation, chronic kidney disease stage>3, dialysis, hypertension, chronic lung disease, diabetes, chronic liver disease, percutaneous coronary intervention, myocardial infarction, and CABG.

TABLE 1

Table 1. Baseline characteristics.

Compared to non-readmission patients, readmission patients also had lower diastolic blood pressure before transfer and a higher out-of-hospital cardiac arrest rate before CCU readmission. However, the readmission patients had a lower in-hospital cardiac arrest rate before the first CCU admission compared to the non-readmission patients. During their first CCU stay, readmission patients also tended to have lower rates of IABP use and ECMO use but a higher rate of ventilator use compared to the non-readmission patients. Readmission patients also had higher rates of sepsis and bleeding. However, readmission patients had lower hemoglobin, lower glomerular filtration rate, and lower left ventricle ejection fraction. By basing on clinical practice where LVEF <30% is defined as severe heart failure and several other parameters we analyzed, we had chosen age>80 and LVEF <30% as our threshold to obtain the best predictability. Finally, readmission patients had higher Charlson comorbidity index, higher in-hospital mortality, higher medical costs, and longer CCU stays compared to non-readmission patients.

After univariate and step-wise multivariate logistic regression analysis, factors that revealed positive associations with CCU readmission (Table 2) were history of CHF [OR 1.794, 95% confidence interval (CI)1.601–2.01], history of peripheral artery occlusive disease [OR 1.328, 95% CI 1.067–1.654], history of chronic kidney disease (eGFR <60) [OR 1.689, 95% CI 1.503–1.899], history of diabetes mellitus [OR 1.427, 95% CI 1.28–1.592], history of percutaneous coronary intervention [OR 1.4, 95% CI 1.16–1.689], history of myocardial infarction [OR 1.776, 95% CI 1.558–2.025], sepsis during CCU stay [OR 1.608, 95% CI 1.443–1.791], bleeding during CCU stay [OR 1.184, 95% CI 1.019–1.376], Admission to CCU from emergency department [OR 1.359, 95% CI 1.113–1.661], Age>80 [OR 1.249, 95% CI 1.115–1.398], LVEF <30% [OR 1.329, 95% CI 1.134–1.557]

TABLE 2

Table 2. Multivariable analysis for readmission.

In the multivariable model, variables that were independently associated with CCU readmission risk score were obtained from 90% of the study population (Figure 2A). Scores for CCU readmission risk were assigned to the aforementioned variables based on their relative ORs. Figure 2B shows the relationship between risk score and predicted probability of CCU readmission. Notably, the figure shows that the rise in risk score corresponded with the rise in the predicted risk of CCU readmission. The patients were then stratified by readmission risk into three cohorts: a low risk cohort (0–12), a moderate risk cohort (13–31), and a high-risk cohort (32–40). Table 3 further shows that the predicted risk of readmission was 0–5%, 5–30%, and > 30% in the low-, moderate- and high-risk cohorts, respectively. The predicted readmission rate by risk-cohort in validation cohort is in Supplement Table 1.

FIGURE 2

Figure 2. CCU readmission risk score model nomogram. (A) The CCU readmission risk score and (B) mean predicted CCU readmission by CCU readmission risk score. CCU, coronary care unit; CHF, congestive heart failure; PAOD, peripheral artery occlusive disease; CKD, chronic kidney disease; DM, diabetes mellitus; PCI, percutaneous coronary intervention; MI, myocardial infarction; LVEF, left ventricular ejection fraction.

TABLE 3

Table 3. Predicted readmission rate by risk-cohort (over all).

The model had an AUROC of 0.7038 for predicting CCU readmission rate (Figure 3A). Validation studies of the model in the remaining 10% of our patients obtained an AUROC of 0.7181 (Figure 3B). Figure 4 compares the percentages of patients readmitted between overall cohort among the three groups. The percentage of readmitted patients significantly differed among the low-, moderate-, and high-risk groups (2.74, 10.2 and 31.73%, respectively; p < 0.0001). For predicting CCU readmission, our risk score model had a sensitivity of 54.51%, a specificity of 73.82% and a negative predictive value of 95.81%. The validation cohort had a sensitivity of 50.6%, a specificity of 73.67% and a negative predictive value of 95.76%. The validation study confirmed that the proposed risk prediction model had acceptable calibration, discrimination and validation.

FIGURE 3

Figure 3. AUC curve for readmission risk score model and validation group. (A) Area under the receiver operator curve (AUC) for risk score model in predict coronary care unit readmission. (B) AUC for risk score model in predict coronary care unit readmission in validation group.

FIGURE 4

Figure 4. Readmission percentage in different risk group. Readmission percentage between overall cohort regarding low (0–12), moderate (13–31), high risk group (32–40). P for trend <0.0001 between 3 group.

Discussion

To our knowledge, this study is the first to investigate risk factors for CCU readmission and to establish a model for predicting CCU readmission. This study revealed that readmission patients have a much higher mortality risk and a higher medical cost compared to non-readmission patients. The proposed model for predicting CCU readmission risk based on eleven known risk factors showed acceptable predictive accuracy in both the risk score model group and validation group.

Special Findings of Our Study

As patients admitted to CCU have unique clinical characteristics, risk factors for readmission included cardiovascular medical history such as old age, CHF, peripheral artery occlusive disease, chronic kidney disease, diabetes mellitus, percutaneous coronary intervention, myocardial infarction and poor left ventricular ejection fraction. Additionally, sepsis and bleeding during CCU were associated with CCU readmission. Interestingly, patients who had received ECMO and patients initially admitted to CCU for in-hospital cardiac arrest had a lower readmission rate. We hypothesized that the lower readmission rate resulted from a higher mortality rate in these patients either during or after their CCU stay. These patients usually have poor prognosis and tend to be treated conservatively. After the patient is admitted to CCU, family members often realize that prognosis is poor and agree to sign a “do-not-resuscitate” order. Therefore, when the condition of the patient deteriorates, the patient is seldom transferred back to CCU. Additionally, certain demographic characteristics and comorbidities had strong associations with CCU readmission and poor outcomes. Readmission was significantly associated with long CCU stay, high mortality rate, and high medical costs.

Comparison With Other Scoring Systems

Several studies in the literature have developed tools for predicting adverse outcomes after ICU discharge (4, 8–11). Most have assessed outcomes of readmission and hospital mortality in medical ICU and surgical ICU patients. For example, the Modified Early Warning Score developed by Reini et al. is often used to estimate the risk of ICU readmission within 72 h after ICU discharge (14). Gajic et al. (11) developed the Stability and Workload Index for Transfer score to predict ICU readmission within 7 days after ICU discharge. The calculated AUROCs ranged from 0.66 to 0.92 (4, 5, 11–14).

Mišić et al. (13) and Mišić et al. (12) also applied machine learning methods for readmission prediction among surgical populations (12, 13). Characteristics that increased the probability of CCU readmission in our study somewhat differed from those in previous reports. A common limitation of previous works is the lack of model calibration or discrimination and the lack of validation in CCU patients. Most CCU patients require care for an episode of acute CHF or acute myocardial infarction whereas most medical ICU patients are admitted for sepsis or acute respiratory failure (15, 16).

Clinical Implications

This study developed and validated a model for using a comprehensive set of variables for clinical prediction of CCU readmission after discharge. After calibration, the proposed model demonstrated acceptable discrimination and calibration in identifying patients at high risk for CCU readmission at the time of CCU discharge. The model may be applicable for evaluating the efficacy of targeted therapeutic interventions aimed at reducing CCU readmission. Notably, none of the model parameters are modifiable in subsequent hospitalization. Therefore, to reduce the potential for CCU readmission, improved measures for preventing infection and hemorrhage should be implemented during the course of a CCU stay (17). To our knowledge, this model is the first designed specifically for predicting CCU readmission, and we hope that information provided by this model can be used to support clinical decision making, including discharge planning. Additionally, CCU physicians and personnel can use the model to determine whether a CCU patient can be safely discharged. Although not all CCU readmissions and deaths are avoidable, identifying high-risk patients can provide clinicians with insight into the appropriate timing of CCU discharge. For patients with high risk of CCU readmission or mortality, an appropriate CCU discharge plan should be made in advance and should include delayed discharge until stabilization, discharge to a step-down unit, and aggressive follow up in care wards, e. g., use of telemetry devices.

Study Limitation

This study had several weaknesses inherent in the use of a database analysis. First, the analysis did not exclude patients readmitted to CCU for observation after a specialized procedure or operation, e.g., left main stent placement or carotid artery stenting. Second, due to resource allocation issues and lack of ICU beds, some patients designated for transfer to medical ICU were transferred to CCU instead. Third, since this study only included patients admitted to CCU, the prediction model developed in this study is not expected to be applicable to other care units such as ICUs. A final limitation is that, since this study analyzed data contained in a medical records database, the causes of CCU readmission could not be determined.

Conclusions

In conclusion, this study identified 11 factors associated with increased CCU readmission risk and then developed a model for assessing this risk. The model may benefit clinicians in preventing CCU readmissions, which would then improve patient outcomes and medical care quality.

Data Availability Statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics Statement

The studies involving human participants were reviewed and approved by the Institutional Review Committee on Human Research at Chang Gung Memorial Hospital (IRB: 201900829B0C501). Written informed consent was not required for this study, in accordance with the local legislation and institutional requirements.

Author Contributions

C-HT and C-IC conceived the idea. C-HT, T-YC, and P-JW implemented the study. C-CW, C-HL, W-JC, and C-IC supervised the study. C-HT and T-YC conducted data analysis. C-HT and P-JW interpreted the data. C-HT drafted the manuscript. C-IC provided critical revision. All authors reviewed the manuscript for important intellectual content, read, and approved the final manuscript.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Acknowledgments

The authors thank Mr. Pai-Wei Lee, Mrs. Ya-Chi Huang, Mrs. Chiu-Yi Hsu, and Mr. Yu-Tung Huang Maintenance Project of the Center for Big Data Analytics and Statistics and all of its members at Chang Gung Memorial Hospital for support in the design of this study and in data analysis and interpretation.

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fcvm.2022.825181/full#supplementary-material

References

1. Rosenberg AL, Hofer TP, Hayward RA, Strachan C, Watts CM. Who bounces back? Physiologic and other predictors of intensive care unit readmission. Crit Care Med. (2001) 29:511–8. doi: 10.1097/00003246-200103000-00008

PubMed Abstract | CrossRef Full Text | Google Scholar

2. Rosenberg AL, Watts C. Patients readmitted to ICUs: a systematic review of risk factors and outcomes. Chest. (2000) 118:492–502. doi: 10.1378/chest.118.2.492

PubMed Abstract | CrossRef Full Text | Google Scholar

3. Capuzzo M, Moreno RP, Alvisi R. Admission and discharge of critically ill patients. Curr Opin Crit Care. (2010) 16:499–504. doi: 10.1097/MCC.0b013e32833cb874

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Campbell AJ, Cook JA, Adey G, Cuthbertson BH. Predicting death and readmission after intensive care discharge. Br J Anaesth. (2008) 100:656–62. doi: 10.1093/bja/aen069

PubMed Abstract | CrossRef Full Text | Google Scholar

5. Fernandez R, Serrano JM, Umaran I, Abizanda R, Carrillo A, Lopez-Pueyo MJ, et al. Ward mortality after ICU discharge: a multicenter validation of the Sabadell score. Intensive Care Med. (2010) 36:1196–201. doi: 10.1007/s00134-010-1825-5

PubMed Abstract | CrossRef Full Text | Google Scholar

6. Kansagara D, Englander H, Salanitro A, Kagen D, Theobald C, Freeman M, et al. Risk prediction models for hospital readmission: a systematic review. JAMA. (2011) 306:1688–98. doi: 10.1001/jama.2011.1515

PubMed Abstract | CrossRef Full Text | Google Scholar

7. Donzé J, Aujesky D, Williams D, Schnipper JL. Potentially avoidable 30-day hospital readmissions in medical patients: derivation and validation of a prediction model. JAMA Intern Med. (2013) 173:632–8. doi: 10.1001/jamainternmed.2013.3023

PubMed Abstract | CrossRef Full Text | Google Scholar

8. van Walraven C, Dhalla IA, Bell C, Etchells E, Stiell IG, Zarnke K, et al. Derivation and validation of an index to predict early death or unplanned readmission after discharge from hospital to the community. CMAJ. (2010) 182:551–7. doi: 10.1503/cmaj.091117

PubMed Abstract | CrossRef Full Text | Google Scholar

9. Wang CL, Wu VC, Kuo CF, Chu PH, Tseng HJ, Wen MS, et al. Efficacy and safety of non–vitamin k antagonist oral anticoagulants in atrial fibrillation patients with impaired liver function: a retrospective cohort study. J Am Heart Assoc. (2018) 7:e009263. doi: 10.1161/JAHA.118.009263

PubMed Abstract | CrossRef Full Text | Google Scholar

10. Shao SC, Chen YY, Yang YHK, Lin SJ, Hung MJ, Chien RN, et al. The Chang Gung Research Database—a multi-institutional electronic medical records database for real-world epidemiological studies in Taiwan. Pharmacoepidemiol Drug Saf. (2019) 28:593–600. doi: 10.1002/pds.4713

PubMed Abstract | CrossRef Full Text | Google Scholar

11. Gajic O, Malinchoc M, Comfere TB, Harris MR, Achouiti A, Yilmaz M, et al. The Stability and Workload Index for Transfer score predicts unplanned intensive care unit patient readmission: initial development and validation. Crit Care Med. (2008) 36:676–82. doi: 10.1097/CCM.0B013E318164E3B0

PubMed Abstract | CrossRef Full Text | Google Scholar

12. Mišić VV, Rajaram K, Gabel E. A simulation-based evaluation of machine learning models for clinical decision support: application and analysis using hospital readmission. NPJ Dig Med. (2021) 4:1–11. doi: 10.1038/s41746-021-00468-7

PubMed Abstract | CrossRef Full Text | Google Scholar

13. Mišić VV, Gabel E, Hofer I, Rajaram K, Mahajan A. Machine learning prediction of postoperative emergency department hospital readmission. Anesthesiology. (2020) 132:968–80. doi: 10.1097/ALN.0000000000003140

PubMed Abstract | CrossRef Full Text | Google Scholar

14. Reini K, Fredrikson M, Oscarsson A. The prognostic value of the Modified Early Warning Score in critically ill patients: a prospective, observational study. Eur J Anaesthesiol. (2012) 29:152–7. doi: 10.1097/EJA.0b013e32835032d8

PubMed Abstract | CrossRef Full Text | Google Scholar

15. Badawi O, Breslow MJ. Breslow. Readmissions and death after ICU discharge: development and validation of two predictive models. PloS ONE. (2012) 7:e48758. doi: 10.1371/journal.pone.0048758

PubMed Abstract | CrossRef Full Text | Google Scholar

16. Jo YS, Lee YJ, Park JS, Yoon HI, Lee JH, Lee CT, et al. Readmission to medical intensive care units: risk factors and prediction. Yonsei Med J. (2015) 56:543–9. doi: 10.3349/ymj.2015.56.2.543

PubMed Abstract | CrossRef Full Text | Google Scholar

17. van Diepen S, Graham MM, Nagendran J, Norris CM. Predicting cardiovascular intensive care unit readmission after cardiac surgery: derivation and validation of the Alberta Provincial Project for Outcomes Assessment in Coronary Heart Disease (APPROACH) cardiovascular intensive care unit clinical prediction model from a registry cohort of 10,799 surgical cases. Crit Care. (2014) 18:1–9. doi: 10.1186/s13054-014-0651-5

PubMed Abstract | CrossRef Full Text | Google Scholar