Signatures of illness in children requiring unplanned intubation in the pediatric intensive care unit: A retrospective cohort machine-learning study

Acute respiratory failure requiring the initiation of invasive mechanical ventilation remains commonplace in the pediatric intensive care unit (PICU). Early recognition of patients at risk for respiratory failure may provide clinicians with the opportunity to intervene and potentially improve outcomes. Through the development of a random forest model to identify patients at risk for requiring unplanned intubation, we tested the hypothesis that subtle signatures of illness are present in physiological and biochemical time series of PICU patients in the early stages of respiratory decompensation. We included 116 unplanned intubation events as recorded in the National Emergency Airway Registry for Children in 92 PICU admissions over a 29-month period at our institution. We observed that children have a physiologic signature of illness preceding unplanned intubation in the PICU. Generally, it comprises younger age, and abnormalities in electrolyte, hematologic and vital sign parameters. Additionally, given the heterogeneity of the PICU patient population, we found differences in the presentation among the major patient groups – medical, cardiac surgical, and non-cardiac surgical. At four hours prior to the event, our random forest model demonstrated an area under the receiver operating characteristic curve of 0.766 (0.738 for medical, 0.755 for cardiac surgical, and 0.797 for non-cardiac surgical patients). The multivariable statistical models that captured the physiological and biochemical dynamics leading up to the event of urgent unplanned intubation in a PICU can be repurposed for bedside risk prediction.

Introduction

Acute respiratory failure requiring the initiation of invasive mechanical ventilation remains commonplace in the pediatric intensive care unit (PICU) (1). Emergent endotracheal intubation in children is associated with an increased risk of complications as compared to intubations performed electively (2). Early recognition of patients at risk for respiratory failure may provide clinicians with the opportunity to intervene and potentially improve outcomes (3).

Predictive analytics leverages physiologic and biochemical data in the development of algorithms that can identify signatures of illness present early in the process of clinical decompensation (4–6). Display of algorithmic-derived scores or indices alert care providers to patients at risk for clinical decompensation. Our group previously found a signature of respiratory distress leading to unplanned intubation in the adult intensive care unit (ICU) population that we validated externally at another site (3, 7, 8). Display of a risk estimate for this event in a surgical ICU was associated with a fall in septic shock by 50% that was not matched in a medical ICU that did not have the display (9). Through the development of a random forest model to identify patients at risk for requiring urgent unplanned intubation, we tested the hypothesis that subtle signatures of illness are present in physiological and biochemical time series of PICU patients in the early stages of respiratory decompensation (10).

Materials and methods

Study design and definitions

We performed a retrospective cohort study from January 2014 to May 2016 at the University of Virginia (UVA) Children’s Hospital. We included all admissions to the PICU, a 17-bed combined cardiac and medical/surgical unit with stored continuous physiologic monitoring data. We defined admission as a unique hospitalization that included PICU stay irrespective of the number of transitions into and out of the PICU. The Institutional Review Board at the University of Virginia approved this study. We followed the Strengthening the Reporting of Observational Studies in Epidemiology guidelines (11).

We identified all intubations performed in the UVA Children's Hospital associated with a PICU admission during the period of study through querying of individual electronic health records. We established our primary end point of an unplanned intubation event based on the time of the event recorded in the electronic health record and a non-elective indication for intubation as documented in the UVA contribution to the National Emergency Airway Registry for Children (NEAR4KIDS) database. We also performed a sensitivity analysis using only the computable phenotype of intubation event recorded in the electronic health record without regard to patient location or indication for intubation (includes e.g., patients intubated in Operating Room by Anesthesia prior to surgery) to evaluate the importance of manual chart review.

The 12-hour window prior to each unplanned intubation event was identified and classified (provided the patient was in the PICU and not already intubated) as case data. We classified data collected at all other times in patients with or without unplanned intubation events as control data. We excluded intubations within 15 min of PICU admission, or those within one hour of a prior extubation. We excluded admissions without archived physiologic monitoring data due to technical complications. Multiple unplanned intubation events in individual patients were treated as independent events for the purposes of analysis as each 12-hour window of case data was distinct.

We excluded times during which a patient was mechanically ventilated. Presence of mechanical ventilation was determined by extracting the ventilator respiratory rate flowsheet vital sign from the electronic data warehouse. We defined mechanical ventilation as starting at the time of the first ventilator respiratory rate and ending at the time of the last ventilator respiratory rate. The period of mechanical ventilation was split when the time between consecutive measurements from a patient was longer than 12 h, and identified the patient as not ventilated in the interim (see Supplementary Figure A1).

Physiologic data acquisition and predictors

Continuous cardiorespiratory monitoring consisted of waveforms (3 leads of ECG sampled at 240 Hz, pulse plethysmography at 120 Hz, and invasive blood pressure tracings at 120 Hz) and vital signs (heart rate, respiratory rate, peripheral oxygen saturation, invasive blood pressure, ventilator measured respiratory rate, and sample-and-hold non-invasive blood pressure) sampled at 0.5 Hz. The GE monitor (GE Healthcare, Chicago, IL) reported sample-and-hold non-invasive blood pressure at 0.5 Hz, while monitor-smoothed invasive blood pressure was measured continuously. We calculated, in 30-minute windows with 50% overlap, the following 18 measures: the mean and standard deviation of heart rate (HR), respiratory rate (RR), pulse oximetry (SO2); mean non-invasive systolic and diastolic blood pressure, or invasive blood pressure in its absence (SBP and DBP); the three pairwise cross-correlations between HR, RR, and SO2; standard deviation of heart inter-beat intervals (sRRI); local dynamics score (LDs) and local dynamics density (LDd) of heart inter-beat intervals (12); coefficient of sample entropy (COSEn) (13); the slope of log variance vs. log scale between scales 4 and 12 for detrended fluctuation analysis of heart inter-beat intervals (DFA) (14); the probability of atrial fibrillation (15). Non-invasive blood pressure was cycled more frequently than every 30-minutes in 95% of epochs in which it was available. Low quality EKG data were excluded from cardiac dynamic calculations (16, 17). LDs and LDd quantify how many heart inter-beat intervals match very many or no other intervals. COSEn quantifies the repeatability of the heart inter-beat interval time series. Finally, DFA quantifies the way in which the variability of heart inter-beat intervals depends upon time scale. The cardiorespiratory dynamics measured from continuous cardiorespiratory monitoring were calculated as described in (18) using CoMET® (AMP3D Inc., a Nihon Kohden Company, Charlottesville, VA). Cardiorespiratory dynamics were not available to the care team.

Vital signs (heart rate, respiratory rate, oxygen saturation, blood pressure, and temperature) were extracted from flowsheets. Additionally, we extracted fraction of inspired oxygen and supplemental oxygen flowrate and combined it with peripheral oxygen saturation to estimate the PO₂ to FiO₂ ratio (ePFR) following the methods of Gadrey, et al. (19, 20). Eleven frequently measured laboratory measurements (serum sodium, potassium, chloride, bicarbonate, blood urea nitrogen (BUN), creatinine, glucose, calcium, white blood cell count, hematocrit, platelet count) were extracted from the electronic data warehouse. Anion gap and BUN to creatinine ratio were included as features. We combined these intermittent features with continuously measured features using sample-and-hold. Vital signs and labs older than 24 and 48 h, respectively, were censored. Finally, we included age as a continuous predictor.

We also performed a sensitivity analysis by excluding features from continuous cardiorespiratory monitoring, both for the chart reviewed unplanned intubations and the computable phenotype.

Subgroup analysis

The study cohort includes patients admitted for a variety of reasons. To investigate differences in physiological signatures between these populations, we identified the patient type for each admission in our retrospective analysis. We defined patient type for each admission based on operating room records: admissions without any operating room records were identified as medical patients, those with a thoracic cardiovascular operating room record were identified as cardiac surgery, and those with any other type of operating room record were identified as non-cardiac surgery.

Physiological signature of illness

We construct physiological signatures of illness specific to age and each patient type in the cohort. For this, predictiveness curves were calculated to display the independent association of vital signs, laboratory values, and continuous cardiorespiratory monitoring parameters with unplanned intubation. To reduce bias due to repeated measures and missing data, we used a bootstrapping technique to estimate the predictiveness curves. For each patient type and for all patients, we randomly sampled eight measurements within 12 h before unplanned intubation and eight measurements from each admission without an intubation event or far from the time of intubation. We calculated the relative risk of unplanned intubation at each decile of the sampled variable and each decile of age over the surrounding quintile. We repeated this process of sampling and calculating relative risk 30 times then averaged to obtain a bootstrapped predictiveness curve and displayed the results as a heat map. Toddenroth and colleagues present an excellent introduction to the construction and interpretation of heat maps in clinical research (21).

Model development

We developed a model on the entire cohort and used cross validation for estimating performance characteristics. Modeling was performed in R (R Foundation for Statistical Computing, Vienna, Austria) using the randomForest packages (22, 23). We used random forest to construct a multivariable model to account for the high-dimensional relationships between predictors, and especially to account for the known age-dependence of many features. We justify its use over other logistic regression based on our prior finding that random forest more accurately captured the influence of age in the signature of sepsis in this population (24). As predictors, we used 18 measures from continuous cardiorespiratory monitoring, 8 vital signs, 13 laboratory measurements, and age. We performed a sensitivity analysis by including a random variable to identify features that were not significantly associated with unplanned intubation. We imputed missing data based on patient age. We first identified the deciles of age and calculated the median of each feature in each decile. Missing values were then imputed with the median for the decile associated with the age of the patient at the missing measurement time.

We constructed a random forest model (25) to identify data for patients that had an unplanned intubation event in the next 12 h. The forest consisted of 800 classification trees, and 6 features (the square root of the number of features) were sampled as candidates at each split (26). We adjusted for the imbalance between the intubation event and non-intubation event data by bootstrapping the event data and sampling the same number of non-event data. The output of the model was the fraction of trees that classified a record as an event. We divided this output by the average value to obtain predicted relative risk, then multiplied by the average probability of unplanned intubation in the next 12 h to get predicted probability.

The predicted risk was calculated using 5-fold cross validation (27). Briefly, hospital admissions were randomly sampled into 5 groups containing approximately 20% of admissions. We used the first fold (20%) as the test set, and we built a model on the remaining 4 folds (80% of admissions). The predicted risk for the test set was estimated using this model, and the procedure repeated for each of 5 folds admissions. This patient-wise method provides less biased performance estimates than cross-validation that selects folds based on row, because in the latter method data from an individual admission may be included in many folds and therefore be used for training and testing.

We also followed this model development and validation procedure to compare performance characteristics for models to predict: the computable phenotype for unplanned intubation excluding continuous cardiorespiratory monitoring features; the computable phenotype for unplanned intubation using all features; the chart reviewed events of unplanned intubation excluding continuous cardiorespiratory monitoring features.

Performance characteristics of predictive models

The model developed here is intended for continuous risk estimation. We contextualized the performance and potential clinical impact of our predictive models relative to the HRC index. We evaluated our model by plotting the time-course of cross-validated predicted risk leading up to the time of unplanned intubation event. Early detection requires not only high-predicted risk before diagnosis, but also an increasing risk to indicate worsening patient status towards critical illness. We also calculated the area under the receiver operating characteristic (AUC) based on cross-validated predictions. The AUC is equivalent to the C-statistic: it is the probability that, when randomly drawing one predicted probability from case and control data, the case data has a higher prediction. A random classifier has the expected AUC of 0.5 and perfect discrimination has AUC of 1.0.

We find a surprisingly wide variety of methods for calculating AUC in the literature. One approach to achieve artificially high AUC is to report “encounter-based” AUCs, meaning that each patient is characterized by the highest AUC during the hospitalization. Thus, the score representing the case patient may have occurred long before or long after the diagnosis was made. This is not useful to the bedside clinician. It is not possible to compare two AUCs unless the data sets and the evaluation criteria are the same. Here we report AUC based on evaluation of all case and control records where a patient is at risk for unplanned intubation in the PICU.

Results

Table 1 shows the study cohort. There were 2767 PICU admissions included in the study. When we require a non-elective indication for intubation using the UVA contribution to the NEAR4KIDS database we found 92 admissions with 116 unplanned intubations. The most common indications noted in the NEAR4KIDS database for unplanned intubations were respiratory failure (64%), both respiratory failure and hemodynamic instability (14%), and hemodynamic instability alone (12%). Patients who experienced unplanned intubation during their stay were younger and were more likely to have had cardiac surgery during their stay. When we did not require a non-elective indication from the UVA contribution to the NEAR4KIDS database we found 944 admissions using the computable phenotype alone.

TABLE 1

Table 1. Demographics of the study population.

Figure 1 shows the characteristics of the study cohort. In Figure 1A, patients admitted for cardiac surgery were predominantly infants admitted with complex congenital heart disease, while medical and non-cardiac surgery patients were more distributed across the pediatric age range. In Figures 1B–D, the estimates are made at each quartile of age for the specific patient type. Unplanned intubation was more likely for younger patients regardless of patient type, see Figure 1B. Unplanned intubation increased length of stay all patient types and ages, and increased mortality for most in Figures 1C,D.

FIGURE 1

Figure 1. Characteristics of the study population. (A) Histogram of admissions for each patient type as a function of age. (B) Rate of unplanned intubation for each patient type as a function of age. (C) Mortality rate for each patient type with (red) and without (black) unplanned intubation. Error bars are 95% confidence interval. (D) Median length of stay in days for each patient type with (red) and without (black) unplanned intubation. Error bars show the IQR.

Figure 2 shows an example of the physiological signature of unplanned intubation for cardiac surgery (left), medical (center), and non-cardiac surgery (right) patients. In each panel, each colored column estimates the empirical relative risk for unplanned intubation as a function of respiratory rate for one decile of age based on the surrounding quintile. That is, for all measurements in each decile of age and respiratory rate, we calculated the probability of unplanned intubation in the next 12 h and divided by the average rate of unplanned intubation in the cohort (0.0063). The youngest 10% of medical patients, for example, have about 3-fold more risk than average (redder) for unplanned intubation when respiratory rate is in the lower 30% (less than about 27 breaths per minute) independent of all other features. Averaging across all columns and panels yields an age and patient type marginalized risk profile for respiratory rate as shown in Figure 3. The deciles of respiratory rate and age are maintained across the rows and panels. There was a distinct physiological signature of illness for unplanned intubation depending on patient type. High respiratory rate is a signature of unplanned intubation for non-cardiac surgery patients and older medical patients, while in younger medical patients it is low respiratory rate and for cardiac surgery patients there is not a strong signature in respiratory rate. See Supplementary Figure A2 for all features.

FIGURE 2

Figure 2. Empirical relative risk for cardiac surgery (left), medical (center), and non-cardiac surgery (right) patients as a function of age and mean respiratory rate from continuous monitoring data. On each panel, younger patients are to the left, and slower respiratory rates are to the bottom. Each colored tile estimates the relative risk of unplanned intubation for each decile of age and respiratory rate based on the surrounding quintile. Deciles of respiratory rate and age remain the same across patient type panels. Higher relative risk is redder while lower is bluer.

FIGURE 3

Figure 3. Risk profiles for exemplary features in the random forest model. These are marginal risk profiles that average out dependence on other features, such as the dependence of heart rate on age. The (natural) log odds of unplanned intubation in the next 12 h (ordinate) is shown as a function of the value of each measured variable (abscissa) holding all other features at their median values. Lower blood pressure, for example, is associated with increased risk for unplanned intubation independent of changes in any other feature.

Figure 3 shows the marginal risk profiles of several example features for the random forest model based on all patients. Each panel shows the log odds of unplanned intubation (ordinate) as a function of the value of each feature (abscissa). We estimated log odds by varying each feature across its range and estimated the log odds from the predicted probability while keeping all other features at their median values, then calculated the logit of predicted probability based on the model. Increasing heart rate from about 70 to 150 with all other features at the median values for the sample, for example, increases the log odds of unplanned intubation in the next 12 h from about −6 to −5, or an increase in probability from 0.0025 to 0.0067. Note that this marginalized representation removes the dependence captured by the model of each feature on age. See Supplementary Figure A3 for all features. All features were found to be more important than the random feature. Small changes in heart rate, respiratory rate, blood pressure, and others, alone or in combination, can be signatures of changing risk for unplanned intubation.

The AUC for the random forest model based on cross-validated predictions is 0.696 for all patients (0.692 for medical, 0.673 for cardiac surgery, and 0.727 for non-cardiac surgery). The AUC depends on the definition of case data (24): the AUC for the random forest model using a definition of 4 h before the event (rather than 12 h before) is 0.766 (0.738 for medical, 0.755 for cardiac surgical, and 0.797 for non-cardiac surgical). For comparison, the AUC of HRC monitoring for neonatal sepsis [that resulted in 20% mortality reduction (28)] within 24 h was 0.70 in the development cohort (29). In adults, prospective display of models for unplanned intubation and hemorrhage with AUC of 0.68 and 0.71, respectively (7), were associated with a 50% reduction of septic shock in an adult surgical ICU (9).

Figure 4 shows a sensitivity analysis for using a computable phenotype rather than chart review, and for excluding continuous cardiorespiratory monitoring features from the model. Each AUC is based on the cross-validated probabilities. Models using continuous cardiorespiratory monitoring features perform better than those without, and models based on chart reviewed events perform better than those based on a computable phenotype: the AUC using a computable phenotype is 29% lower than using manual chart review (0.638 and 0.696, respectively) and the AUC using chart review but excluding continuous cardiorespiratory monitoring features is 12% lower (0.671).

FIGURE 4

Figure 4. Area under the receiver operating characteristic (AUC) for predicting unplanned intubation based on chart review. For comparison, the AUC is also shown for combinations of models built to detect a computable phenotype and either using all features or excluding continuous monitoring data. Each AUC is based on 5-fold cross-validation with confidence intervals based on 200 bootstrap runs resampled by admission.

Figure 5 shows the average model output leading up to the time of unplanned intubation. For each event of unplanned intubation, we estimated the relative risks are based on the cross-validated predicted probabilities from the model. We then averaged the relative risks at each time to event for all events with data. To test that the predicted risk is increasing leading up to the time of unplanned intubation, we tested the null hypothesis that risk estimates are less than or equal to the values for the same patients 8 h prior. Open circles indicate that we reject the null hypothesis at the 0.05 significance level based on a Wilcoxon rank sum test. The figure indicates that all patient types have indications of physiological derangement 4–6 h prior to unplanned intubation.

FIGURE 5

Figure 5. Average time series of model outputs leading up to the time of unplanned intubation. Results are cross-validated predictions from the model based on all patients and are shown for all patients as well as patients of each patient type. Open circles indicate risk estimates are significantly higher than risk estimates for the same patients 8 h prior based on a Wilcoxon signed rank test.

Discussion

Like adults, children have a physiologic signature of illness preceding unplanned intubation in the PICU. Generally, it comprises younger age, and abnormalities in electrolyte, hematologic and vital sign parameters. Given the heterogeneity of the patient population, it is not surprising that there are differences in the presentation among the major patient groups – medical, non-cardiac surgical, and cardiac surgical. We have argued elsewhere that predictive analytics monitoring is not a one-size-fits-all enterprise (30).

There are a variety of indications for unplanned intubation including oxygen failure, ventilation failure, and unstable hemodynamics. For the individual patient, it is often the case that multiple indications for intubation exist, demonstrating the complexity of physiology at play (31). For example, a patient with infantile botulism or spinal muscular atrophy may first exhibit signs of decompensation secondary to neuromuscular weakness which can progress to ventilation failure and ultimately oxygenation failure. Identifying physiologic and biochemical signatures of illness that help discriminate patients at risk for decompensation may allow for earlier intervention and potentially even prevention of the need for intubation.

We observed overlap in the signatures of illness between the medical and non-cardiac surgical patients. The leading indication for intubation in both the medical and non-cardiac surgical patients was respiratory failure (e.g., oxygen and/or ventilation failure) typically driven by primary disorders of respiratory physiology (e.g., pneumonia, acute lung injury). This is best illustrated by the heat map in Figure 2 exhibiting the influence of respiratory rate on the risk of intubation. Cardiac surgical patients, on the other hand, may be more likely to exhibit hemodynamic instability necessitating intubation, distinguishing them from the other two patient types.

This work is presented in the context that the multivariable statistical models that we use to look for signatures of illness can be repurposed for bedside risk prediction to supplement clinical decision making, including potential improvement in patient outcomes (32–36). The precedent is the heart rate characteristics index, a logistic regression model that used time series mathematical measures to identify early signatures of infection in preterm infants (37). Preterm infants in intensive care represent a mostly homogeneous cohort with a consistent signature of deterioration well-suited for logistic regression. The PICU, by contrast, may consist of multiple patient cohorts over a wide range of ages with different physiology and therefore physiological signatures, thus requiring more adaptable modeling approaches.

We found components of the signature in the labs (white blood cell count), vital signs (BP), and from the bedside monitor (every-2-second respiratory rate, variability of the SpO₂). We affirmed the importance of continuous cardiorespiratory monitoring data in discovering dynamic signatures of illness (38–40) excluding it reduced the AUC by 12%. This finding underscores the importance of using all available sources of information in bedside predictive analytics monitoring (41).

Individual chart review proved to be of great importance in defining the signature of illness. When we used a computable phenotype, we found cases in 852 more patient admissions, but the statistical models returned poorer performance with a 29% reduction in AUC. Like sepsis, the gold standard for case identification for model training is individual chart review (42).

Our study has several limitations including the single-center observational nature. Our model was constructed using data from a mixed cardiac/medical-surgical PICU, limiting its applicability broadly to all PICUs and patient populations. We were limited to the time of the intubation, rather than the time the clinical decision to intubate was made, to define the event. We were also limited by the indication(s) for intubation (e.g., respiratory failure, hemodynamic instability) as recorded in the NEAR4KIDS database. To our knowledge, the inter-rater reliability for establishing intubation indication(s) for the NEAR4KIDS database has not been evaluated. The small number of admissions from teenagers, and the low rate of unplanned intubation in that age range, limits the ability to capture a complete signature of illness. Physiological monitoring data included epochs with artifact due to, for example, physical therapy or suctioning. We also did not control for the impact of routine medications, such as sedatives and vasoactives, or non-invasive respiratory support on physiology. We acknowledge that these factors contribute to the complex and nuanced clinical scenarios in the PICU, and that results of a predictive model are to be taken into consideration with many other factors.

Additionally, we were limited to the time of the intubation, rather than the time the clinical decision to intubate was made, to define the event. Several factors may influence the timing of the clinical decision to intubate. While one patient may demonstrate clinical decompensation over a period of hours and another may experience an acute catastrophic event, we were encouraged by the observation of improved AUC at four hours prior to the event. Next steps include external validation of our models and evaluation through prospective integration into clinical care.

We have studied physiological and biochemical dynamics leading up to the event of unplanned intubation in a PICU. We observed that patients who experienced unplanned intubation were younger, more likely to have a co-morbid condition, and more likely to have undergone cardiac surgery. In addition, we identified different signatures of illness for medical, cardiac surgical, and non-cardiac surgical patients. While all this information is already incorporated into clinical practice by PICU teams, using predictive analytics prospectively to accurately estimate risk based on high-dimensional physiological signatures can support clinical decision making, direct new approaches for continuous monitoring in the PICU, and improve patient outcomes.

Data availability statement

The datasets presented in this article are not readily available because the conditions of institutional approval do not provide for transmission of dataset outside of sponsoring institution. Requests to access the datasets should be directed to ms7uw@hscmail.mcc.virginia.edu.

Ethics statement

The studies involving human participants were reviewed and approved by University of Virginia School of Medicine Institutional Review Board. Written informed consent from the participants’ legal guardian/next of kin was not required to participate in this study in accordance with the national legislation and the institutional requirements.

Author contributions

MCS, JRM, DEL and MTC conceived the study. MCS and MAA performed data collection and abstraction. MCS and MTC performed data analyses. MCS, JRM, LPM, JKM, DEK and MTC contributed to analysis interpretation. All authors contributed to the article and approved the submitted version.

Funding

No funding sources to report for the research being submitted. Funds for open access publication fees provided by the Center for Advanced Medical Analytics, University of Virginia School of Medicine.

Conflict of interest

Authors LPM and MTC were employed by company Nihon Kohden Digital Health Solutions. The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fped.2022.1016269/full#supplementary-material.

SUPPLEMENTARY FIGURE A1

Histogram of time between ventilator respiratory rate entries in the electronic medical record on a logarithmic scale. Most entries are within 12 hours of the preceding entry.

SUPPLEMENTARY FIGURE A2

Empirical relative risk for cardiac surgery (left), medical (center), and non-cardiac surgery (right) patients as a function of age and each feature. On each panel, younger patients are to the left, and lower values of the feature are to the bottom. Each colored tile estimates the relative risk of unplanned intubation for each decile of age and the feature based on the surrounding quintile. Deciles remain the same across patient type panels. Higher relative risk is redder while lower is bluer.

SUPPLEMENTARY FIGURE A3

Marginal risk profiles for each feature in the random forest model. Note that marginal risk profiles average out any interdependence between features, such as the dependence of heart rate on age. The (natural) log odds of unplanned intubation in the next 12 hours (ordinate) is shown as a function of the value of each measured variable (abscissa) holding all other features at their median values.

References

1. Schneider J, Sweberg T. Acute respiratory failure. Crit Care Clin. (2013) 29:167–83. doi: 10.1016/j.ccc.2012.12.004

PubMed Abstract | CrossRef Full Text | Google Scholar

2. Carroll CL, Spinella PC, Corsi JM, Stoltz P, Zucker AR. Emergent endotracheal intubations in children: be careful if it is late when you intubate. Pediatr Crit Care Med. (2010) 11:343–8. doi: 10.1097/PCC.0b013e3181ce6d19

PubMed Abstract | CrossRef Full Text | Google Scholar

3. Politano AD, Riccio LM, Lake DE, Rusin CG, Guin LE, Josef CS, et al. Predicting the need for urgent intubation in a surgical/trauma intensive care unit. Surgery. (2013) 154:1110–6. doi: 10.1016/j.surg.2013.05.025

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Davis JP, Wessells DA, Moorman JR. Coronavirus disease 2019 calls for predictive analytics monitoring-a new kind of illness scoring system. Crit Care Explor. (2020) 2:e0294. doi: 10.1097/CCE.0000000000000294

PubMed Abstract | CrossRef Full Text | Google Scholar

5. Bates DW, Saria S, Ohno-Machado L, Shah A, Escobar G. Big data in health care: using analytics to identify and manage high-risk and high-cost patients. Health Aff. (2014) 33:1123–31. doi: 10.1377/hlthaff.2014.0041

CrossRef Full Text | Google Scholar

6. Bates DW, Zimlichman E. Finding patients before they crash: the next opportunity to improve patient safety. BMJ Qual Saf. (2015) 24:1–3. doi: 10.1136/bmjqs-2014-003499

PubMed Abstract | CrossRef Full Text | Google Scholar

7. Moss TJ, Lake DE, Calland JF, Enfield KB, Delos JB, Fairchild KD, et al. Signatures of subacute potentially catastrophic illness in the ICU: model development and validation. Crit Care Med. (2016) 44:1639–48. doi: 10.1097/CCM.0000000000001738

PubMed Abstract | CrossRef Full Text | Google Scholar

8. Callcut RA, Xu Y, Moorman JR, Tsai C, Villaroman A, Robles AJ, et al. External validation of a novel signature of illness in continuous cardiorespiratory monitoring to detect early respiratory deterioration of ICU patients. Physiol Meas. (2021) 42:095006. doi: 10.1088/1361-6579/ac2264

CrossRef Full Text | Google Scholar

9. Ruminski CM, Clark MT, Lake DE, Kitzmiller RR, Keim-Malpass J, Robertson MP, et al. Impact of predictive analytics based on continuous cardiorespiratory monitoring in a surgical and trauma intensive care unit. J Clin Monit Comput. (2019) 33:703–11. doi: 10.1007/s10877-018-0194-4

PubMed Abstract | CrossRef Full Text | Google Scholar

10. Trujillo Rivera EA, Chamberlain JM, Patel AK, Morizono H, Heneghan JA, Pollack MM. Dynamic mortality risk predictions for children in ICUs: development and validation of machine learning models. Pediatr Crit Care Med. (2022) 23:344–52. doi: 10.1097/PCC.0000000000002910

PubMed Abstract | CrossRef Full Text | Google Scholar

11. von Elm EAD, Egger M, Pocock SJ, Gøtzsche PC, Vandenbroucke JP. The strengthening the reporting of observational studies in epidemiology (STROBE) statement: guidelines for reporting observational studies. Ann Intern Med. (2007) 147:573–7. doi: 10.7326/0003-4819-147-8-200710160-00010

PubMed Abstract | CrossRef Full Text | Google Scholar

12. Moss TJ, Clark MT, Lake DE, Moorman JR, Calland JF. Heart rate dynamics preceding hemorrhage in the intensive care unit. J Electrocardiol. (2015) 48:1075–80. doi: 10.1016/j.jelectrocard.2015.08.007

PubMed Abstract | CrossRef Full Text | Google Scholar

13. Lake DE, Moorman JR. Accurate estimation of entropy in very short physiological time series: the problem of atrial fibrillation detection in implanted ventricular devices. Am J Physiol Heart Circ Physiol. (2011) 300:H319–25. doi: 10.1152/ajpheart.00561.2010

PubMed Abstract | CrossRef Full Text | Google Scholar

14. Peng CK, Havlin S, Stanley HE, Goldberger AL. Quantification of scaling exponents and crossover phenomena in nonstationary heartbeat time series. Chaos. (1995) 5:82–7. doi: 10.1063/1.166141

PubMed Abstract | CrossRef Full Text | Google Scholar

15. Carrara M, Carozzi L, Moss TJ, de Pasquale M, Cerutti S, Ferrario M, et al. Heart rate dynamics distinguish among atrial fibrillation, normal sinus rhythm and sinus rhythm with frequent ectopy. Physiol Meas. (2015) 36:1873–88. doi: 10.1088/0967-3334/36/9/1873

PubMed Abstract | CrossRef Full Text | Google Scholar

16. Clark MT, Rusin CG, Hudson JL, Lee H, Delos JB, Guin LE, et al. Breath-by-breath analysis of cardiorespiratory interaction for quantifying developmental maturity in premature infants. J Appl Physiol. (2012) 112:859–67. doi: 10.1152/japplphysiol.01152.2011

PubMed Abstract | CrossRef Full Text | Google Scholar

17. Li Q, Mark RG, Clifford GD. Robust heart rate estimation from multiple asynchronous noisy sources using signal quality indices and a Kalman filter. Physiol Meas. (2008) 29:15–32. doi: 10.1088/0967-3334/29/1/002

PubMed Abstract | CrossRef Full Text | Google Scholar

18. Moss TJ, Clark MT, Calland JF, Enfield KB, Voss JD, Lake DE, et al. Cardiorespiratory dynamics measured from continuous ECG monitoring improves detection of deterioration in acute care patients: a retrospective cohort study. PLoS One. (2017) 12:e0181448. doi: 10.1371/journal.pone.0181448

PubMed Abstract | CrossRef Full Text | Google Scholar

19. Gadrey SM, Lau CE, Clay R, Rhodes GT, Lake DE, Moore CC, et al. Imputation of partial pressures of arterial oxygen using oximetry and its impact on sepsis diagnosis. Physiol Meas. (2019) 40:115008. doi: 10.1088/1361-6579/ab5154

PubMed Abstract | CrossRef Full Text | Google Scholar

20. Gadrey SM, Mohanty P, Haughey SP, Jacobsen BA, Dubester KJ, Webb KM, et al. Overt and occult hypoxemia in patients hospitalized with novel coronavirus disease 2019. medRxiv. (2022) 2022:06.14.22276166. doi: 10.1101/2022.06.14.22276166

PubMed Abstract | CrossRef Full Text | Google Scholar

21. Toddenroth D, Ganslandt T, Castellanos I, Prokosch HU, Burkle T. Employing heat maps to mine associations in structured routine care data. Artif Intell Med. (2014) 60:79–88. doi: 10.1016/j.artmed.2013.12.003

PubMed Abstract | CrossRef Full Text | Google Scholar

22. Liaw AWM. Classification and regression by random forest. R News. (2002) 2:18–22. ISSN: 1609-3631

Google Scholar

23. R Core T. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing 2017.

24. Spaeder MC, Moorman JR, Tran CA, Keim-Malpass J, Zschaebitz JV, Lake DE, et al. Predictive analytics in the pediatric intensive care unit for early identification of sepsis: capturing the context of age. Pediatr Res. (2019) 86:655–61. doi: 10.1038/s41390-019-0518-1

PubMed Abstract | CrossRef Full Text | Google Scholar

25. Sanchez-Pinto LN, Luo Y, Churpek MM. Big data and data science in critical care. Chest. (2018) 154:1239–48. doi: 10.1016/j.chest.2018.04.037

PubMed Abstract | CrossRef Full Text | Google Scholar

26. Breiman L. Random forests. Mach Learn. (2001) 45:5–32. doi: 10.1023/A:1010933404324

CrossRef Full Text | Google Scholar

27. Keim-Malpass J, Enfield KB, Calland JF, Lake DE, Clark MT. Dynamic data monitoring improves predictive analytics for failed extubation in the ICU. Physiol Meas. (2018) 39:075005. doi: 10.1088/1361-6579/aace95

PubMed Abstract | CrossRef Full Text | Google Scholar

28. Moorman JR, Carlo WA, Kattwinkel J, Schelonka RL, Porcelli PJ, Navarrete CT, et al. Mortality reduction by heart rate characteristic monitoring in very low birth weight neonates: a randomized trial. J Pediatr. (2011) 159:900–6. doi: 10.1016/j.jpeds.2011.06.044

PubMed Abstract | CrossRef Full Text | Google Scholar

29. Griffin MP, O'Shea TM, Bissonette EA, Harrell FE Jr., Lake DE, Moorman JR. Abnormal heart rate characteristics are associated with neonatal mortality. Pediatr Res. (2004) 55:782–8. doi: 10.1203/01.PDR.0000119366.21770.9E

PubMed Abstract | CrossRef Full Text | Google Scholar

30. Blackwell JN, Keim-Malpass J, Clark MT, Kowalski RL, Najjar SN, Bourque JM, et al. Early detection of in-patient deterioration: one prediction model does not fit all. Crit Care Explor. (2020) 2:e0116. doi: 10.1097/CCE.0000000000000116

PubMed Abstract | CrossRef Full Text | Google Scholar

31. Nishisaki A, Turner DA, Brown CA, Walls RM, Nadkarni VM. National Emergency Airway Registry for Children, A national emergency airway registry for children: landscape of tracheal intubation in 15 PICUs. Crit Care Med. (2013) 41:874–85. doi: 10.1097/CCM.0b013e3182746736

PubMed Abstract | CrossRef Full Text | Google Scholar

32. Sanchez-Pinto LN, Bennett TD. Evaluation of machine learning models for clinical prediction problems. Pediatr Crit Care Med. (2022) 23:405–8. doi: 10.1097/PCC.0000000000002942

PubMed Abstract | CrossRef Full Text | Google Scholar

33. Zimmet AN, Clark MT, Gadrey SM, Bell TD, Zimmet AM, Moorman JR, et al. Pathophysiologic signatures of bloodstream infection in critically ill adults. Crit Care Explor. (2020) 2:e0191. doi: 10.1097/CCE.0000000000000191

PubMed Abstract | CrossRef Full Text | Google Scholar

34. Moorman JR. The principles of whole-hospital predictive analytics monitoring for clinical medicine originated in the neonatal ICU. NPJ Digit Med. (2022) 5:41. doi: 10.1038/s41746-022-00584-y

PubMed Abstract | CrossRef Full Text | Google Scholar

35. Badawi O, Liu X, Hassan E, Amelung PJ, Swami S. Evaluation of ICU risk models adapted for use as continuous markers of severity of illness throughout the ICU stay. Crit Care Med. (2018) 46:361–7. doi: 10.1097/CCM.0000000000002904

PubMed Abstract | CrossRef Full Text | Google Scholar

36. Escobar GJ, Liu VX, Schuler A, Lawson B, Greene JD, Kipnis P. Automated identification of adults at risk for in-hospital clinical deterioration. N Engl J Med. (2020) 383:1951–60. doi: 10.1056/NEJMsa2001090

PubMed Abstract | CrossRef Full Text | Google Scholar

37. Griffin MP, Lake DE, Bissonette EA, Harrell FE Jr., O'Shea TM, Moorman JR. Heart rate characteristics: novel physiomarkers to predict neonatal infection and death. Pediatrics. (2005) 116:1070–4. doi: 10.1542/peds.2004-2461

PubMed Abstract | CrossRef Full Text | Google Scholar

38. Holder AL, Clermont G. Using what you get: dynamic physiologic signatures of critical illness. Crit Care Clin. (2015) 31:133–64. doi: 10.1016/j.ccc.2014.08.007

PubMed Abstract | CrossRef Full Text | Google Scholar

39. Slight SP, Franz C, Olugbile M, Brown HV, Bates DW, Zimlichman E. The return on investment of implementing a continuous monitoring system in general medical-surgical units. Crit Care Med. (2014) 42:1862–8. doi: 10.1097/CCM.0000000000000340

PubMed Abstract | CrossRef Full Text | Google Scholar

40. Horton WB, Barros AJ, Andris RT, Clark MT, Moorman JR. Pathophysiologic signature of impending ICU hypoglycemia in bedside monitoring and electronic health record data: model development and external validation. Crit Care Med. (2022) 50:e221–30. doi: 10.1097/CCM.0000000000005171

PubMed Abstract | CrossRef Full Text | Google Scholar

41. Monfredi O, Keim-Malpass J, Moorman JR. Continuous cardiorespiratory monitoring is a dominant source of predictive signal in machine learning for risk stratification and clinical decision support. Physiol Meas. (2021) 42:090301. doi: 10.1088/1361-6579/ac2130

CrossRef Full Text | Google Scholar

42. Rhee C, Dantes R, Epstein L, Murphy DJ, Seymour CW, Iwashyna TJ, et al. Incidence and trends of sepsis in US hospitals using clinical vs claims data, 2009-2014. JAMA. (2017) 318:1241–9. doi: 10.1001/jama.2017.13836

PubMed Abstract | CrossRef Full Text | Google Scholar