Vanshika Vats1†
Aditya Nagori1,2,3†
Pradeep Singh1†
Raman Dutt4‡
Harsh Bandhey1‡
Mahika Wason1‡
Rakesh Lodha5
Tavpritesh Sethi1,5*- 1Indraprastha Institute of Information Technology, Delhi, India
- 2CSIR-Institute of Genomics and Integrative Biology, New Delhi, India
- 3Academy of Scientific and Innovative Research (AcSIR), Ghaziabad, India
- 4Computer Science and Engineering, Shiv Nadar University, Greater Noida, India
- 5Department of Pediatrics, All India Institute of Medical Sciences, New Delhi, India
Shock is one of the major killers in intensive care units, and early interventions can potentially reverse it. In this study, we advance a noncontact thermal imaging modality for continuous monitoring of hemodynamic shock working on 1,03,936 frames from 406 videos recorded longitudinally upon 22 pediatric patients. Deep learning was used to preprocess and extract the Center-to-Peripheral Difference (CPD) in temperature values from the videos. This time-series data along with the heart rate was finally analyzed using Long-Short Term Memory models to predict the shock status up to the next 6 h. Our models achieved the best area under the receiver operating characteristic curve of 0.81 ± 0.06 and area under the precision-recall curve of 0.78 ± 0.05 at 5 h, providing sufficient time to stabilize the patient. Our approach, thus, provides a reliable shock prediction using an automated decision pipeline that can provide better care and save lives.
1 Introduction
Hemodynamic shock is an acute and initially reversible condition that can act as an early signal to future-end organ failure (Herget-Rosenthal et al., 2018). It manifests due to inadequate blood perfusion than what is essentially required. Such a condition could cause tissue malfunction leading to rapid organ deterioration which can eventually result in death. The mortality rate due to delay in detection and treatment of such shock is as high as 34% in ICU patients admitted in developing countries (Divatia et al., 2016). Thus, proactive monitoring and therapy of hemodynamic shock can prevent the risk of impending organ failure and mortality (Rivers et al., 2001; Early Goal-Directed Therapy Collaborative Group of Zhejiang Province, 2010; Vincent and De Backer, 2013). However, delayed or insufficient monitoring can increase the risk of hemodynamic shock. Modern ICUs are equipped with a large number of sensors that generate humongous amounts of data besides a large number of clinical investigations. ICU clinicians and nursing staff are exposed to this large quantity of data on a real time basis. The overall burnout can go as high as 80.5% which can contribute to delayed or insufficient care (Rotenstein, 2018). Moreover, the resource-limiting areas have a higher risk of shock ignorance mainly due to a lack of technological advancement and a low doctor-to-patient ratio (Bagcchi, 2015). Most of the methods dealing with the shock in current times are invasive or require repeated contact, making the patient prone to hospital-acquired infections. The noninvasive ways such as Noninvasive Blood Pressure monitoring and Ultrasonography are not continuous but require contact with the delicate skin of infants which are infection-prone too (Oranges et al., 2015). Techniques such as Electrocardiography (ECG) and photoplethysmography (PPG) (Liu et al., 2020; Nachman et al., 2020) can also provide cardiovascular parameters such as the respiratory rate and heart rate in a noninvasive and wireless way to serve as important factors for shock monitoring. However, their contact-based approach can prove to be distressing and epidermal damaging for the fragile skin of neonatal patients (Barbosa Pereira et al., 2018). Thus, the use of artificial intelligence (AI) and frugal open-source technologies such as low-cost noncontact thermal monitoring can be effective for shock management.
We have created a body thermal imaging data cohort for the pediatric population under the Safe ICU initiative which warehouses 1.5 million hours of physiological vital data, laboratory investigation, treatment charts, doctors, and nurse investigation charts (Sethi et al., 2017). We have used these data to build automated hemodynamic monitoring and prediction of the future onset of the hemodynamic shock pipeline. Nowadays, nonintrusive vital monitoring measures have been gaining popularity because of their feasibility. Measures such as quantified capillary refill time (Q-CRT) and peripheral perfusion index (PPI) are being used for hemodynamic monitoring as noninvasive and transmitted light-based methods for assessing peripheral perfusion (Falotico et al., 2020). Vitals such as respiration and heartbeat have been known to be measured noninvasively using a twin-core fiber (TCF)-based sensor (Tan et al., 2019). Some studies have been successful in tracking peripheral blood oxygen saturation (SpO2) using broad-band lighting and an RGB camera (Guazzi et al., 2015). Balakrishnan et al. (2013) take just an RGB video as input and are able to detect heart rate and beat lengths by monitoring the subtle head movements. Heart rate can also be measured through videos using DeepPhys (Chen and McDuff, 2018) and Eulerian video magnification (EVM) combined with wavelet transformation (Froesel et al., 2020).
Thermal imaging, another nonintrusive and noncontact modality, is now increasingly being used in medical literature to extract the temperature information from the body and used for studies such as breast cancer detection (Ng, 2009; Mambou et al., 2018) and even diabetes mellitus (type 1) detection (Selvarani and Suresh, 2019). Thermography of the skin surfaces to study blood flow, cardiac pulse, and breath rate (Pavlidis et al., 2007) is being used as a basis for broader works such as preliminary hypertension assessment (Thiruvengadam and Mariamichael, 2018). Thermography of the human hand and foot skin surfaces has also been used to study blood flow using spectral filtering (Sagaidachnyi et al., 2017). A similar use of the thermography sensor has been proposed to make the disease activity detection in rheumatoid arthritis patients (Pauk et al., 2019). Lately, thermal imaging has also been used to detect the respiratory rate of humans (Liu et al., 2019). For instance, in cardiorespiratory signal monitoring, Barbosa Pereira et al. (2018) use thermal fluctuations beneath the nostrils for measuring the respiratory cycle. Since human bodies can be considered black bodies with an emissivity value of the skin close to 0.98 (Bagavathiappan et al., 2009), the infrared energy, and hence temperature information, can be suitably tapped by IR detectors such as Thermovision-550 (Thiruvengadam and Mariamichael, 2018), MWIR-Pheonix-FLIR (FLIR Systems, 2016), and SeekThermal®(Seek Thermal, 2022) easily available in the market. Even though some of the aforementioned techniques might incur a higher cost due to the nature of the thermal imaging sensors used, our study uses a simple SeekThermal smartphone camera attachment which can feasibly be used to capture the thermal context of the body. During hemodynamic instability, redistribution of blood occurs which causes a temperature difference in the center and periphery of the body, which acts as one of the first possible indicators of the advent of shock (Lima and Bakker, 2005; Lima et al., 2009; van Genderen et al., 2012; Ortiz-Dosal et al., 2014). Our study leverages this temperature difference, that is, Centre-to-Peripheral Difference (CPD) (Bourcier et al., 2016; Houwink et al., 2016), to study hemodynamic shock with the abdomen as center and feet as peripheral regions (Lima and Bakker, 2005; Hasanin et al., 2017). The difference observed is more in the case of a shock than of no-shock (Lima and Bakker, 2005; Nagori et al., 2019). We chose SeekThermal® Camera for our study as it has affordable, minimal hardware architecture and can be easily mounted on a smartphone. CPD has been exploited along with some vitals and the use of machine learning methods such as Histogram of Oriented Gradients features with Random Forest classifier for detection and prediction of shock with a single snapshot of an image at a time point (Nagori et al., 2019), but a single image snapshot would present only limited information as it may be an outlier caused by random fluctuation. This may reduce the robustness and reliability of the model output. In addition, using the handcrafted features for machine learning algorithms might cost us time. To expand upon the study, we aim to reduce the manual tasks performed to calculate the features using deep learning methods and work on the continuous time-series data, improving the efficacy of detection and prediction.
In this study, we developed a future risk prediction model (Figure 1) using continuously streamed videos of body infrared patterns and state-of-the-art computer vision (CV) and deep-learning (DL) techniques. Thus, the aim of our work was to build an automated, minimal contact pipeline for proactive hemodynamic monitoring. Automated detection and prediction will help prevent the adverse outcomes related to hemodynamic instability. Hence, the impact this problem creates and the scope of extending the solution make this study worthwhile in saving the lives of many pediatric patients.
FIGURE 1. Shock prediction steps. The summary of the shock pipeline shows the steps from video frame extraction to shock prediction. Step 1 comprises sampling videos to extract frames. Step 2 classifies frames into covered or uncovered, while also finding the presence of multiple people in the frame and mask them, to avoid confusion. The masked images are then input to the ResUNet based segmentation model and CPD is hence extracted. The series of CPDs are then passed through a time-series sequence classifier, and finally, the predictions are made for shock for the next 6 h.
2 Materials and Methods
For this study, we carried out thermal monitoring as a part of the SafeICU (Sethi et al., 2017) warehouse. We constructed an automated CPD extraction pipeline by body part segmentation on thermal videos using deep-learning models. Each time-point was labeled for shock and non-shock using an age-specific shock-index (Acker et al., 2017). The time-series data of the extracted CPD signals, combined with the heart rate and the labels, were preprocessed for case-control cohort design. We constructed cohorts at 0 h for detection and 1–6 h lead-time for prediction tasks. We then used multivariate Long Short Term Memory (LSTM) (Hochreiter and Schmidhuber, 1997) models to detect and predict the future onset of the shock. We finally evaluated the performance metrics of the models using a 10-fold cross-validation method.
2.1 Data Collection and Study Design
2.1.1 SafeICU Framework
All the data reported in this research were collected from the Pediatric ICU of AIIMS, New Delhi, a tertiary care hospital in India. There were eight beds including neonatal beds in the pediatric ICU. We set up our servers to collect the real time physiological vitals periodic data. Vitals (heart rate, respiratory rate, and blood pressure) were routinely monitored using electronic sensors, that is, the heart rate was measured by ECG sensors, blood pressure was measured using arterial catheters, and respiratory rate and oxygen saturation were measured using pulse oximeter probes. Socket programming (network protocol) based on in-house software applications was used to extract the data from the Mindray™ central monitoring station (CMS). However, with an exception of heart rate, our study does not require these contact-based vitals for shock prediction, thus making the study of minimal contact. We also warehoused laboratory investigations, daily doctor and nurse notes, treatment charts, and thermal imaging.
2.1.2 Ethical Approval and Patient Consent
The study was carried out with the approval of the Pediatric Intensive Care Unit of All India Institute of Medical Sciences, New Delhi, India. Since thermal images only capture infrared radiation, these did not reveal the patient’s identity and the study did not involve any contact or change in routine patient care. Hence, a waiver of consent was sought and granted by the Institute Ethics Committee (Ref. No. IEC/NP-211/08.05.2015, AA-2/09.02.2017). All experiments were performed in accordance with relevant guidelines and regulations as approved by the ethics committee.
2.1.3 Cohort Based on Binary Shock Index
The SAFE-ICU described earlier has warehoused over 1.5 million patient hours of the monitoring data from the PICU. It is used to extract the time-stamped vital data for the patients at 0–6 h heart rate and blood pressure recordings. The shock index (SI) was calculated as the ratio of the median heart rate and median arterial systolic blood pressure, considered the gold standard for BP measurement in critical care settings (Pittman et al., 2004; Lehman et al., 2013). SI was computed over the median of moving sequential windows of 30 data points at a resolution of 15 s. Shock-index pediatric age-adjusted (SIPA) is used to compute shock/no-shock age-specific binarized outcomes for each patient (Acker et al., 2017).
2.1.4 Thermal Imaging
Seek Thermal Compact (UW-AAA) with a resolution of 480*640 was used to capture thermal videos at 15 fps on iron theme settings. The camera, having a thermal sensor of 206*156 pixels was kept at a median (IQR) distance of 2.0828 m (0.1524) for long beds and 1.524 m (0.2665) for small beds. The ICU room temperature was maintained at 25°C. Temperature calibration was performed by matching the imager temperature with other known sources. Temperature from the thermal camera was found to be accurate with temperature measurement from other sources such as temperature probes and thermometers. Standard thermal video capturing and operating procedures were followed in order to ensure minimal disturbances by the extraneous factors, say, patient positioning, device handling, etc. (Supplementary Methods S1). Thermal cameras only capture infrared radiation so as to make sure that study does not reveal the patient’s identity. The camera was placed properly and at a good distance from the patient so that there was neither direct contact involved nor any change in patient routine care. The thermal videos were captured in a standard color scale guaranteeing that the full body of infants was visible. Thermal videos of every single patient were collected at different time points on different days. Thus each patient possibly has different values for shock status, which in turn eliminates bias due to the patient’s propensity characteristics, say gender, age, etc. Thermal artifacts were a lesser concern as we were calculating the relative gradient of temperature instead of the absolute temperature. Vital data with respect to the time-stamp of the videos were extracted from the data warehouse at 15-s intervals (SAFE-ICU) (Sethi et al., 2017). A comparison was made between the shock and non-shock groups using either the Wilcoxon rank-sum test or two-tailed Student’s t-test after testing for normality by the D'Agostino-Pearson normality test using GraphPad Prism version 6.00, GraphPad Software, La Jolla California USA.
2.2 Classification Into Covered and Uncovered
The patients in ICU are kept under observation for a long duration. Being a very critical area, the patients are kept covered by a blanket most of the time, which is removed only for a short period, generally when a nurse/doctor comes to provide care. A ResNet-152 (He et al., 2016) architecture was trained using PyTorch (Paszke et al., 2019) framework in Python3 (Python Software Foundation, 2018) to classify each frame into covered and uncovered, i.e., abdomen and feet are visible (Figure 2). To train the data, images were augmented and normalized by their mean and variance especially extracted out to suit the thermal data. The model was finally implemented on videos sampled at 1 fps.
FIGURE 2. Shock prediction pipeline. Illustration of the pipeline followed for the detection and prediction of shock and no-shock. Each frame of the video was examined for uncovered/covered. The uncovered frames were then filtered off the presence of people other than the patient present in the frame. The frame was finally passed from the segmentation and CPD extraction model, further collecting the sequences used for LSTM time-series classification. The time sequences had CPD and heart rate as features and an appropriate window length of 256 s was chosen. Since the data were highly imbalanced, the SMOTE upsampling method was used in training the LSTM model. The detection and prediction of shock were carried out at 0 h and for the next 6 h, respectively.
2.3 Multiple Person Detection
In the ICUs, caregivers providing care to the patient might come into the field of the camera mounted over the bed. For the CPD extraction task, bounding-box detection of multiple persons was used to filter out this additional presence so as not to confuse the algorithm. To avoid loss of crucial data where the caregiver and patient may be in the same frame, the bounding box corresponding to the larger area, that is, the patient was kept and the rest was masked. This region of interest was used for CPD extraction. For training purposes, a variety of images of the patient alone and along with the caretaker were manually annotated with bounding boxes using Yolo-Annotation-Tool-New (Murugavel, 2019) and augmented with linear transformations and Gaussian noise and passed through YOLOv3 (Redmon and Farhadi, 2018) based on Darknet-53 with a confidence threshold of 0.4.
The architecture had to be retrained from scratch as thermal data are very different from the COCO dataset, used in the original YOLOv3 manuscript. The YOLOv3’s output was the dimensions and position of the predicted bounding boxes detecting the people present in the frame, the correctness of which was verified through Intersection over Union (IoU) metric by calculating the ratio of overlapping of the predicted and ground truth bounding boxes with their union area.
2.4 Segmentation and CPD Extraction From Abdomen and Feet
Nagori A. et al. proved that the probability of shock depends directly on CPD. For this study, the abdomen was taken as the center and the peripheral was the foot. Images were annotated manually pixel-wise using js-annotator-tool. The target maps contained three one-hot encoded layers corresponding to the abdomen, feet, and background. The input images were normalized with mean and variance especially extracted from the distribution of the dataset in use. Appropriate image padding was carried out to ensure that the aspect ratio of images remains the same in case of any change in the input dimension. To account for a low dataset of pixel-wise segmented images for training, a ResUNet with ResNet-18, pretrained on ImageNet, was used as an encoder. UNet (Ronneberger et al., 2015) is specifically introduced to segment the less abundantly found medical data, helps to gather more local and global information even in the dearth of data, and thus efficiently segmenting out the images. A cutoff threshold was set on the predicted outputs to remove any weakly predicted pixels. The area detected was used as a region of interest in the original image and the mode of the detected probabilities was taken as the point of temperature extraction from the segmented-out abdomen and feet. The difference was divided by the abdomen value to keep CPD robust from thermal noise.
2.5 LSTM Time Series Sequence Classification
The videos were sampled at 1 fps to extract the CPD data from every uncovered window possible. Windows of 256 data-points corresponding to 256 s (4.26 min, padded, if necessary) were taken as an input to the LSTM-based classifier. The windows greater than 256 were split in an overlapping fashion. Each CPD, along with the heart rate at its corresponding time point was taken to finally label it with the shock index, and hence, the presence of shock/no-shock. The missing heart rate data at certain points were imputed with linear interpolation if the missing data were less than 10% of the time series length. Since the data are highly imbalanced with more non-shock sequences, the train-set is augmented with SMOTE (Chawla et al., 2002). The LSTM sequence classifier was followed by a series of dense layers with a dropout of 0.2, which then passed through a sigmoid layer to output binary shock index, and hence, shock/no-shock.
Experiments on tsfresh (Christ et al., 2016) feature classification with linear-mixed-effects models (Bates et al., 2015) and random-forests (Cutler et al., 2012), and the direct end-to-end classification of images/videos are shown in Supplementary Methods S2.
2.6 Outcome Variable—Binary Shock Index
The SAFE-ICU initiative has enabled this research to gather the PICU data and extract vitals and corresponding timestamps at the 0th hour (time of video capturing) and the next 6 h. The shock index was taken as the median heart rate and the median arterial systolic blood pressure, for moving sequential windows of 30 data points at a resolution of 15 s. SIPA (Acker et al., 2017) was then used to compute the age-specific binary outcome for each patient.
2.7 Model Evaluation
The video data were first partitioned patient-wise such as to keep train, validation, and test sets unseen from each other. For the 10-fold cross-validation, the data were partitioned with a ratio of 60:20:20 into these three sets in a stratified manner, that is, keeping the distribution of low-percentage shock class comparable in all three sets. Training data was augmented for the low-found shock class using SMOTE; the validation and test sets remain unchanged in their size in each respective fold. The model analysis was mostly conducted on the area under precision-recall curve (AUPRC) and the area under receiver operating characteristic (AUROC) curve. Other standard metrics like F1-score, PPV, NPV, specificity, and sensitivity were evaluated at the Youden’s Index (J) (Ruopp et al., 2008). Since there is a high significance of prevalence in the medical domain, calculating the metrics at the Youden’s Index becomes important.
3 Results
3.1 Patient Characteristics, Preprocessing, and Cohort Building
Statistical inferences of the cohort characteristics between shock and non-shock groups are depicted in Table 1. Shock and Non-shock conditions were derived using the age-specific shock index cutoff values. It can be observed that the most significant difference between the two categories is in the heart rate, as expected, along with the respiratory rate. We obtained a median length of stay of 6.808 days with a total of 132 shock instances and 274 non-shock instances. The study uses arterial systolic blood pressure, the gold standard for BP measurement in critical care settings (Pittman et al., 2004; Lehman et al., 2013), from 22 patients monitored longitudinally as a required vital for the calculation of shock index for the training purposes which is available for only a small subset of patients. However, once the model is trained, this is no longer a limitation at inference time as the trained model does not require blood pressure for inference (making a prediction of shock status). The pipeline uses >1,00,000 frames and the corresponding shock indices for training the prediction models.
TABLE 1. Cohort characteristics and statistical significance of control (non-shock) vs. affected (shock) classes. The p-values were calculated using either the Wilcoxon rank sum test (W) or Student’s t-test (t) after testing for normality by the D’Agostino-Pearson normality test. (n, number of sequences; IQR, interquartile range).
3.2 Segmentation of Abdomen and Feet Achieved a Total Dice Loss of 0.0391 Using ResUNet
Since thermal images lack texture, it is important for the model to recognize the structural shape features. ResUNet was specifically used to make the task possible on relatively less available thermal data and to capture both the local and global information from the image. A dice loss of 0.0391 (dice coefficient = 0.9609) with a Binary Cross-Entropy loss of 0.0692 was achieved for segmenting out the abdomen, feet, and background. The mode intensities of segmented areas were then used to find out CPD and hence build the longitudinal models from continuous long-duration videos. The results are showcased in Supplementary Table S1.
3.3 The LSTM Model Was Found to be the Best Performing
We compared three models to finally arrive at the best-performing one. Linear-Mixed Effects (Bates et al., 2015), Random Forest (Cutler et al., 2012), and LSTM (Hochreiter and Schmidhuber, 1997) were tested at various time points from the observation taken. Based on various metric evaluations, LSTM was found to be the best performing on our given time series data and hence was chosen as the primary choice for our study. The F1 score comparison of the three models is shown in Figure 3, and the rest of the evaluations are shown in Supplementary Figure S1.
FIGURE 3. Quality assessment of models. The F1 score comparison for the three models (i.e., Linear Mixed-Effects model, Random Forest, and LSTM) tested on (A) CPD and heart rate and (B) CPD parameter only. It can be observed that the LSTM model outperforms the other two models in both cases, making it the primary choice of this research. The rest of the comparison plots are depicted in Supplementary Figure S1.
3.4 Evaluation of the LSTM Model at Different Lead Times Reveals That CPD Increases the Model Performance
The SAFE-ICU resources allowed us to match the time sequences with their corresponding states of shock/non-shock for the next 6 h since the observation was taken. We tested the LSTM model performance for 1–6 h of lead time on CPD and the heart rate as shown in Table 2. As shown in Figure 4, we got the best performance at AUROC 0.81 ± 0.06 and AUPRC 0.78 ± 0.05 at 5 h. This gives a good time window to alert the medical practitioners to start the therapy for preventing the advent of shock.
TABLE 2. Performance of the proposed model predicting the presence of shock/non-shock using automated CPD and heart rate. The time pt. column depicts the subsequent hours from the time of taking the observation, at which the results were recorded. The unequal number of shock and non-shock sequences is due to the absence of patient data with the increasing number of hours. (S/NS, number of shock/non-shock sequences present; AUPRC, area under precision-recall curve; AUROC, area under receiver operating characteristics; PPV, positive predictive value; NPV, negative predictive value; D, detection; P, prediction).
FIGURE 4. Quality evaluation on the LSTM model. (A) Quality evaluation of the LSTM time-series classification models at lead points of up to 6 h. The best performance of (B) AUPRC and (C) AUROC was obtained at a 5-h prediction. The rest of the performance metric results are demonstrated in Supplementary Figure S2. The results of all are shown in Table 2. The standard error (SE) for each is calculated from cross-validation, by taking k = 10.
4 Discussion
The study presents a computer vision and deep-learning-based continuous and noncontact shock detection and prediction model which leverages Center-to-Peripheral Difference (CPD) as one of its main parameters. Care needs to be taken to evaluate the parameters in a noninvasive way such as not to cause infections by contact. The role of noncontact, wireless health monitoring has gained much importance, especially during the times of the COVID-19 pandemic. The noncontact estimation of multiple physiological parameters can enable telediagnosis and remote management of patients, at the same time decreasing the contact-based infection risk. Among the available noncontact modalities, recent advancements in thermal imaging have made thermal imaging a key technology. Affordable smartphone thermal cameras have been used to detect the breathing rate and hence the perceived mental stress (Cho et al., 2019) and even help identify sports injuries (Quintana et al., 2015). In ICUs, studies providing noncontact measurements such as ECG (Majumder et al., 2018) and other physiological measurements using noncontact cameras (Jorge et al., 2022) have been explored. Wireless sensors have shown potential for the detection of complex underlying neurological disorders, such as Shah et al. (2019) use wireless sensor networks (WSN) for monitoring pill-rolling tremors which can be proven to be a precursor to Parkinson’s disease. Similarly, Liu et al. (2018) use WSN to detect and monitor abnormal breathing patterns as an underlying condition for more gruesome diseases like hyperthyreosis and sleep apnea. Since the field of thermal image inspection has only started to be explored, it can thus be leveraged, along with some noninvasively monitored vital parameters, for the prediction of shock. The use of deep learning methods proves to be really beneficial in reducing manual preprocessing and increasing the accuracy of methods.
In this study, we have extended our previous study on hemodynamic shock prediction with longitudinal continuous monitoring of body thermal patterns which are found to be predictive for future shock prediction (Nagori et al., 2019). Longitudinal monitoring of temperature gradient opens up a rich source of information about patient physiology. The underlying stochastic patterns can have a discriminative value for future hemodynamic shock risk. We leveraged these reasonings to extract the center-to-peripheral intensity difference from the thermal images in a time-series fashion. We do so by applying our data-specific trained screening filters for covered and uncovered patient detection models, and multiple person detection models, which are followed by segmenting the body parts into abdomen and foot using artificial intelligence–based models. The extracted CPD time series along with vitals were used to predict the future (1–6 h) hemodynamic shock using sequence models (LSTM). This model was also compared with Random-Forest and Linear Mixed-Effects models, known for working with longitudinal data but LSTM outperformed the other two. We compared the results of our method with Nagori A. et al.’s study too and observable improvement can be seen in the majority of the metrics (Supplementary Table S2). We also trained and checked the performance of models that can classify the length of videos and images for the future risk of hemodynamic shock. CNNs, which have shown state-of-the-art performance for image classification, were used for the task. However, we found that domain features, such as CPD, performed better than direct image and video classification for the future risk of shock. Even after many experiments performed for a direct classification of images using the concepts of TV-Chambolle (Condat, 2013) denoising, data augmentation, and undersampling/oversampling, we were able to get the best AUROC of only 0.60 for shock detection using ResNet-50. This can be because of the cluttered background with diapers and tubes, and an increased region of interest for the information extraction from a limited variety of images. This also tells the importance of the domain-specific features over end-to-end CNN-based approaches. We built the models for multiple time points till 6 h which showed the best performance at a lead time of 5 h. The metric AUPRC and F1 scores are most significant for an imbalanced dataset such as ours as it does not get biased by the presence of true negatives and thus gives a clearer perspective of a classifier’s utility. The results till 6 h show a promising window with a good prediction rate which can prove to be helpful for the doctors to help find a buffer time prior to the shock event and therefore start the treatment.
Our study currently applies to cold shock as neonates, and young children are more commonly present with “cold shock”—a state of elevated systemic vascular resistance (SVR) and low cardiac output with cold extremities due to vasoconstriction and delayed capillary refill (Martin and Weiss, 2015). The peripheral vasoconstriction can be identified with the center to peripheral temperature difference using thermal imaging that we leveraged in our study to identify the cold shock. We also acknowledge the limitations and uncertainty caused by using skin temperature and core-peripheral temperature gradient in isolation (Schey et al., 2010); therefore, we introduced the heart rate as a crucial cardiovascular parameter (Table 1) in our CPD-based method for shock prediction. Respiratory rate (RR) is also an important indicator of cardiorespiratory status (Liu et al., 2020) as indicated in Table 1. However, we did not encapsulate RR in our models owing to its limited availability at all-time points in our dataset and in a bid to keep the method parsimonious. Nevertheless, we did study the effects of including RR with the available measurements till 6 h, and it showed promising results as demonstrated in Supplementary Table S5, which can be taken up in our future research. Models that are predictive of future occurrence of events suffer from lower model performance indicators and a trade-off is needed between sensitivity and specificity. While probability thresholds can be easily set to decrease the false positives (at the cost of sensitivity), our model is intended to be used as a screening tool as a shock is a potentially lethal condition. Moreover, the FPR varies between 10 and 15% at different time points and certainly can be improved further with the incorporation of additional features. We will be taking up further performance improvement in our future research to bring it down to less than 5%. The other limitations of our method arise during the quality data procurement. Monitoring thermal patterns could be hard at times when the patient is being covered or being operated on; caregivers can also block the recordings. However, these issues can be resolved by keeping ∼4.3 min of uncovered slots for monitoring. Nevertheless, we have built an AI system that is efficient enough to identify the covered and uncovered images and can remove the caregiver or person other than the patient. Thus, our system is able to extract the thermal patterns efficiently. Using CPD as a co-acting parameter eliminates the use of invasive arterial monitoring and cuff-based blood pressure monitors which can cause infections due to the repeated use on multiple patients. Our models require thermal recordings which do not identify an individual’s identity, hence are safe for patient privacy and do not take much time away from the time assigned for care. Our methods can be extended beyond the intensive care settings, that is, to help in the community-based monitoring by the developing countries’ rural healthcare workers, such as the Accredited Social Health Activist (ASHA), run by the women promoting accessible healthcare in rural Indian settings. This can be made possible by training them to use a specifically designed frugal mobile app for thermal shock detection and prediction, which can work with readily available Android/iOS smartphones, with only model weights preinstalled. The results can be made more robust by expanding the dataset by including more patients in the study. Also, the dataset can be made varied and generalized by including observations from multiple other clinical sites. Thus, the study performed illustrates a great noninvasive and minimal feature architecture that promises to be a life-saver by informing the clinicians about shock well in advance.
Data Availability Statement
The raw data supporting the conclusion of this article will be made available by the authors, without undue reservation.
Ethics Statement
The studies involving human participants were reviewed and approved by the Institute Ethics Committee, AIIMS New Delhi. Written informed consent from the participants’ legal guardian/next of kin was not required to participate in this study in accordance with the national legislation and the institutional requirements.
Author Contributions
Concept and design: TS, RL, AN, and PS. Standard thermal video data acquisition: PS, TS, and RL. Vital data acquisition: AN, PS, TS, and RL. Cohort construction using vital data and thermal videos: PS and AN. Segmentation models and extraction of CPD: VV. Multiple person detection and covered–uncovered detection: VV. LSTM, models, and evaluation: VV, AN, and HB. Mixed-effects and random forest models and evaluation: PS, AN, and VV. Statistical analysis: VV and AN. Direct classification models: RD, HB, and MW. Interpretation of results: VV, AN, PS, and TS. Manuscript and revision: VV, AN, PS, RL, and TS. The final version of the manuscript was approved by all the authors involved in this research.
Funding
This work was supported by the Wellcome Trust/DBT India Alliance Fellowship IA/CPHE/14/1/501504 awarded to TS.
Conflict of Interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Publisher’s Note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors, and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
Acknowledgments
We thank Varun Prakash and Anil Sharma for the technical support provided at the PICU, AIIMS, New Delhi. TS and PS also acknowledge the support received from the Department of Science and Technology vide project DST/INT/ISR/P-21/2017.
Supplementary Material
The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fphys.2022.862411/full#supplementary-material
References
Acker S. N., Bredbeck B., Partrick D. A., Kulungowski A. M., Barnett C. C., Bensard D. D. (2017). Shock Index, Pediatric Age-Adjusted (SIPA) Is More Accurate Than Age-Adjusted Hypotension for Trauma Team Activation. Surgery 161 (3), 803–807. doi:10.1016/j.surg.2016.08.050
Bagavathiappan S., Saravanan T., Philip J., Jayakumar T., Raj B., Karunanithi R., et al. (2009). Infrared Thermal Imaging for Detection of Peripheral Vascular Disorders. J. Med. Phys. 34 (1), 43. doi:10.4103/0971-6203.48720
Bagcchi S. (2015). India Has Low Doctor to Patient Ratio, Study Finds. BMJ 351, h5195. doi:10.1136/bmj.h5195
Balakrishnan G., Durand F., Guttag J. (2013). “Detecting Pulse from Head Motions in Video,” in IEEE Conference on Computer Vision and Pattern Recognition, 3430–3437. doi:10.1109/CVPR.2013.440
Barbosa Pereira C., Czaplik M., Blazek V., Leonhardt S., Teichmann D. (2018). Monitoring of Cardiorespiratory Signals Using Thermal Imaging: A Pilot Study on Healthy Human Subjects. Sensors 18 (5), 1541. doi:10.3390/s18051541
Bates D. M., Mächler M., Bolker B. M., Walker S. C. (2015). Fitting Linear Mixed-Effects Models Using {lme4}. J. Statist. Software 67 (1), 1–48. doi:10.18637/jss.v067.i01
Bourcier S., Pichereau C., Boelle P.-Y., Nemlaghi S., Dubée V., Lejour G., et al. (2016). Toe-to-room Temperature Gradient Correlates with Tissue Perfusion and Predicts Outcome in Selected Critically Ill Patients with Severe Infections. Ann. Intensive Care 6 (1), 63. doi:10.1186/s13613-016-0164-2
Chawla N. V., Bowyer K. W., Hall L. O., Kegelmeyer W. P. (2002). Smote: Synthetic Minority Over-sampling Technique. J. Artif. Intell. Res. 16, 321–357. doi:10.1613/jair.953
Chen W., McDuff D. (2018). DeepPhys: Video-Based Physiological Measurement Using Convolutional Attention Networks. Munich, Germany: ECCV.
Cho Y., Julier S. J., Bianchi-Berthouze N. (2019). Instant Stress: Detection of Perceived Mental Stress through Smartphone Photoplethysmography and Thermal Imaging. JMIR Ment. Health 6 (4), e10140. doi:10.2196/10140
Christ M., Kempa-Liehr A. W., Feindt M. (2016). Distributed and Parallel Time Series Feature Extraction for Industrial Big Data Applications. ArXiv. e-prints: 1610.07717. doi:10.48550/arXiv.1610.07717
Condat L. (2013). A Direct Algorithm for 1-D Total Variation Denoising. IEEE Signal Process. Lett. 20, 1054–1057. doi:10.1109/LSP.2013.2278339
Cutler A., Cutler D. R., Stevens J. R. (2012). Random Forests. New York: Springer, 157–175. doi:10.1007/978-1-4419-9326-7_5
Divatia J. V., Amin P. R., Ramakrishnan N., Kapadia F. N., Todi S., Sahu S., et al. (2016). Intensive Care in India: The Indian Intensive Care Case Mix and Practice Patterns Study. Indian J. Crit. Care Med. 20, 216–225. doi:10.4103/0972-5229.180042
Early Goal-Directed Therapy Collaborative Group of Zhejiang Province (2010). The Effect of Early Goal-Directed Therapy on Treatment of Critical Patients with Severe Sepsis/septic Shock: a Multi-Center, Prospective, Randomized, Controlled Study. Zhongguo wei zhong bing ji jiu yi xue 22, 331.
Falotico J. M., Shinozaki K., Saeki K., Becker L. B. (2020). Advances in the Approaches Using Peripheral Perfusion for Monitoring Hemodynamic Status. Front. Med. 7, 614326. doi:10.3389/fmed.2020.614326
FLIR Systems (2016). FLIR Systems, 70 Castilian Dr., Goleta, California 93117. Available at: http://www.flir.com (Accessed December 23, 2021).
Froesel M., Goudard Q., Hauser M., Gacoin M., Ben Hamed S. (2020). Automated Video-Based Heart Rate Tracking for the Anesthetized and Behaving Monkey. Sci. Rep. 10, 17940. doi:10.1038/s41598-020-74954-5
Guazzi A. R., Villarroel M., Jorge J., Daly J., Frise M. C., Robbins P. A., et al. (2015). Non-contact Measurement of Oxygen Saturation with an RGB Camera. Biomed. Opt. Express 6 (9), 3320–3338. doi:10.1364/BOE.6.003320
Hasanin A., Mukhtar A., Nassar H. (2017). Perfusion Indices Revisited. J. Intensive Care 5, 24. doi:10.1186/s40560-017-0220-5
He K., Zhang X., Ren S., Sun J. (2016). “Deep Residual Learning for Image Recognition,” in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, United States. 770–778. doi:10.1109/cvpr.2016.90
Herget-Rosenthal S., Saner F., Chawla L. S. (2008). Approach to Hemodynamic Shock and Vasopressors. Clin. J. Am. Soc. Nephrol. 3 (2), 546–553. doi:10.2215/CJN.01820407
Hochreiter S., Schmidhuber J. (1997). Long Short-Term Memory. Neural Comput. 9 (8), 1735–1780. doi:10.1162/neco.1997.9.8.1735
Houwink A. P. I., Rijkenberg S., Bosman R. J., van der Voort P. H. J. (2016). The Association between Lactate, Mean Arterial Pressure, Central Venous Oxygen Saturation and Peripheral Temperature and Mortality in Severe Sepsis: a Retrospective Cohort Analysis. Crit. Care 20, 56. doi:10.1186/s13054-016-1243-3
Jorge J., Villarroel M., Tomlinson H., Gibson O., Darbyshire J. L., Ede J., et al. (2022). Non-contact Physiological Monitoring of Post-operative Patients in the Intensive Care Unit. npj Digit. Med. 5, 4. doi:10.1038/s41746-021-00543-z
Lehman L.-w. H., Saeed M., Talmor D., Mark R., Malhotra A. (2013). Methods of Blood Pressure Measurement in the ICU*. Crit. care Med. 41 (1), 34–40. doi:10.1097/CCM.0b013e318265ea46
Lima A., Bakker J. (2005). Noninvasive Monitoring of Peripheral Perfusion. Intensive Care Med. 31 (10), 1316–1326. doi:10.1007/s00134-005-2790-2
Lima A., Jansen T. C., van Bommel J., Ince C., Bakker J. (2009). The Prognostic Value of the Subjective Assessment of Peripheral Perfusion in Critically Ill Patients. Crit. Care Med. 37 (3), 934–938. doi:10.1097/CCM.0b013e31819869db
Liu H., Chen F., Hartmann V., Khalid S. G., Hughes S., Zheng D. (2020). Comparison of Different Modulations of Photoplethysmography in Extracting Respiratory Rate: from a Physiological Perspective. Physiol. Meas. 41, 094001. doi:10.1088/1361-6579/abaaf0
Liu H., Allen J., Zheng D., Chen F. (2019). Recent Development of Respiratory Rate Measurement Technologies. Physiol. Meas. 40, 07TR01. doi:10.1088/1361-6579/ab299e
Liu L., Shah S., Zhao G., Yang X. (2018). Respiration Symptoms Monitoring in Body Area Networks. Appl. Sci. 8 (4), 568. doi:10.3390/app8040568
Majumder S., Chen L., Marinov O., Chen C.-H., Mondal T., Deen M. J. (2018). Noncontact Wearable Wireless ECG Systems for Long-Term Monitoring. IEEE Rev. Biomed. Eng. 11, 306–321. doi:10.1109/RBME.2018.2840336
Mambou S., Maresova P., Krejcar O., Selamat A., Kuca K. (2018). Breast Cancer Detection Using Infrared Thermal Imaging and a Deep Learning Model. Sensors 18 (9), 2799. doi:10.3390/s18092799
Martin K., Weiss S. L. (2015). Initial Resuscitation and Management of Pediatric Septic Shock. Minerva Pediatr. 67 (2), 141–158.
Murugavel M. (2019). Yolo-Annotation-Tool-New. Available at: https://github.com/ManivannanMurugavel/Yolo-Annotation-Tool-New- (Accessed November 5, 2020).
Nachman D., Constantini K., Poris G., Wagnert-Avraham L., Gertz S. D., Littman R., et al. (2020). Wireless, Non-invasive, Wearable Device for Continuous Remote Monitoring of Hemodynamic Parameters in a Swine Model of Controlled Hemorrhagic Shock. Sci. Rep. 10, 17684. doi:10.1038/s41598-020-74686-6
Nagori A., Dhingra L. S., Bhatnagar A., Lodha R., Sethi T. (2019). Predicting Hemodynamic Shock from Thermal Images Using Machine Learning. Sci. Rep. 9 (1), 91. doi:10.1038/s41598-018-36586-8
Ng E. Y.-K. (2009). A Review of Thermography as Promising Non-invasive Detection Modality for Breast Tumor. Int. J. Therm. Sci. 48 (5), 849–859. doi:10.1016/j.ijthermalsci.2008.06.015
Oranges T., Dini V., Romanelli M. (2015). Skin Physiology of the Neonate and Infant: Clinical Implications. Adv. Wound Care 4 (10), 587–595. doi:10.1089/wound.2015.0642
Ortiz-Dosal A., Kolosovas-Machuca E. S., Rivera-Vega R., Simón J., González F. J. (2014). Use of Infrared Thermography in Children with Shock: A Case Series. SAGE Open Med. Case Rep. 2. 2050313X1456177. doi:10.1177/2050313X14561779
Paszke A., Gross S., Massa F., Lerer A., Bradbury J., Chanan G., et al. (2019). “Pytorch: An Imperative Style, High-Performance Deep Learning Library,” in Advances in Neural Information Processing Systems (Vancouver, Canada: Curran Associates, Inc.), 32, 8024–8035.
Pauk J., Wasilewska A., Ihnatouski M. (2019). Infrared Thermography Sensor for Disease Activity Detection in Rheumatoid Arthritis Patients. Sensors 19 (16), 3444. doi:10.3390/s19163444
Pavlidis I., Dowdall J., Sun N., Puri C., Fei J., Garbey M. (2007). Interacting with Human Physiology. Comput. Vis. Image Underst. 108 (1–2), 150–170. doi:10.1016/j.cviu.2006.11.018
Pittman J. A. L., Ping J. S., Mark J. B. (2004). Arterial and Central Venous Pressure Monitoring. Int. Anesthesiol. Clin. 42 (1), 13–30. doi:10.1016/j.atc.2006.08.00810.1097/00004311-200404210-00004
Python Software Foundation (2018). Python Programming Language-Official Website. Available at: http://www.python.org/ (Accessed July 14, 2020).
Quintana M. S., Fernández-Jaén T., Fernández-Cuevas I., Gómez-Carmona P. M., Arnaiz-Lastras J., Pérez M.-D., et al. (2015). Infrared Thermography as a Support Tool for Screening and Early Diagnosis in Emergencies. J. Med. Imaging Hlth Inf. 5 (6), 1223–1228. doi:10.1166/jmihi.2015.1511
Rivers E., Nguyen B., Havstad S., Ressler J., Muzzin A., Knoblich B., et al. (2001). Early Goal-Directed Therapy in the Treatment of Severe Sepsis and Septic Shock. N. Engl. J. Med. 345, 1368–1377. doi:10.1056/NEJMoa010307
Ronneberger O., Fischer P., Brox T. (2015). U-net: Convolutional Networks for Biomedical Image Segmentation. Munich, Germany: CoRR.
Rotenstein L. S., Torre M., Ramos M. A., Rosales R. C., Guille C., Sen S., et al. (2018). Prevalence of Burnout Among Physicians: A Systematic Review. JAMA 320, 111131–111150. doi:10.1001/jama.2018.12777
Ruopp M. D., Perkins N. J., Whitcomb B. W., Schisterman E. F. (2008). Youden Index and Optimal Cut-point Estimated from Observations Affected by a Lower Limit of Detection. Biom. J. 50, 419–430. doi:10.1002/bimj.200710415
Sagaidachnyi A. A., Fomin A. V., Usanov D. A., Skripal A. V. (2017). Thermography-based Blood Flow Imaging in Human Skin of the Hands and Feet: a Spectral Filtering Approach. Physiol. Meas. 38 (2), 272–288. doi:10.1088/1361-6579/aa4eaf
Schey B. M., Williams D. Y., Bucknall T. (2010). Skin Temperature and Core-Peripheral Temperature Gradient as Markers of Hemodynamic Status in Critically Ill Patients: a Review. Heart & Lung 39 (1), 27–40. doi:10.1016/j.hrtlng.2009.04.002
Seek Thermal (2022). Seek Thermal Compact, Seek Thermal Inc. Available at: https://www.thermal.com/compact-series.html (Accessed January 22, 2022).
Selvarani A., Suresh G. R. (2019). Infrared Thermal Imaging for Diabetes Detection and Measurement. J. Med. Syst. 43 (2), 23. doi:10.1007/s10916-018-1140-1
Sethi T., Nagori A., Bhatnagar A., Gupta P., Fletcher R., Lodha R. (2017). “Validating the Tele-Diagnostic Potential of Affordable Thermography in a Big-Data Data-Enabled ICU,” in Proceedings of the Special Collection on eGovernment Innovations in India New Delhi, India: Association for Computing Machinery, 64–69. doi:10.1145/3055219.3055234
Shah S. A., Yang X., Abbasi Q. H. (2019). Cognitive Health Care System and its Application in Pill‐rolling Assessment. Int. J. Numer. Model 32, 2632. doi:10.1002/jnm.2632
Tan F., Chen S., Lyu W., Liu Z., Yu C., Lu C., et al. (2019). Non-invasive Human Vital Signs Monitoring Based on Twin-Core Optical Fiber Sensors. Biomed. Opt. Express 10 (11), 5940–5951. doi:10.1364/BOE.10.005940
Thiruvengadam J., Mariamichael A. (2018). A Preliminary Study for the Assessment of Hypertension Using Static and Dynamic IR Thermograms. Biomed. Tech. 63 (2), 197–206. doi:10.1515/bmt-2016-0237
van Genderen M. E., van Bommel J., Lima A. (2012). Monitoring Peripheral Perfusion in Critically Ill Patients at the Bedside. Curr. Opin. Crit. Care 18 (3), 273–279. doi:10.1097/MCC.0b013e3283533924
Keywords: hemodynamic shock, deep learning, ICU—intensive care unit, artificial intelligence, thermal imaging, computer vision
Citation: Vats V, Nagori A, Singh P, Dutt R, Bandhey H, Wason M, Lodha R and Sethi T (2022) Early Prediction of Hemodynamic Shock in Pediatric Intensive Care Units With Deep Learning on Thermal Videos. Front. Physiol. 13:862411. doi: 10.3389/fphys.2022.862411
Received: 25 January 2022; Accepted: 07 June 2022;
Published: 11 July 2022.
Edited by:
Yasha Hasija, Delhi Technological University, IndiaReviewed by:
Haipeng Liu, Coventry University, United KingdomJingwei Ji, Google (United States), United States
Copyright © 2022 Vats, Nagori, Singh, Dutt, Bandhey, Wason, Lodha and Sethi. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Tavpritesh Sethi, dGF2cHJpdGVzaHNldGhpQGlpaXRkLmFjLmlu
†These authors have contributed equally to this work and share first authorship
‡These authors have contributed equally to this work