Deep learning-based prediction for significant coronary artery stenosis on coronary computed tomography angiography in asymptomatic populations

Background: Although coronary computed tomography angiography (CCTA) is currently utilized as the frontline test to accurately diagnose coronary artery disease (CAD) in clinical practice, there are still debates regarding its use as a screening tool for the asymptomatic population. Using deep learning (DL), we sought to develop a prediction model for significant coronary artery stenosis on CCTA and identify the individuals who would benefit from undergoing CCTA among apparently healthy asymptomatic adults.

Methods: We retrospectively reviewed 11,180 individuals who underwent CCTA as part of routine health check-ups between 2012 and 2019. The main outcome was the presence of coronary artery stenosis of ≥70% on CCTA. We developed a prediction model using machine learning (ML), including DL. Its performance was compared with pretest probabilities, including the pooled cohort equation (PCE), CAD consortium, and updated Diamond-Forrester (UDF) scores.

Results: In the cohort of 11,180 apparently healthy asymptomatic individuals (mean age 56.1 years; men 69.8%), 516 (4.6%) presented with significant coronary artery stenosis on CCTA. Among the ML methods employed, a neural network with multi-task learning (19 selected features), one of the DL methods, was selected due to its superior performance, with an area under the curve (AUC) of 0.782 and a high diagnostic accuracy of 71.6%. Our DL-based model demonstrated a better prediction than the PCE (AUC, 0.719), CAD consortium score (AUC, 0.696), and UDF score (AUC, 0.705). Age, sex, HbA1c, and HDL cholesterol were highly ranked features. Personal education and monthly income levels were also included as important features of the model.

Conclusion: We successfully developed the neural network with multi-task learning for the detection of CCTA-derived stenosis of ≥70% in asymptomatic populations. Our findings suggest that this model may provide more precise indications for the use of CCTA as a screening tool to identify individuals at a higher risk, even in asymptomatic populations, in clinical practice.

1. Introduction

Coronary heart disease (CHD) is a leading cause of morbidity and mortality worldwide, contributing to one-third of global deaths (1, 2). Since the treatment of CHD causes a considerable medical and socioeconomic burden, recent clinical interests have been focused on risk stratification, early detection, and prevention of the disease (3, 4). Although traditional risk classification tools, such as Pooled Cohort Equation (PCE) or Systemic Coronary Risk Estimation, have been a priority to estimate the risk of CHD (3, 4), these are imprecise and less practical, and may result in unnecessary long-term therapies or loss of opportunity for timely management (5). Recently, coronary computed tomography angiography (CCTA) has emerged as the frontline test to noninvasively evaluate CHD, providing excellent diagnostic accuracy and prognostic implication (6, 7). The SCOT-HEART investigators demonstrated that CCTA-guided preventive therapy could bring a significant reduction in cardiovascular death or nonfatal myocardial infarction in patients with stable chest pain (7). Indeed, the advent of CCTA can bring a paradigm shift in this field (8). On the other hand, there is still debate in the screening of CHD using CCTA among apparently healthy asymptomatic populations, because of unclear superiority to traditional approach and lack of cost-effectiveness (4, 9). Data from large-scale cohorts consistently reported that the prevalence of occult coronary atherosclerosis was not negligible in the asymptomatic populations, with approximately 5% of them having significant stenosis (10, 11). In addition, as the prognostic value of CCTA was well validated in this population (12, 13), some experts state that CCTA has the potential as a good screening tool for asymptomatic individuals (9, 14). Considering the expansion of indications for CCTA in clinical practice, it is required to identify which patient would be beneficial from CCTA.

The development of deep learning (DL) has recently achieved professional-level performance in various clinical data analyses, such as medical imaging and electronic health records, and has attracted much attention in the field of medical diagnosis (15–17). As DL models consist of a large number of parameters compared to statistical methods, this complexity enables the ability to express complex correlations between variables and provide meaningful insights for better medical decision-making. In particular, DL methods have been increasingly applied in medical imaging interpretation with reliable results (18, 19), and also in prediction model development (17). Hence, we sought to develop a DL-based risk prediction model for coronary artery stenosis on CCTA in apparently healthy asymptomatic adults and identify the beneficiaries for introducing CCTA as a screening tool.

2. Materials and methods

2.1. Study population and dataset composition

A total of 11,753 medical records were retrospectively collected from individuals who underwent CCTA for the purpose of health check-ups at the Healthcare System Gangnam Centre, Seoul National University Hospital, between January 2012 and December 2019. The individuals chose to undergo the examinations for health status evaluation of their own will. If an individual had symptoms, he/she was recommended to visit the corresponding outpatient clinic rather than a health check-up. Clinical and laboratory information of the study participants were retrieved from their examination results, which were performed on the same day as the CCTA. Individuals with prior coronary revascularization (n = 150) and those without appropriate clinical information (n = 423) were excluded from the analysis. Finally, we established an entire dataset of 11,180 cases. We split the dataset into two subsets to develop and validate the model. Specifically, we used 9,578 records (85.7%) collected between 2012 and 2018 as a training set and the remaining 1,602 data (14.3%) collected in 2019 as a test set for model evaluation (Figure 1). The study protocol conformed to the ethical guidelines of the Declaration of Helsinki and was approved by the Institutional Review Board of Seoul National University Hospital (IRB No. H-2004-117-1118). Owing to the retrospective nature of the study, the board waived the requirement for written informed consent.

FIGURE 1

Figure 1. Schematic flowchart of the study population. AUC, area under the curve; ROC, receiver operating characteristic; SHAP, SHapley Additive ExPlanations.

2.2. CCTA image acquisition and analysis

CCTA image acquisition, post-processing, and interpretation were performed according to the guidelines of the Society of Cardiovascular Computed Tomography (20). A 256-detector row scanner (Brilliance iCT 256; Philips Medical Systems Inc., Cleveland, OH, USA) was used to acquire the images with proper quality using either a retrospectively electrocardiography (ECG)-gated or prospectively ECG-triggered protocol, as appropriate. Two level III-equivalent experienced radiologists, who were blinded to the clinical data, assessed, and interpreted all CCTA images. The coronary artery calcium score was measured quantitatively by the sum of the area of coronary calcification using the Agatston scoring system (in units) (21). A coronary atherosclerotic plaque was evaluated in all coronary artery segments with a diameter of ≥2 mm and defined as any distinguishable lesion of >1 mm² within or adjacent to the coronary arterial lumen in at least two independent image planes. The presence, location, and severity of coronary atherosclerotic plaques were evaluated at per-segment and per-patient levels using the modified 15-segment criteria (22).

The presence of obstructive CHD was the main outcome of this study, which was defined as the detection of significant coronary artery stenosis having ≥70% maximal diameter stenosis in any of the four major coronary arteries on CCTA. The severity of maximal coronary stenosis was quantified by visual estimation, with an agreement between two independent radiologists.

2.3. Coronary artery stenosis prediction model development

The dataset consists of 11,180 individuals’ medical records including 73 variables. The model development comprised of three steps: (1) data pre-processing, (2) model training and evaluation, and (3) feature importance analysis. The overall process is illustrated in Figure 1, and the key components of this study are presented in Figure 2.

FIGURE 2

Figure 2. Schematic diagram for the prediction model development. This is a schematic diagram of a newly developed DL-based prediction model for significant coronary artery stenosis on CCTA in 11,180 asymptomatic populations who underwent a routine health check-up. Using various parameters that physicians can readily access in clinical practice, the DL-based model could provide more precise indications of CCTA for the purpose of screening. CCTA, coronary computed tomography angiography; CHD, coronary heart disease; DL, deep learning.

2.3.1. Data pre-processing

The raw format of each record is composed of categorical and numerical variables of different scales with some missing values. The rates of missing values are varied according to the variable types. Every missing value was imputed by the average value of its variable type. To handle with heterogeneous variables, min-max normalization was applied to several categorical variables, and Gaussian normalization was applied to the rest of numerical variables. For several types of numerical variables including age, systolic blood pressure (BP), diastolic BP were discretized using known criteria before normalization The stenosis values (%) were discretized into four categories as follows: diameter stenosis of <30%, 30%–50%, 50%–70%, and ≥70%.

2.3.2. Model training and evaluation

It is well-known that the effect of each variable on stenosis varies. In addition, using all variables in model training may decrease generalization ability, and is not computationally efficient. For these reasons, a stepwise forward/backward feature selection method was used to select more significant variables for stenosis prediction modelling. The final 19 input variables were determined as the sum of 15 variables collected from the feature selection method (Supplementary Table S1) and additional 4 variables known to be clinically important [body mass index (BMI), smoking, hypertension, and diabetes].The feature selection was performed on 12 different randomly split train/validation sets using different random seeds. The feature selection criterion was based on the area under the curve (AUC) calculated from the receiver operating characteristic (ROC) curve of the validation set.

We utilized the DL model to predict the CCTA class based on input features. Our neural network model comprised of two parts. The first part included two locally connected layers among the predefined input variable groups based on prior clinical knowledge, and the second part consisted of two fully connected layers with 512 dimensions. The model’s performance was evaluated based on its ability to classify significant coronary artery stenosis with a threshold of 70%, which was clinically defined as obstructive CHD (23). The models were trained in 2 different settings. The first was a multi-class classification task, where the model directly predicted the class among four categories of diameter stenosis. The second setting involved rearranging the single multiclass classification task into three binary classifications based on the criteria of 30%, 50%, and 70%, which is multi-task learning. The initial learning rate was set to 0.05 and implemented step-wise decay. The activation function used in the model is sigmoid linear unit. We did not use any regularization method in the model.

Currently, to the best of our knowledge, there was no reliable risk prediction model of coronary artery stenosis in apparently healthy asymptomatic populations. Therefore, the model performance was compared with three established clinical scoring systems: coronary artery disease (CAD) consortium scores (24) and the updated Diamond-Forrester (UDF) method (25) as estimates for the pretest probability of obstructive CAD in patients with chest pain and PCE as a well-known tool to estimate the 10-year risk of the clinically relevant endpoint of atherosclerotic cardiovascular disease (26).

Additionally, we implemented conventional machine learning models, including linear regression, random forest, and eXtreme Gradient Boosting (XGBoost), as comparison targets to verify the prediction performance of our neural network model.

2.3.3. Feature importance analysis

Interpreting the reasons behind the model’s decision is crucial, especially for clinical purposes. To enhance the interpretability of the model predictions, we adopted the SHapley Additive ExPlanations (SHAP) (27) analysis after the model training. It enables the deep neural network to be interpretable, providing each variable’s contribution to the model decision. Based on the obtained SHAP values, the relative significance of each variable to the prediction target was identified.

2.4. Clinical and laboratory evaluation

Clinical and laboratory data were collected, as published previously (28). Anthropometric measurements were taken by a trained nurse on the day of the health examination. BMI was calculated as weight divided by height in meters (kg/m²), and waist circumference (WC) was measured at the midpoint between the lower costal margin and iliac crest. BP and heart rate were taken as average values after 2 measurements using an automated BP monitor with at least 5-min interval in a seated position. To collect information on smoking, alcohol intake, education, monthly income, and personal and family medical histories, a self-reported questionnaire was used. Laboratory data included white blood cell (WBC) count, hemoglobin, serum total cholesterol, high-density lipoprotein (HDL) cholesterol, fasting glucose, glycated hemoglobin (HbA1c), serum albumin, estimated glomerular filtration rate (eGFR), and urine albumin to creatinine ratio levels. An automatic analyzer at the Department of Laboratory Medicine at Seoul National University Hospital (Toshiba 200 FR autoanalyzer; Toshiba, Tokyo, Japan) was used to analyze all laboratory tests.

2.5. Statistical analysis

Continuous variables are described as mean ± standard deviation or median (interquartile range) and categorical variables as numbers (%). To compare the differences in baseline characteristics between the study groups, Student’s t-test was performed for continuous variables, and Pearson’s chi-square test was applied for categorical variables as required. ROC curves were plotted to identify the predictive power of ML-based models, including our newly developed neural network model, and the comparative scoring systems. The AUC value from each curve was calculated and compared using Bootstrap method with 200 subsampling (29). All statistical analyses were performed using the python, numpy, pandas, and seaborn package of version 3.9, 1.22.3, 1.3.3, and 0.11.2, respectively. A value of two-sided p < 0.05 was considered statistically significant.

3. Results

3.1. Baseline characteristics of the study population

A total of 11,180 cases (mean age, 56.1 years; men 69.8%) were enrolled in this study. The mean BMI and WC were 24.4 kg/m² and 87.5 cm, respectively; approximately 37.6% of the study population was considered obese. On the examination day, systolic and diastolic BP were averaged as 119.7 and 78.5 mmHg, respectively. Conventional cardiovascular risk factors, including hypertension, diabetes, dyslipidemia, and current smoking, were found in 24.8%, 8.5%, 17.3%, and 19.1% of the total subjects, respectively. Approximately one-fifth (21.6%) of the study participants had a family history of premature cardiovascular disease, and 1,341 (12.0%) and 1,933 (17.3%) of them were on anti-platelet agents and statins treatment, respectively. In terms of socioeconomic status, university graduates or higher accounted for 74.0%, and half of the total subjects earned US$8,000 or more per month. The mean values of fasting glucose and HbA1c were 104.9 mg/dl and 5.8%, respectively. Lipid profiles were as follows: total cholesterol 192.5 mg/dl, HDL cholesterol 52.9 mg/dl, LDL cholesterol 115.5 mg/dl, and triglycerides 106.0 mg/dl (median). Subjects in the training set were likely to be older; current smokers; have more comorbidities including hypertension, diabetes, and dyslipidemia; and more educated with higher income than those in the test set. The detailed baseline characteristics of the analyzed cases are presented in Table 1.

TABLE 1

Table 1. Baseline characteristics of study population.

3.2. Feature selection for DL-based prediction model

Figure 3 illustrates the improvement in prediction performance resulting from forward feature selection using all variables collected from the validation set. Supplementary Table S1 presents the cumulative ROC-AUC with adding variables to model, from top to bottom. For the prediction of significant coronary artery stenosis, age, sex, socioeconomic status including education and monthly income level, dyslipidemia, and several laboratory variables including eGFR, hemoglobin, HbA1c, and HDL cholesterol were the highly ranked features. As expected, age and sex were the top 2 predictive features of the DL-based model. Among the conventional cardiovascular risk factors, HbA1c, HDL cholesterol, non-HDL cholesterol, systolic BP, and a history of dyslipidemia were sequentially important in estimating the possibility of obstructive CHD on CCTA. In addition, WBC count and albumin level significantly contributed to the performance of the DL-based model. Interestingly, we observed that personal education and monthly income levels were selected for the DL-based model to predict obstructive CHD. As mentioned above, 4 variables known to be cardiovascular risk factors were added on 15 selected features from forward feature selection method. Therefore, a total of 19 selected variables were used in the neural network model development.

FIGURE 3

Figure 3. Increase of predictive value upon forward feature selection. Features in Y-axis are arranged by forward selection in order. The bar graph, generated from the validation dataset, shows how performance gains when variables are added to the model, on a logarithmic scale. Error bar shows its 95% confidence interval. Scatter dot plot shows an increase in ROC-AUC of the test dataset when using top-k features, indicating that performance saturates shortly after adding a few variables to the model. CVD, cardiovascular disease; ECG, electrocardiography; eGFR, estimated glomerular filtration rate; HbA1, glycated hemoglobin; HDL, high-density lipoprotein; OH, hydroxy; SBP, systolic blood pressure; WBC, white blood cells; other abbreviations as Figure 1.

3.3. Comparison of prediction models for coronary artery stenosis

Among all the participants, obstructive CHD was observed in 516 (4.6%; 4.2% in training set and 6.8% in test set). Individual prediction scores were calculated in the test set, and the results were expressed as a function of the presence of obstructive CHD, presenting a bimodal score distribution with a higher prevalence of obstructive CHD in subjects with higher scores (Figure 4). When comparing the predictive power of machine learning models, the neural network with multi-task learning produced the better performance to predict significant coronary artery stenosis of ≥70% [AUC 0.782, 95% confidence interval (CI) 0.749–0.820] than Random Forest (AUC 0.695, 95% CI 0.656–0.730), XGBoost (AUC 0.732, 95% CI 0.680–0.788), and logistic regression (AUC 0.749, 95% CI 0.708–0.795) (all p < 0.001) (Figure 5A, Supplementary Table S2). This result supported that the neural network-based model for predicting obstructive CHD outperformed the conventional DL methodologies. The sensitivity, specificity, positive predictive value, negative predictive value, and balanced accuracy of the neural network with multi-task learning were 0.757, 0.675, 0.143, 0.975, and 71.6%, respectively.

FIGURE 4

Figure 4. Outcome distribution of individuals with obstructive and non-obstructive CHD. The X-axis represents the probability of coronary artery stenosis, and the Y-axis represents the number of individuals in the test set. The blue and orange bars indicate individuals predicted to have coronary artery stenosis of <30% and ≥70%, respectively.

FIGURE 5

Figure 5. Comparison of predictive performances among a neural network-based model, other DL-based models, and clinical risk scoring models with selected features. Both plots show the ROC-AUC of various models with selected features. (A) Our neural network-based prediction model demonstrates significantly better performance than previously known clinical scoring systems, with ROC-AUC of 0.782, even in asymptomatic populations. (B) Among the machine learning-based models, a neural network with multi-task shows the best performance to predict significant coronary artery stenosis on CCTA. CAD, coronary artery disease; DL, deep learning; XGBoost, eXtreme gradient boosting; other abbreviations as Figures 1, 2.

Compared with PCE (AUC 0.719, 95% CI 0.699–0.743), CAD consortium score (AUC 0.696, 95% CI 0.677–0.717), and UDF scores (AUC 0.705, 95% CI 0.684–0.726), our neural network model had a significantly higher area under the ROC curves for the prediction of obstructive CHD than the clinical scoring systems (all p < 0.001) (Figure 5B, Supplementary Table S2).

Furthermore, we trained ML-based models without the feature selection process. The neural network model outperformed the others in this setting as well (Supplementary Figure S1). Notably, the feature selection not only preserved the model’s performance but also improved its generalization to outperform the models trained on all features, as demonstrated by the higher ROC-AUC value.

A further comparative analysis of model predictability was performed, and the correlation between the predicted and actual stenosis ratios was calculated. Each calibration plot of the proposed neural network, CAD consortium, and UDF was shown in Supplementary Figure S2, respectively. The proposed neural network provided robust and accurate predictability in all probability regions, whereas the others showed less correlation, suggesting a better performance in the asymptomatic individuals.

3.4. Explainability of the neural network model

Figure 6 depicts the SHAP value of each input feature in predicting the probability of obstructive CHD as obtained from the SHAP analysis. In order of SHAP value, sex, age, and HDL cholesterol levels were the most significant factors in predicting obstructive CHD in asymptomatic individuals.

FIGURE 6

Figure 6. SHAP value of a neural network-based model. The bar graph shows how each feature contributes to the model decision in test set. As the contribution of each input feature is described as an absolute value, the sign of the effect of each variable is not represented. Abbreviations as Figures 1, 3.

4. Discussion

In the present study including 11,180 apparently healthy asymptomatic adults who underwent CCTA for routine health check-ups, the main findings were as follows: (1) obstructive CHD, defined as significant coronary artery stenosis of ≥70% on CCTA, was found in 4.6% of this asymptomatic population; (2) the DL-based risk prediction model for significant coronary artery stenosis was successfully developed using a multi-task learning with feature selection, which demonstrated superior performance compared to the clinical pretest probabilities and other ML-based models; (3) among the variables that were readily assessed in routine clinical practice, age, sex, education and monthly income level, dyslipidemia, and laboratory variables including eGFR, hemoglobin, HbA1c, and HDL cholesterol were the highly ranked features to predict significant coronary artery stenosis on CCTA.

Although clinicians have bent their best efforts to appropriately manage CHD and improve the prognosis for decades, the global burden of CHD, in terms of disabilities and deaths, continued to increase (1, 30). Early detection and prevention are the most effective ways to reduce the impact of CHD, given the limited healthcare resources (31). A conventional approach in clinical practice is to start validated clinical risk scoring systems, such as PCE or SCORE, and direct downstream testing (3, 4). However, since these probabilistic risk scores were developed in older populations and mainly validated in symptomatic patients for the purpose of predicting major cardiovascular events, including myocardial infarction, stroke, or cardiovascular death, the risk in younger, healthy, and asymptomatic populations is likely to be underestimated and inaccurate (32, 33). The advent of CCTA has provided a paradigm shift to assess cardiovascular risk, particularly in low-risk asymptomatic populations. Many previous studies using CCTA have reported a higher prevalence of silent coronary atherosclerosis in the asymptomatic populations than expected, and further, individuals with obstructive CHD were not uncommon (10, 11). In the SCAPIS cohort, approximately 1 of 3 men and 1 of 4 women were found to have coronary atherosclerosis in the subgroup classified as low risk by PCE or SCORE from the general population (11). In addition, Choi et al. demonstrated that 5% and 2% of asymptomatic adults with a mean age of 50 years had CCTA-derived coronary artery stenosis of ≥50% and ≥75%, respectively (10). However, currently, no reliable prediction model is proven to detect coronary artery stenosis in apparently healthy asymptomatic populations. Hence, there is a need for a clinical method to easily screen obstructive CHD candidates from low-risk asymptomatic groups that would be missed under the traditional approach.

In this study, given the insufficient clinical methodology for predicting risk in the asymptomatic population, we utilized the DL-based method and successfully developed the risk prediction neural network model to detect significant coronary artery stenosis of ≥70% on CCTA in the self-referred apparently healthy asymptomatic population. Our model provided balanced accuracy of 71.6% and achieved an AUC value of 0.782 for the test set, demonstrating good performance and reproducibility. Obviously, ML or DL-based risk prediction models with excellent performance in various study populations have been presented before (17, 34–38). However, previous studies have mainly used ML or DL to predict the prognosis in patients with symptomatic or established cardiovascular disease, such as those with suspected or known CAD, those with acute coronary syndrome, and those who underwent coronary angiography or coronary artery bypass grafting (CABG). In the CONFIRM registry, an ML-based model including clinical parameters and CACS estimated the risk of obstructive CAD on CCTA in suspected CAD patients (17). More recently, the risk of post-CABG mortality was successfully estimated with acceptable performance using various ML models, in 16,850 patients who underwent isolated CABG (36). Although a few studies were performed in the asymptomatic healthy populations, they used only CACS, a simpler tool for estimating total atherosclerotic burden (39, 40). The current study successfully developed the DL-based model in predicting those who are likely to have significant coronary artery stenosis on CCTA, using clinical parameters that are obtained from routine health check-ups among the asymptomatic, apparently healthy individuals. Indeed, this allows the potential to detect CAD early and improve prognosis, considering that CCTA is emerging as a frontline test for the diagnosis of CAD.

Our DL-based risk prediction model included conventional risk factors such as age, sex, systolic BP, HbA1c, and lipid-related variables as well as unfamiliar risk factors such as serum albumin level, WBC count, and socioeconomic status. These are variables that clinicians rarely pay attention to when evaluating whether an individual has obstructive CHD. However, considering that WBC count (41) and serum albumin level (42) were once noted for a close link with CHD, our results were able to bring about a re-perception of solid but overlooked risk factors using the DL method. This is quite consistent with the previous study (43) in that our model was developed using the clinical features that can be readily found in electronic health records in clinical practice. In addition, recent studies have demonstrated that the integration of socioeconomic status into traditional risk factors can allow better risk stratification and prognosis for individuals at risk (44). More noticeably, our DL-based model provided better predictive power than the CAD consortium score, UDF, and other well-known probabilistic risk scores, indicating that the group-specific risk prediction model should be newly established in the low-risk asymptomatic population.

Among the various ML methods available, we chose a DL-based approach to develop our risk prediction model. DL-based methods offer several advantages over conventional approaches in terms of flexibility and generality (45). Traditional statistical methods such as simple linear regression assume that dependent variables should be normally distributed and independent of each other with homoscedasticity (46). Furthermore, the errors obtained from regression analysis should be uncorrelated and constant (47). These constraints raise the practical issues when analyzing heterogeneous real-world data. However, DL-based methods are relatively free of these constraints regarding the distribution of variables. Owing to their applicability, DL-based methods have shown prominent outcomes in various real-world clinical studies in recent times.

In order to improve the interpretability of the proposed risk prediction model, we introduced two additional methods. First, we implemented a forward/backward feature selection during the model training, which resulted in the exclusion of several variables that had relatively less contribution to prediction of significant coronary artery stenosis, thus ensuring the generalization of the model. Second, we conducted SHAP analysis on the prediction results of the test set to analyze the relative significance of each variable. Our findings revealed that sex and age were the most significant factors affecting the degree of coronary artery stenosis on CCTA, which is in accordance with previous studies (12, 24, 25, 27). As previously mentioned, the dataset utilized in this study was collected from the general population and contained both categorical and numerical features. Our neural network outperformed other conventional methods, verifying the superiority and robustness of DL-based methods for analyzing complex data distributions in the real world. In addition, SHAP analysis allows clinicians to identify the specific features of patient data that the model considers to be crucial in the prediction of stenosis. Finally, our model exhibited superior prediction performance on both the test and training sets compared to previous models. Based on these observations, our neural network could alleviate two common limitations of DL-based methods: the lack of explainability (black-box) and loss of generality (overfitting) (48).

This study has some limitations noteworthy to mention. First, this was a single-center single-ethnicity retrospective observational cohort study comprising apparently healthy self-referred asymptomatic individuals. Thus, it may cause selection and referral bias that limits the generalizability of the model. However, we demonstrated that the prevalence of CCTA-derived obstructive CHD was not trivial even in the lower-risk asymptomatic group by applying the DL-based model composed of familiar clinical variables. It is possible to help identify who could benefit from the use of CCTA as a screening tool and prevent the overuse of diagnostic imaging modalities. Obviously, our DL-based risk prediction model is an aspect of tailored medicine that is expected to improve an individual’s prognosis. Further multi-ethnic prospective studies are required to validate our results. Second, the study endpoint was the presence of obstructive CHD, defined as ≥70% maximal diameter stenosis in coronary arteries on CCTA, which was assessed by visual estimation. Accordingly, the percentage of stenosis may be overestimated, especially in cases with severe coronary artery calcification or motion artifacts. To minimize this limitation, all CCTA images were independently analyzed by two level-III experienced radiologists blinded to the clinical information, with full agreement. Additional validation using volumetric measurements is required in future studies. Third, this study used the full routine health check-up dataset, including many clinical, laboratory, and imaging variables. External validation was not performed, because it was difficult to find other large-scale independent cohorts involving a full dataset, which might have led to overfitting. Further studies in other populations should be considered to validate the current model and extend the indications. Lastly, as shown in Table 1, the mean values of several variables were significantly different between the train and test datasets. This situation could be problematic because it may lead to poor generalizability of the model. However, despite the different characteristics, our model overcame the gap and achieved robust performances on both sets.

5. Conclusions

In conclusion, in a cohort comprising 11,180 apparently healthy asymptomatic adults, we successfully developed a DL-based risk prediction model for detecting CCTA-derived obstructive CHD with acceptable accuracy. Our novel model showed better predictive performance than previous well-known risk scoring systems, suggesting more precise indications for CCTA as a screening tool for further risk stratification in asymptomatic populations. Therefore, the utilization of this DL-based model may help clinicians make medical decisions in terms of early diagnosis and primary prevention, and promote the cardiovascular health of individuals.

Data availability statement

The datasets presented in this article are not readily available. Requests to access the datasets should be directed to the corresponding author [MJK], upon reasonable request.

Ethics statement

The studies involving human participants were reviewed and approved by the Institutional Review Board of Seoul National University Hospital (IRB No. H-2004-117-1118). Written informed consent for participation was not required for this study in accordance with the national legislation and the institutional requirements.

Author contributions

Conception and design: HL, BK, JJ, and MK; data acquisition: HL, HP, SC, and MK; data analysis and interpretation: HL, BK, JJ, and MK; statistical analysis: BK and JJ; drafting and finalizing the paper: HL, BK, JJ, and MK; critical revision of the paper for important intellectual content: HP, SY, and SC. All authors contributed to the article and approved the submitted version.

Funding

This study was supported by the SNUH Healthcare System Gangnam Center Research Fund (Grant No. 2020-02), Institute of Information & communications Technology Planning & Evaluation grant funded by the Korea government (Grant No. 2021-0-01343), and Basic Science Research Program through the National Research Foundation of Korea funded by the Ministry of Education (Grant No. 2022R1A6A3A01087603).

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fcvm.2023.1167468/full#supplementary-material

References

1. Virani SS, Alonso A, Aparicio HJ, Benjamin EJ, Bittencourt MS, Callaway CW, et al. Heart disease and stroke statistics-2021 update: a report from the American heart association. Circulation. (2021) 143(8):e254–743. doi: 10.1161/CIR.0000000000000950

PubMed Abstract | CrossRef Full Text | Google Scholar

2. Mortality GBD, Causes of Death C. Global, regional, and national life expectancy, all-cause mortality, and cause-specific mortality for 249 causes of death, 1980–2015: a systematic analysis for the global burden of disease study 2015. Lancet. (2016) 388(10053):1459–544. doi: 10.1016/S0140-6736(16)31012-1

PubMed Abstract | CrossRef Full Text | Google Scholar

3. Arnett DK, Blumenthal RS, Albert MA, Buroker AB, Goldberger ZD, Hahn EJ, et al. 2019 ACC/AHA guideline on the primary prevention of cardiovascular disease: a report of the American College of Cardiology/American Heart Association task force on clinical practice guidelines. Circulation. (2019) 140(11):e596–646. doi: 10.1161/CIR.0000000000000678

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Visseren FLJ, Mach F, Smulders YM, Carballo D, Koskinas KC, Back M, et al. 2021 ESC guidelines on cardiovascular disease prevention in clinical practice. Eur Heart J. (2021) 42(34):3227–337. doi: 10.1093/eurheartj/ehab484

PubMed Abstract | CrossRef Full Text | Google Scholar

5. Karmali KN, Persell SD, Perel P, Lloyd-Jones DM, Berendsen MA, Huffman MD. Risk scoring for the primary prevention of cardiovascular disease. Cochrane Database Syst Rev. (2017) 3:CD006887. doi: 10.1002/14651858.CD006887

PubMed Abstract | CrossRef Full Text | Google Scholar

6. Miller JM, Rochitte CE, Dewey M, Arbab-Zadeh A, Niinuma H, Gottlieb I, et al. Diagnostic performance of coronary angiography by 64-row CT. N Engl J Med. (2008) 359(22):2324–36. doi: 10.1056/NEJMoa0806576

PubMed Abstract | CrossRef Full Text | Google Scholar

7. Investigators S-H, Newby DE, Adamson PD, Berry C, Boon NA, Dweck MR, et al. Coronary CT angiography and 5-year risk of myocardial infarction. N Engl J Med. (2018) 379(10):924–33. doi: 10.1056/NEJMoa1805971

PubMed Abstract | CrossRef Full Text | Google Scholar

8. Abdelrahman KM, Chen MY, Dey AK, Virmani R, Finn AV, Khamis RY, et al. Coronary computed tomography angiography from clinical uses to emerging technologies: JACC state-of-the-art review. J Am Coll Cardiol. (2020) 76:1226–43. doi: 10.1016/j.jacc.2020.06.076

PubMed Abstract | CrossRef Full Text | Google Scholar

9. Korosoglou G, Chatzizisis YS, Raggi P. Coronary computed tomography angiography in asymptomatic patients: still a taboo or precision medicine? Atherosclerosis. (2021) 317:47–9. doi: 10.1016/j.atherosclerosis.2020.12.001

PubMed Abstract | CrossRef Full Text | Google Scholar

10. Choi EK, Choi SI, Rivera JJ, Nasir K, Chang SA, Chun EJ, et al. Coronary computed tomography angiography as a screening tool for the detection of occult coronary artery disease in asymptomatic individuals. J Am Coll Cardiol. (2008) 52(5):357–65. doi: 10.1016/j.jacc.2008.02.086

PubMed Abstract | CrossRef Full Text | Google Scholar

11. Bergstrom G, Persson M, Adiels M, Bjornson E, Bonander C, Ahlstrom H, et al. Prevalence of subclinical coronary artery atherosclerosis in the general population. Circulation. (2021) 144(12):916–29. doi: 10.1161/CIRCULATIONAHA.121.055340

PubMed Abstract | CrossRef Full Text | Google Scholar

12. Beller E, Meinel FG, Schoeppe F, Kunz WG, Thierfelder KM, Hausleiter J, et al. Predictive value of coronary computed tomography angiography in asymptomatic individuals with diabetes mellitus: systematic review and meta-analysis. J Cardiovasc Comput Tomogr. (2018) 12(4):320–8. doi: 10.1016/j.jcct.2018.04.002

PubMed Abstract | CrossRef Full Text | Google Scholar

13. Perez de Isla L, Alonso R, Gomez de Diego JJ, Muniz-Grijalvo O, Diaz-Diaz JL, Zambon D, et al. Coronary plaque burden, plaque characterization and their prognostic implications in familial hypercholesterolemia: a computed tomographic angiography study. Atherosclerosis. (2021) 317:52–8. doi: 10.1016/j.atherosclerosis.2020.11.012

PubMed Abstract | CrossRef Full Text | Google Scholar

14. Lee KK, Wereski R, Williams MC, Mills NL. Population screening with coronary computed tomography angiography and the prevention of coronary events. Circulation. (2021) 144(12):930–3. doi: 10.1161/CIRCULATIONAHA.121.055784

PubMed Abstract | CrossRef Full Text | Google Scholar

15. Pina A, Helgadottir S, Mancina RM, Pavanello C, Pirazzi C, Montalcini T, et al. Virtual genetic diagnosis for familial hypercholesterolemia powered by machine learning. Eur J Prev Cardiol. (2020) 27(15):1639–46. doi: 10.1177/2047487319898951

PubMed Abstract | CrossRef Full Text | Google Scholar

16. Char DS, Shah NH, Magnus D. Implementing machine learning in health care—addressing ethical challenges. N Engl J Med. (2018) 378(11):981–3. doi: 10.1056/NEJMp1714229

PubMed Abstract | CrossRef Full Text | Google Scholar

17. Al’Aref SJ, Maliakal G, Singh G, van Rosendael AR, Ma X, Xu Z, et al. Machine learning of clinical variables and coronary artery calcium scoring for the prediction of obstructive coronary artery disease on coronary computed tomography angiography: analysis from the CONFIRM registry. Eur Heart J. (2020) 41(3):359–67. doi: 10.1093/eurheartj/ehz565

CrossRef Full Text | Google Scholar

18. Chilamkurthy S, Ghosh R, Tanamala S, Biviji M, Campeau NG, Venugopal VK, et al. Deep learning algorithms for detection of critical findings in head CT scans: a retrospective study. Lancet. (2018) 392(10162):2388–96. doi: 10.1016/S0140-6736(18)31645-3

PubMed Abstract | CrossRef Full Text | Google Scholar

19. Gulshan V, Peng L, Coram M, Stumpe MC, Wu D, Narayanaswamy A, et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal Fundus photographs. JAMA. (2016) 316(22):2402–10. doi: 10.1001/jama.2016.17216

PubMed Abstract | CrossRef Full Text | Google Scholar

20. Abbara S, Blanke P, Maroules CD, Cheezum M, Choi AD, Han BK, et al. SCCT Guidelines for the performance and acquisition of coronary computed tomographic angiography: a report of the society of cardiovascular computed tomography guidelines committee: endorsed by the North American Society for Cardiovascular Imaging (NASCI). J Cardiovasc Comput Tomogr. (2016) 10(6):435–49. doi: 10.1016/j.jcct.2016.10.002

PubMed Abstract | CrossRef Full Text | Google Scholar

21. Agatston AS, Janowitz WR, Hildner FJ, Zusmer NR, Viamonte M Jr., Detrano R. Quantification of coronary artery calcium using ultrafast computed tomography. J Am Coll Cardiol. (1990) 15(4):827–32. doi: 10.1016/0735-1097(90)90282-T

PubMed Abstract | CrossRef Full Text | Google Scholar

22. Achenbach S, Moselewski F, Ropers D, Ferencik M, Hoffmann U, MacNeill B, et al. Detection of calcified and noncalcified coronary atherosclerotic plaque by contrast-enhanced, submillimeter multidetector spiral computed tomography: a segment-based comparison with intravascular ultrasound. Circulation. (2004) 109(1):14–7. doi: 10.1161/01.CIR.0000111517.69230.0F

PubMed Abstract | CrossRef Full Text | Google Scholar

23. Reina-Gutierrez T, Serrano-Hernando FJ, Sanchez-Hervas L, Ponce A, de Ceniga MV, Martin A. Recurrent carotid artery stenosis following endarterectomy: natural history and risk factors. Eur J Vasc Endovasc Surg. (2005) 29(4):334–41. doi: 10.1016/j.ejvs.2004.10.007

PubMed Abstract | CrossRef Full Text | Google Scholar

24. Genders TS, Steyerberg EW, Hunink MG, Nieman K, Galema TW, Mollet NR, et al. Prediction model to estimate presence of coronary artery disease: retrospective pooled analysis of existing cohorts. Br Med J. (2012) 344:e3485. doi: 10.1136/bmj.e3485

CrossRef Full Text | Google Scholar

25. Genders TS, Steyerberg EW, Alkadhi H, Leschka S, Desbiolles L, Nieman K, et al. A clinical prediction rule for the diagnosis of coronary artery disease: validation, updating, and extension. Eur Heart J. (2011) 32(11):1316–30. doi: 10.1093/eurheartj/ehr014

PubMed Abstract | CrossRef Full Text | Google Scholar

26. Goff DC Jr., Lloyd-Jones DM, Bennett G, Coady S, D’Agostino RB, Gibbons R, et al. 2013 ACC/AHA guideline on the assessment of cardiovascular risk: a report of the American College of Cardiology/American Heart Association task force on practice guidelines. Circulation. (2014) 129(25 Suppl 2):S49–73. doi: 10.1161/01.cir.0000437741.48606.98

PubMed Abstract | CrossRef Full Text | Google Scholar

27. Lundberg SM, Lee SI. A unified approach to interpreting model predictions. Adv Neural Inf Process Syst. (2017). doi: 10.48550/arXiv.1705.07874

CrossRef Full Text | Google Scholar

28. Lee H, Park HE, Yoon JW, Choi SY. Clinical significance of body fat distribution in coronary artery calcification progression in Korean population. Diabetes Metab J. (2021) 45(2):219–30. doi: 10.4093/dmj.2019.0161

PubMed Abstract | CrossRef Full Text | Google Scholar

29. Robin X, Turck N, Hainard A, Tiberti N, Lisacek F, Sanchez JC, et al. pROC: an open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinformatics. (2011) 12:77. doi: 10.1186/1471-2105-12-77

PubMed Abstract | CrossRef Full Text | Google Scholar

30. Roth GA, Mensah GA, Johnson CO, Addolorato G, Ammirati E, Baddour LM, et al. Global burden of cardiovascular diseases and risk factors, 1990−2019: update from the GBD 2019 study. J Am Coll Cardiol. (2020) 76(25):2982–3021. doi: 10.1016/j.jacc.2020.11.010

PubMed Abstract | CrossRef Full Text | Google Scholar

31. McConnachie A, Walker A, Robertson M, Marchbank L, Peacock J, Packard CJ, et al. Long-term impact on healthcare resource utilization of statin treatment, and its cost effectiveness in the primary prevention of cardiovascular disease: a record linkage study. Eur Heart J. (2014) 35(5):290–8. doi: 10.1093/eurheartj/eht232

PubMed Abstract | CrossRef Full Text | Google Scholar

32. Akosah KO, Schaper A, Cogbill C, Schoenfeld P. Preventing myocardial infarction in the young adult in the first place: how do the national cholesterol education panel III guidelines perform? J Am Coll Cardiol. (2003) 41:1475–9. doi: 10.1016/S0735-1097(03)00187-6

PubMed Abstract | CrossRef Full Text | Google Scholar

33. Nasir K, Michos ED, Blumenthal RS, Raggi P. Detection of high-risk young adults and women by coronary calcium and national cholesterol education program panel III guidelines. J Am Coll Cardiol. (2005) 46(9):1931–6. doi: 10.1016/j.jacc.2005.07.052

PubMed Abstract | CrossRef Full Text | Google Scholar

34. Wang Y, Chen H, Sun T, Li A, Wang S, Zhang J, et al. Risk predicting for acute coronary syndrome based onmachine learning model with kinetic plaque features fromserial coronary computed tomography angiography. Eur Heart J Cardiovasc Imaging. (2022) 23(6):800–10. doi: 10.1093/ehjci/jeab101

PubMed Abstract | CrossRef Full Text | Google Scholar

35. Li D, Xiong G, Zeng H, Zhou Q, Jiang J, Guo X. Machine learning-aided risk stratification system for the prediction of coronary artery disease. Int J Cardiol. (2021) 326:30–4. doi: 10.1016/j.ijcard.2020.09.070

PubMed Abstract | CrossRef Full Text | Google Scholar

36. Khalaji A, Behnoush AH, Jameie M, Sharifi A, Sheikhy A, Fallahzadeh A, et al. Machine learning algorithms for predicting mortality after coronary artery bypass grafting. Front Cardiovasc Med. (2022) 9:977747. doi: 10.3389/fcvm.2022.977747

PubMed Abstract | CrossRef Full Text | Google Scholar

37. Nakanishi R, Slomka PJ, Rios R, Betancur J, Blaha MJ, Nasir K, et al. Machine learning adds to clinical and CAC assessments in predicting 10-year CHD and CVD deaths. JACC Cardiovasc Imaging. (2021) 14(3):615–25. doi: 10.1016/j.jcmg.2020.08.024

PubMed Abstract | CrossRef Full Text | Google Scholar

38. Han D, Kolli KK, Al’Aref SJ, Baskaran L, van Rosendael AR, Gransar H, et al. Machine learning framework to identify individuals at risk of rapid progression of coronary atherosclerosis: from the PARADIGM registry. J Am Heart Assoc. (2020) 9(5):e013958. doi: 10.1161/JAHA.119.013958

PubMed Abstract | CrossRef Full Text | Google Scholar

39. Lee J, Lim JS, Chu Y, Lee CH, Ryu OH, Choi HH, et al. Prediction of coronary artery calcium score using machine learning in a healthy population. J Pers Med. (2020) 10(3):96. doi: 10.3390/jpm10030096

PubMed Abstract | CrossRef Full Text | Google Scholar

40. Han D, Kolli KK, Gransar H, Lee JH, Choi SY, Chun EJ, et al. Machine learning based risk prediction model for asymptomatic individuals who underwent coronary artery calcium score: comparison with traditional risk prediction approaches. J Cardiovasc Comput Tomogr. (2020) 14(2):168–76. doi: 10.1016/j.jcct.2019.09.005

PubMed Abstract | CrossRef Full Text | Google Scholar

41. Madjid M, Awan I, Willerson JT, Casscells SW. Leukocyte count and coronary heart disease: implications for risk assessment. J Am Coll Cardiol. (2004) 44(10):1945–56. doi: 10.1016/j.jacc.2004.07.056

PubMed Abstract | CrossRef Full Text | Google Scholar

42. Arques S. Human serum albumin in cardiovascular diseases. Eur J Intern Med. (2018) 52:8–12. doi: 10.1016/j.ejim.2018.04.014

PubMed Abstract | CrossRef Full Text | Google Scholar

43. Forrest IS, Petrazzini BO, Duffy Á, Park JK, Marquez-Luna C, Jordan DM, et al. Machine learning-based marker for coronary artery disease: derivation and validation in two longitudinal cohorts. Lancet. (2023) 401(10372):215–25. doi: 10.1016/S0140-6736(22)02079-7

PubMed Abstract | CrossRef Full Text | Google Scholar

44. Schultz WM, Kelli HM, Lisko JC, Varghese T, Shen J, Sandesara P, et al. Socioeconomic status and cardiovascular outcomes: challenges and interventions. Circulation. (2018) 137(20):2166–78. doi: 10.1161/CIRCULATIONAHA.117.029652

PubMed Abstract | CrossRef Full Text | Google Scholar

45. Rajula HSR, Verlato G, Manchia M, Antonucci N, Fanos V. Comparison of conventional statistical methods with machine learning in medicine: diagnosis, drug development, and treatment. Medicina (B Aires). (2020) 56(9):455. doi: 10.3390/medicina56090455

CrossRef Full Text | Google Scholar

46. Berry WD. Understanding regression assumptions. vol. 92. Newbury Park, CA: Sage. (1993).

47. Casson RJ, Farmer LDM. Understanding and checking the assumptions of linear regression: a primer for medical researchers. Clin Exp Ophthalmol. (2014) 42(6):590–6. doi: 10.1111/ceo.12358

PubMed Abstract | CrossRef Full Text | Google Scholar

48. Markus AF, Kors JA, Rijnbeek PR. The role of explainability in creating trustworthy artificial intelligence for health care: a comprehensive survey of the terminology, design choices, and evaluation strategies. J Biomed Inform. (2021) 113:103655. doi: 10.1016/j.jbi.2020.103655

PubMed Abstract | CrossRef Full Text | Google Scholar