Sophia Rasheeqa Ismail1,2
Muhamad Khairul Nazrin Khalil1Mohd Shawal Faizal Mohamad3Shamsul Azhar Shah2*- 1Nutrition, Metabolic and Cardiovascular Research Centre, Institute for Medical Research, National Institutes of Health, Shah Alam, Malaysia
- 2Department of Community Health, Faculty of Medicine, National University of Malaysia, Kuala Lumpur, Malaysia
- 3Department of Cardiology, Hospital Canselor Tuanku Muhriz, Kuala Lumpur, Malaysia
Background: The cultural and genetic diversity of the Southeast Asian population has contributed to distinct cardiovascular disease risks, incidence, and prognosis compared to the Western population, thereby raising concerns about the accuracy of predicted risks of existing prognostic models.
Objectives: We aimed to evaluate the predictive performances of validated, recalibrated, and developed prognostic risk prediction tools used in the Southeast Asian population with acute myocardial infarction (AMI) events for secondary events
Methods: We searched MEDLINE and Cochrane Central databases until March 2022. We included prospective and retrospective cohort studies that exclusively evaluated populations in the Southeast Asian region with a confirmed diagnosis of an AMI event and evaluated for risk of secondary events such as mortality, recurrent AMI, and heart failure admission. The CHARMS and PRISMA checklists and PROBAST for risk of bias assessment were used in this review.
Results: We included 7 studies with 11 external validations, 3 recalibrations, and 3 new models from 4 countries. Both short- and long-term outcomes were assessed. Overall, we observed that the external validation studies provided a good predictive accuracy of the models in the respective populations. The pooled estimate of the C-statistic in the Southeast Asian population for GRACE risk score is 0.83 (95%CI 0.72–0.90, n = 6 validations) and for the TIMI risk score is 0.80 (95%CI: 0.772–0.83, n = 5 validations). Recalibrated and new models demonstrated marginal improvements in discriminative values. However, the method of predictive accuracy measurement in most studies was insufficient thereby contributing to the mixed accuracy effect. The evidence synthesis was limited due to the relatively low quality and heterogeneity of the available studies.
Conclusion: Both TIMI and GRACE risk scores demonstrated good predictive accuracies in the population. However, with the limited strength of evidence, these results should be interpreted with caution. Future higher-quality studies spanning various parts of the Asian region will help to understand the prognostic utility of these models better.
Systematic review registration: https://www.crd.york.ac.uk/prospero/display_record.php?%20RecordID=228486.
Introduction
Cardiovascular diseases (CVDs) are a major cause of disability and premature death globally—especially in low- and middle-income countries (1). In Southeast Asia, the burden of CVD has been reported to be increasing throughout the region even in rural and urban settings of low-income countries such as Cambodia and Myanmar (2–5). Patients having a recent acute myocardial infarction (AMI) event, a common presentation of coronary artery disease, are at a higher risk of secondary events such as recurrent AMI, heart failure, and even mortality (6). Despite the global improvements in prevention and healthcare, the prevalence of major adverse cardiovascular events (MACE) has been increasing with several trends in Southeast Asia being worse than in other regions (7–9). Amongst the determinants of MACE, ethnicity is an important contributor to the relationship between risk factors and coronary artery disease severity (10). Pre-eminently, evidence on risk factors of MACE in Asian populations has been conflicting (10–13). This highlights the importance of using locally adapted strategies and locally validated tools for better prevention strategies to improve survival and quality of life.
Usage of CVD risk prediction models in clinical medicine is important for stratifying risks in individuals to allow for a more personalized, and eventually cost-effective treatment. Thus far, the vast majority of the CVD prediction models were derived and validated in Western populations- with only a small number validated in the Southeast Asian population (14). The cultural and genetic diversity of the Southeast Asian population has contributed to distinct CVD risks, incidence, and prognosis compared to the Western population (15–17), thereby raising concerns about under-or overestimation of the predicted outcomes (18–20). Therefore, it is pivotal to evaluate the accuracy of prognostic models for CVD for appropriate secondary events prevention and control strategies. In this study, we aimed to systematically evaluate the predictive performances of prognostic risk prediction tools used in the Southeast Asian population with an AMI event for MACE and to explore the predictive performances of recalibrated and newly developed prognostic risk models for the Southeast Asian population with an AMI event.
Methods
Study design
This systematic review was conducted in accordance with the Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modeling Studies (CHARMS) Checklist (21) and reported in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) checklist (22) (Supplementary File 1). The protocol of this systematic review was registered with PROSPERO (Registration number: CRD42021228486). The registered protocol is available at https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=228486.
Eligibility criteria
We included prospective and retrospective cohort studies that exclusively evaluated populations from any of the following countries in the Southeast Asia regions, such as Malaysia, Singapore, Brunei, Indonesia, Myanmar, Vietnam, Thailand, Indonesia, Cambodia, Laos, and Timor-Leste. Participants with a confirmed diagnosis of an AMI event such as ST-segment myocardial infarction (STEMI), non-ST segment myocardial infarction, or unspecified type of AMI were included. We included both single endpoints and composite endpoints that reported any MACE. This included all-cause mortality, cardiac-related mortality, any type of recurrent AMI, admission for heart failure, and stroke. Outcomes developed both in-hospital and following discharge were included with no restrictions in timelines. Prognostic risk models included were either externally validated models (defined as the assessment of previously developed models in a new setting, new timeline, or study population than that of the derivation cohort), recalibrated models (defined as adjustments to the model equation of a previously developed model to a new setting or study population), or a newly developed model (defined as the development of a new prognostic model from a new setting or study population). We included any type of multivariable prognostic models (e.g., Cox proportional hazards models and logistic regression models). We excluded studies that did not evaluate predictive measures such as calibration and discrimination in their validation study.
Search databases
We searched for potential articles in MEDLINE and Cochrane Central (which included PubMed, EMBASE, CINAHL, Clinicaltrials.gov and ICTRP registry) until 21 March 2022. A manual search for additional relevant studies and review articles using references from retrieved articles was also performed. A detailed description of our search strategy is available in Supplementary File 2. No restriction in language was applied.
Selection of studies, data extraction, and management
Two authors (SRI and MKNK) independently screened the titles and/or abstracts for potentially eligible studies, and then independently evaluated the full text of the shortlisted articles to determine eligibility. We outlined the study selection process in a PRISMA diagram. Two authors (SRI and MKNK) independently extracted all data from each included study using a standardized data collection form. We developed the data collection form for this review by incorporating the items in the CHARMS Checklist (21) and other items from a similar review (23). This included the source of data, participants (eligibility criteria, recruitment method, and description), outcomes (type and definition of outcomes), candidate predictors (number, type, definition, and handling of predictors), sample size (number of outcomes, events per variable), missing data (number and handling of missing data), model development (if applicable), model performance (calibration, discrimination, and classification measures), model evaluation, and results. All data recorded were checked for accuracy by another review author. Disagreements along any of these steps were resolved by discussion (e.g., inclusion and exclusion of unsure articles), with the input of a third author (MSFM or SAS) when necessary.
Assessment of the risk of bias
We used the Prediction model study Risk of Bias Assessment Tool (PROBAST) to assess the risk of bias (ROB) of the included studies (24). The PROBAST includes the following four steps: (1) Specification of the systematic review question(s); (2) Classification of the type of prediction model evaluation; (3) Assessment of risk of bias and applicability; (4) Overall judgment. In the ROB assessment, four domains were evaluated as follows: (1) Participants; (2) Predictors; (3) Outcome; (4) Analysis. Each domain was judged either low, high, or unclear ROB. We used the published guidelines by the PROBAST authors as a guiding tool for our ROB assessment (25). Signaling questions were rated as yes (Y), probably yes (PY), probably no (PN), no (N), or no information (NI). Two review authors (SRI and MKNK) assessed each of the included studies for risk of bias independently (Supplementary File 3). Disagreements along these steps were resolved via discussion (e.g., differences in judgments of a particular domain between SRI and MKNK), with the input of a third author (SAS) when necessary.
Predictive performance of models
We compared each prognostic model using the following three different methods: (1) Discrimination; (2) Calibration, and (3) Reclassification, if applicable. Discrimination is the ability of a prediction model to differentiate between two outcome classes. In this review, we evaluated the discriminative ability of the model between those with and without the MACE. For binary outcomes, discrimination is generally presented as the area under the receiver operating characteristic curve (AUC), or the concordance statistics (C-statistic) (26). C-statistic ranges between 0 and 1 with 0.5 being defined as random concordance and 1 as a perfect concordance (26). Calibration reflects the goodness-of-fit of the model between the observed outcomes and predictors. For binary outcomes, the most used calibration measure is the Hosmer-Lemeshow goodness-of-fit test (26). The Hosmer-Lemeshow test is reported with a p-value, in which a p-value < 0.05 is considered a poor calibration of the model (26). Reclassification assesses the improvement in prediction from using a new predictor in addition to existing predictors. Reclassification measures are generally presented as the Net Reclassification Index (NRI). The NRI is reported as the percentage of increment of each of the defined event categories. We had no restriction on the type of statistical tests for the measurement of the predictive performance of each included model.
Evidence synthesis
We initially performed a descriptive analysis approach of all the included studies and categorized by the type of model assessment as follows: externally validated models, recalibrated models, and newly developed models. Subsequently, descriptive and quantitative assessments of the studies were performed using the model. When the same prognostic model was evaluated in multiple validation studies, we calculated the pooled estimate of the predictive performance of these studies through a random-effects meta-analysis that considered any between-study heterogeneity. We evaluated the pooled predictive performance through the discrimination measure of C-statistic. The sample size for each population was deemed to be adequate if there were at least 10 events per candidate predictor (27, 28). All analyses, including the meta-analysis, were performed using R version 4.0.1 (R Foundation for Statistical Computing, Vienna, Austria) (Supplementary File 4).
Results
Our search yielded 1,037 records and 977 records were excluded after titles and abstracts screening. A total of 54 full-text articles were further excluded based on eligibility criteria. Notably, seven studies were eligible for assessment in this review. The PRISMA flow diagram of the study identification processes is shown in Figure 1.
Figure 1. PRISMA flow diagram.
The included studies assessed prognostic models in populations from Indonesia (n = 1) (29), Malaysia (n = 2) (30, 31), Singapore (n = 2) (32, 33), and Thailand (n = 2) (34, 35). The total number of participants in the included studies ranged from 152 to 15,151 participants with a median of 4,701 participants. Notably, five out of seven studies included participants from multicenter registries and ethnicity subgroup group analysis was performed in one study (33). All studies included only local participants and were retrospective in nature. Index disease was STEMI in six studies while the remaining study included the unspecified type of AMI. Outcomes assessed were mortality (n = 7), composite cardiovascular outcomes (n = 2), and hospitalization for heart failure (n = 1). Duration of follow-up included the duration of index hospitalization (n = 4), 30-day of index event (n = 5), 6-month of index event (n = 2), and 1-year of the index event (n = 1).
Validated risk scores were the GRACE score (29, 33–35) and TIMI score (29–31) only. The GRACE score had six external validations from four studies and three recalibrations from one study. The TIMI risk score was validated for in-hospital mortality in two studies, for 30-day mortality in two studies, and for 1-year mortality in one study. All existing models that were developed outside of the Southeast Asia region were evaluated by different authors than the original model's authors except for one study (33). Recalibration of an existing prognostic model was performed only for the GRACE score in one study (33). Prognostic models developed with the local population were available in three studies (30, 32, 33). Out of the three newly developed models, one utilized a machine-learning approach for model development. The overall characteristics of the included studies are summarized in Table 1.
Table 1. Overall characteristics of the included studies.
GRACE score
The GRACE score was validated in three populations as follows: Indonesia (29), Thailand (34, 35), and Singapore (33). Only one study had multicenter recruitment of participants (33). Notably, the two studies from Thailand included participants from the same center and had similar outcomes; however, the number of events and number of participants were different. In-hospital mortality was evaluated in the Malay Singapore, Chinese Singapore, Indian Singapore, and Indonesian populations while 6-month composite cardiovascular outcomes were evaluated in the Thai population in two studies. Adequate events-to-predictor was only achieved in one study (three validations) (33). Discrimination was measured in all validation studies while calibration was only reported in three validation studies. Overall, the GRACE score performed fairly well in all populations with C-statistic ranging between 0.64 and 0.92. The score performed best in the Indonesian population, however, this study was underpowered. The pooled estimate of the C-statistic in the Southeast Asian population is 0.83 (95%CI 0.72–0.90) (Figure 2A). Calibration was only performed in one study (three validations) and was not significant (33). A summary of predictive performance for the GRACE score external validation studies is reported in Table 2.
Figure 2. Meta-analysis of C-statistic for the (A) GRACE score and for the (B) TIMI risk score.
Table 2. Summary of predictive performances of included prognostic models.
Recalibration of the GRACE score to the Singapore population was performed by retaining the regression coefficients from the original GRACE score and by substituting the original GRACE score constants with the constants unique to their population cohort (33). Ethnic-specific models were reported for the Chinese, Malay, and Indian populations. The predictive performance of the recalibrated models was assessed for the risk of in-hospital mortality. Adequate events-to-predictor was achieved in all three recalibrated models. Calibration assessment was not significant in all three models. Discrimination assessed was the lowest for the Indian subgroup (C-statistic 0.84; 95%CI 0.81–0.88) while C-statistic was approximate in the Malay and Chinese models. Overall, there was no difference in predictive measures between the original GRACE score and the recalibrated GRACE score for the Singaporean population. A summary of predictive performance for recalibrated GRACE scores is reported in Table 2.
TIMI score
The TIMI score for STEMI was validated in three studies and two populations, i.e., in Malaysia and Indonesia. Outcomes assessed were in-hospital mortality (n = 2), 30-day mortality (n = 2), and 1-year mortality (n = 1). For the risk of in-hospital mortality, discrimination of the score was better in the Indonesian population (C-statistic 0.84; 95%CI 0.78–0.88) but adequate events-to-predictor was not achieved in this validation. For the risk of 30-day mortality, the discriminative measures were approximate in the two validation studies and adequate events-to-predictor was achieved in both validation studies. For the risk of 1-year mortality, the TIMI risk score performed well in the validated population. The pooled estimate of the C-statistic in the Southeast Asian population for the TIMI score is 0.80 (95%CI: 0.77–0.83) (Figure 2B). Calibration was only performed in one validation and was insignificant (31). A summary of predictive performance for recalibrated GRACE scores is reported in Table 2.
Newly developed models
In this review, we evaluated the following five developed models: the SVMvarImp-SBE-SVM model had three different models for different outcomes, SMIR, and Singapore Score. Although Aziz and colleagues included several other model options, the SVMvarImp-SBE-SVM model was the best performing model (30). The derivative cohort populations were the STEMI population from the Malaysia National Cardiovascular Disease Database for the SVMvarImp-SBE-SVM model and the STEMI population from Singapore Myocardial Infarction Registry for the SMIR and Singapore score. The number of predictors included varied between 8 and 15 predictors. Only the following two predictors were constant across the five newly developed models: age and Killip class. Other predictors could be grouped into either prescribed medications, blood investigations, cardiac investigations/interventions, or past medical history. All new models achieved adequate events-to-predictors' sample size.
The SVMvarImp-SBE-SVM model was the best performing model under the machine-learning approach for the evaluation of the risk of mortality during hospitalization, at 30-day and 1-year mortality (30). Supervised classification machine-learning algorithms used were Logistic Regression, Support Vector Machine, and Random Forest. The number and list of predictors for the SVMvarImp-SBE-SVM model varied according to the outcome. The model for in-hospital mortality included 15 predictors. The model for 30-day mortality included 13 predictors. The model for 1-year mortality included 12 predictors. The list of predictors included is summarized in Supplementary File 5. Of the three models, the discriminative measure was the highest for 30-day mortality (C-statistic 0.90; 95%CI 0.87–0.94) and the lowest for 1-year mortality (C-statistic 0.84; 95%CI 0.80–0.87). Calibration was not reported in all three models. The NRI was the highest for the in-hospital model (20%) followed by 30-day (19%) and 1-year mortality (14%). A summary of predictive performance for SVMvarImp-SBE-SVM models is reported in Table 2.
The SMIR evaluated cardiac mortality during hospitalization, at 30-day and 1-year as well as 1-year hospitalization for heart failure. The model included 9 predictors and was both internally and externally validated. Internal validation was performed through bootstrapping techniques on the derivation cohort. Although calibration was not performed for SMIR, discrimination of the model performed better in the validation cohort (C-statistic 0.903; 95%CI 0.882–0.923) than in the derivation cohort (C-statistic 0.881; 95%CI 0.867–0.896). Reclassification was not reported for this model. The misclassification rate was however reported to assess the discrimination power of the final model: 14.0% for in-hospital cardiac mortality, 14.7% for 30-day cardiac mortality, 16.2% for 1-year cardiac mortality, and 24.0% for 1-year hospitalization for heart failure. A summary of predictive performance for SMIR is reported in Table 2.
The Singapore Score was ethnicity-specific and had different regression coefficients and weightage for Chinese, Malay, and Indian groups. The model included 8 predictors. No external validation was performed for this new model. The model had good and similar discrimination for all three ethnicities. However, there was good calibration for the Malay and Indian groups (p = 0.514 and p = 0.586, respectively) and poor calibration for the Chinese group (p < 0.002). A summary of predictive performance for the Singapore score is reported in Table 2.
Risk of bias of the included studies
We evaluated the risk of bias in each of the included studies with the PROBAST tool. The overall risk of bias and applicability assessment is presented in Figure 3. Briefly, all seven studies were assessed as relatively higher risk of bias but with low concerns of applicability. The studies were evaluated as high risk in the participant's domain due to the usage of registry data and retrospective in nature. Registry data are often not collected for the sole purpose of development, validation, and updating of prediction models thereby limiting prespecified and consistent methods for valid data recording (25). Sensitivity analysis for the type of source data could not be performed as all the included studies were classified as relatively higher risk of bias in the participants' domain. Notably, six out of seven studies were evaluated as having a relatively higher risk of bias in the analysis domain due to improper handling of missing data and/or an insufficient number of events per predictor. One study had insufficient details for the ROB assessment of predictors and outcomes. An insufficient number of events per predictor were judged as a relatively higher risk of bias due to its resulting large standard error and confidence interval leading to higher inaccuracies in the measurement (25). In contrast, improper handling of missing data (e.g., simply excluding enrolled participants with any missing data from the analysis) may lead to biased predictor–outcome associations and biased model performance. As only one study was judged as low risk of bias in the analysis domain, sensitivity analysis could not be conducted. All other domains were deemed to be of low risk of bias and concern for applicability. A detailed description of the risk of bias assessment is provided in Supplementary File 3.
Figure 3. Summary of the risk of bias assessment.
Discussion
This review sought to evaluate the predictive accuracy of prognostic models for Southeast Asian populations with AMI. The two validated models, namely, GRACE and TIMI scores, demonstrated good predictive accuracies in this population although some validations were underpowered. Recalibration of the GRACE score did not demonstrate a significant increase in predictive prediction. Although there are advantages to having local population-derived prediction models, the new models are presented with a longer list of predictors, limited generalizability, and marginal changes in predictive accuracy. Findings of the review were limited due to the small number and relatively higher risk of bias in the included studies.
Most predictive models are developed from specific cohorts, and therefore external validation in a different population examines the generalizability of the model. Morrow and colleagues developed the TIMI Risk Score for STEMI to predict 30-day mortality amongst STEMI patients (36). The model was derived from the Intravenous nPA for Treatment of Infarcting Myocardium Early II (InTIME II) trial and included 14,114 STEMI patients from more than 800 hospitals worldwide. The eight predictors of the TIMI Risk Score for STEMI are age, systolic blood pressure <100 mmHg, heart rate >100 bpm, Killip class II-IV, anterior ST elevation or left-bundle branch block, history of diabetes, hypertension or angina, weight <67 kg, and time to treatment >4 h. As this TIMI score is exclusively used for STEMI individuals, other variations of the TIMI risk score such as TIMI risk score for Unstable Angina/NSTEMI and Dynamic TIMI risk score exist (37, 38). Generally, external validation of these scores demonstrated good predictive accuracy in various populations (39–41).
The GRACE risk score is a scoring system to risk-stratify patients with acute coronary syndrome (ACS) to estimate their 6-month to 3-year mortality (42). Following are the eight predictors included in the original GRACE risk score: older age, history of myocardial infarction, history of heart failure, the increased pulse rate at presentation, lower systolic blood pressure at presentation, elevated initial serum creatinine level, high initial serum cardiac biomarker levels, and ST-segment elevation. In general, validation studies of the GRACE score also provided good predictive accuracies, even amongst indigenous populations, with some exceptions (43–45). Despite having good discriminative power and thus identifying higher-risk patients, there has been evidence of both overestimation and underestimation of outcomes risks of the GRACE risk score in different populations (46–50). Compared to the TIMI score, evidence shows that the GRACE risk score was more accurate in predicting long-term mortality (51). A meta-analysis also concurred that the GRACE risk score performed better than the TIMI risk score for both short- and long-term outcomes (52).
In a validation study in Japan, both original GRACE and simplified GRACE 2.0 risk scores were demonstrated to perform equally well (53), whereas some locally specific risk scores report better predictive accuracies than validation studies of the TIMI risk score and the GRACE risk scores (54, 55). This is expected as the model parameters of these risk scores are tailored to that of the development population and overfitting of the model may occur due to repeated model tuning. Characteristics of these development cohorts differ in terms of their geographic location, patient characteristics, treatment, and availability of resources to existing risk scores. However, a high predictive accuracy (high discrimination and good calibration measures) does not translate to clinical utility. Essentially, prognostic prediction scores that are simple and easy to use with accessible clinical variables promote clinical usefulness.
Patients with established CVD are at a higher risk of cardiovascular event recurrence or death, hence making secondary prevention and accurate prognosis prediction increasingly important to avert subsequent CVD events and to improve life expectancy (56). The usage of CVD risk prediction models has not only provided a more reliable outcome prediction to clinicians but also complemented decision-making to identify those at higher risk and to enhance informed decision-making with the patients while ensuring the cost-effectiveness of care (57). Usage of prognostic risk models for patients presenting with an AMI event, or any coronary artery disease event, risk stratification may be based on the timing of assessment, pre-treatment (such as TIMI score and GRACE score), post-treatment (such as CADILLAC risk score and Dynamic TIMI score), and follow-up (such as DAPT score and GUSTO score) (58).
With the abundance of existing and new prediction risk scores/models, adequate discrimination, calibration, and generalizability determine its usefulness. Overall, we observed that the external validation studies provided a rather good predictive accuracy of the models in the respective populations. However, the method of predictive accuracy measurement was insufficient in most studies thereby providing a mixed accuracy effect. The low strength of evidence in this review was influenced by the inadequate methods of predictive measurement of models, relatively higher risk of bias in almost all of the included studies, small sample size, and a small quantity of included studies. A relatively higher risk of bias in validation studies has also been reported in other reviews (59–61). Although discrimination was the most widely reported measurement, the overall accuracy of the predicted risk could not be established without its corresponding calibration measure. The C-statistic value, for example, is difficult to translate directly into clinical practice (62, 63). A very well discriminating model (C-statistic >0.8) may still be clinically irrelevant if the decision threshold is outside the range of predictions provided by the model. Additionally, the value of the C-statistic is limited in risk categorization and in balancing misclassification errors (62, 64). However, it is also important to note that calibration measures too should not be evaluated independently. Interpretation of calibration measures is influenced by arbitrary groupings of patients, poor power in small data sets, and the reporting of only p-value (65). Although reporting confidence intervals with the p-value is beneficial, this has not been widely practiced. Models demonstrating poor calibration (p < 0.05) generally result from statistical overfitting, measurement errors, and heterogeneity in populations (66). Heterogeneity may exist from the varying disease incidence or prevalence, patient management, and treatment policies (66). Therefore, the interpretation of predictive measures must be performed with caution. Applicability of the model to contemporary and local clinical practice as well as the additional benefit of using the model in current practice should be considered when assessing the clinical usefulness of a risk prediction model (67). Although the new models were derived from the respective local population, their applicability is also limited by having a long list of predictors. The longer list not only increases complexity and time for usage, but several of the predictors are only available in PCI-capable centers. The changes in the discriminative power in these models were also marginal and uncertainties in the improvements of classification as compared to the existing models.
Meta-analysis has the advantage of summarizing quantitative information from related studies (68). However, the heterogeneity of included studies and the type of analysis should be considered during interpretation. In this review, differences were observed in study population characteristics such as age and diagnosis, interventions received, geographical locations, and other eligibility criteria. By adopting a random effect model in the quantitative synthesis, the statistical model assumes that the underlying true parameters vary across the study populations (69). This model is beneficial as it reflects real-life differences in characteristics and treatment as well as sampling variability.
With the growing evidence of clinical prediction models, there are still limited studies performed in regions other than Europe and North America (57, 70). A review of CVD prognostic models in Latin America and the Caribbean provided evidence from only 8 studies and had similar methodological concerns (71). In another review on the diagnostic accuracy of the HEART Score to predict MACE, the Asian population was only evaluated in 4 out of 25 included studies (72). Even then, the low-risk HEART Score Asia-Pacific group reported the highest occurrence of MACE compared to other geographical locations (72). Despite the inclusion of studies from different regions, the absence of a geographical subgroup analysis prohibits a more specific evidence synthesis (73).
This study has the advantage of being the first review to systematically evaluate prognostic models for the Southeast Asian population with AMI. We performed a comprehensive search with a careful selection of studies and extensive data extraction on the key characteristics of prognostic prediction models that included information on the predictors, outcomes, and the studied population. Using the CHARMS checklist for the critical appraisal of the included studies and the PROBAST guidelines for the ROB assessment, we also have the advantage of a systematic and comprehensive review of the included studies. To ensure that the predictive measurement results are not guided by other populations, we included studies that exclusively validate or develop prediction models in the Southeast Asian population. Unfortunately, this review is limited to only patients with AMI as index disease and the inclusion of a limited number of studies and relatively higher risk of bias in included studies.
Conclusion
Despite the wide range of prognostic models, there are still insufficient efforts to externally validate these models in the Southeast Asian population. Our evidence demonstrated a relatively good discrimination ability of both TIMI and GRACE scores, but with the limited strength of evidence, these results should be interpreted with caution. The available evidence is limited due to a small number of countries and due to the significant methodological concerns of the included studies. Future higher-quality studies spanning various parts of the Asian region will help to understand the prognostic utility of these models better.
Data availability statement
The original contributions presented in the study are included in the article/supplementary material, further inquiries can be directed to the corresponding author.
Author contributions
SI conceived the review, screened, and selected studies, extracted data, assessed the risk of bias, performed the statistical analysis, and wrote the review. MK screened and selected studies, extracted data, and assessed the risk of bias. MM and SA wrote the review. All authors contributed to the article and approved the submitted version.
Acknowledgments
We would like to thank the Director-General of Health, Malaysia, for his permission to publish this article.
Conflict of interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Publisher's note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
Supplementary material
The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fcvm.2022.921044/full#supplementary-material
References
1. Smith SC Jr, Collins A, Ferrari R, Holmes DR Jr., Logstrup S, McGhie DV, et al. Our time: a call to save preventable death from cardiovascular disease (Heart Disease and Stroke). J Am Coll Cardiol. (2012) 60:2343–8. doi: 10.1016/j.jacc.2012.08.962
2. Ansah JP, Islam AM, Koh V, Ly V, Kol H, Matchar DB, et al. Systems modelling as an approach for understanding and building consensus on non-communicable diseases (Ncd) management in Cambodia. BMC Health Serv Res. (2019) 19:2. doi: 10.1186/s12913-018-3830-2
3. Purnamasari D. The emergence of non-communicable disease in Indonesia. Acta Med Indones. (2018) 50:273–4.
4. Zaw KK, Nwe N, Hlaing SS. Prevalence of cardiovascular morbidities in Myanmar. BMC Res Notes. (2017) 10:99. doi: 10.1186/s13104-017-2422-2
5. Maharani A. Sujarwoto, Praveen D, Oceandy D, Tampubolon G, Patel A. Cardiovascular disease risk factor prevalence and estimated 10-year cardiovascular risk scores in Indonesia: the smarthealth extend Study. PLoS ONE. (2019) 14:e0215219. doi: 10.1371/journal.pone.0215219
6. Peters SAE, Colantonio LD Dai Y, Zhao H, Bittner V, Farkouh ME, et al. Trends in recurrent coronary heart disease after myocardial infarction among Us women and men between 2008 and 2017. Circulation. (2021) 143:650–60. doi: 10.1161/CIRCULATIONAHA.120.047065
7. Roth GA, Mensah GA, Johnson CO, Addolorato G, Ammirati E, Baddour LM, et al. Global Burden of Cardiovascular Diseases and Risk Factors, 1990-2019: Update from the Gbd 2019 Study. J Am Coll Cardiol. (2020) 76:2982–3021. doi: 10.1016/j.jacc.2020.11.010
8. Rosselló X, Huo Y, Pocock S, Van de Werf F, Chin CT, Danchin N, et al. Global geographical variations in ST-segment elevation myocardial infarction management and post-discharge mortality. Int J Cardiol. (2017) 245:27–34. doi: 10.1016/j.ijcard.2017.07.039
9. Ang CS, Chan KM. A review of coronary artery disease research in Malaysia. Med J Malaysia. (2016) 71:42–57.
10. Gijsberts CM, Seneviratna A, de Carvalho LP, den Ruijter HM, Vidanapthirana P, Sorokin V, et al. Ethnicity modifies associations between cardiovascular risk factors and disease severity in parallel Dutch and Singapore coronary cohorts. PLoS ONE. (2015) 10:e0132278. doi: 10.1371/journal.pone.0132278
11. Meadows TA, Bhatt DL, Cannon CP, Gersh BJ, Röther J, Goto S, et al. Ethnic differences in cardiovascular risks and mortality in atherothrombotic disease: insights from the reduction of atherothrombosis for continued health (Reach) registry. Mayo Clin Proc. (2011) 86:960–7. doi: 10.4065/mcp.2011.0010
12. Vyas MV, Chaturvedi N, Hughes AD, Marmot M, Tillin T. Cardiovascular disease recurrence and long-term mortality in a tri-ethnic british cohort. Heart. (2021) 107:996–1002. doi: 10.1136/heartjnl-2020-317641
13. Franey EG, Kritz-Silverstein D, Richard EL, Alcaraz JE, Nievergelt CM, Shaffer RA, et al. Association of race and major adverse cardiac events (mace): the atherosclerosis risk in communities (aric) cohort. J Aging Res. (2020) 2020:7417242. doi: 10.1155/2020/7417242
14. Park G-M, Han S, Kim SH, Jo M-W, Her SH, Lee JB, et al. Model for assessing cardiovascular risk in a Korean population. Circ Cardiovasc Qual Outcomes. (2014) 7:944–51. doi: 10.1161/CIRCOUTCOMES.114.001305
15. Lam CSP. Heart failure in Southeast Asia: facts and numbers. ESC Heart Fail. (2015) 2:46–9. doi: 10.1002/ehf2.12036
16. Dans A, Ng N, Varghese C, Tai ES, Firestone R, Bonita R. The rise of chronic non-communicable diseases in Southeast Asia: time for action. Lancet. (2011) 377:680–9. doi: 10.1016/S0140-6736(10)61506-1
17. Li J, Dong Y, Wu T, Tong N. Differences between western and Asian type 2 diabetes patients in the incidence of vascular complications and mortality: a systematic review of randomized controlled trials on lowering blood glucose. J Diabetes. (2016) 8:824–33. doi: 10.1111/1753-0407.12361
18. Farzadfar F. Cardiovascular disease risk prediction models: challenges and perspectives. Lancet Global Health. (2019) 7:e1288–e9. doi: 10.1016/S2214-109X(19)30365-1
19. Yusuf S, Hawken S, Ounpuu S, Dans T, Avezum A, Lanas F, et al. Effect of potentially modifiable risk factors associated with myocardial infarction in 52 countries (the interheart study): case-control study. Lancet. (2004) 364:937–52. doi: 10.1016/S0140-6736(04)17018-9
20. Brister SJ, Hamdulay Z, Verma S, Maganti M, Buchanan MR. Ethnic diversity: South Asian ethnicity is associated with increased coronary artery bypass grafting mortality. J Thorac Cardiovasc Surg. (2007) 133:150–4. doi: 10.1016/j.jtcvs.2006.05.068
21. Moons KGM, de Groot JAH, Bouwmeester W, Vergouwe Y, Mallett S, Altman DG, et al. Critical appraisal and data extraction for systematic reviews of prediction modelling studies: the charms checklist. PLoS Med. (2014) 11:e1001744. doi: 10.1371/journal.pmed.1001744
22. Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the Prisma statement. BMJ. (2009) 339:b2535. doi: 10.1136/bmj.b2535
23. Damen JAAG, Hooft L, Schuit E, Debray TPA, Collins GS, Tzoulaki I, et al. Prediction models for cardiovascular disease risk in the general population: systematic review. BMJ. (2016) 353:i2416. doi: 10.1136/bmj.i2416
24. Probast: A Tool to assess the risk of bias and applicability of prediction model studies. Ann Intern Med. (2019) 170:51–8. doi: 10.7326/M18-1376
25. Moons KGM, Wolff RF, Riley RD, Whiting PF, Westwood M, Collins GS, et al. Probast: a tool to assess risk of bias and applicability of prediction model studies: explanation and elaboration. Ann Intern Med. (2019) 170:W1–w33. doi: 10.7326/M18-1377
26. Steyerberg. Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating New York: Springer (2009).
27. Background G. and General strategy. J Clin Epidemiol. (1995) 48:1495–501. doi: 10.1016/0895-4356(95)00510-2
28. Develop V and Compare clinical prediction models involving radiological parameters: study design and statistical methods. Korean J Radiol. (2016) 17:339–50. doi: 10.3348/kjr.2016.17.3.339
29. Martha JW, Sihite TA, Listina D. The difference in accuracy between global registry of acute coronary events score and thrombolysis in myocardial infarction score in predicting in-hospital mortality of acute st-elevation myocardial infarction patients. Cardiol Res. (2021) 12:177–85. doi: 10.14740/cr1247
30. Aziz F, Malek S, Ibrahim KS, Raja Shariff RE, Wan Ahmad WA, Ali RM, et al. Short and longterm mortality prediction after an acute St-elevation myocardial infarction (Stemi) in Asians: a machine learning approach. PLoS ONE. (2021) 16:e0254894. doi: 10.1371/journal.pone.0254894
31. Selvarajah S, Fong AY, Selvaraj G, Haniff J, Uiterwaal CS, Bots ML. An Asian validation of the timi risk score for St-segment elevation myocardial infarction. PLoS ONE. (2012) 7:e40249. doi: 10.1371/journal.pone.0040249
32. Bulluck H, Zheng H, Chan MY, Foin N, Foo DC, Lee CW, et al. Independent predictors of cardiac mortality and hospitalization for heart failure in a multi-ethnic Asian St-segment elevation myocardial infarction population treated by primary percutaneous coronary intervention. Sci Rep. (2019) 9:10072. doi: 10.1038/s41598-019-46486-0
33. Chan MY, Shah BR, Gao F, Sim LL, Chua T, Tan HC, et al. Recalibration of the global registry of acute coronary events risk score in a multiethnic Asian Population. Am Heart J. (2011) 162:291–9. doi: 10.1016/j.ahj.2011.05.016
34. Chotechuang Y, Phrommintikul A, Muenpa R, Patumanond J, Chaichuen T, Kuanprasert S, et al. The prognostic utility of grace risk score in predictive cardiovascular event rate in stemi patients with successful fibrinolysis and delay intervention in Non Pci-capable hospital: a retrospective cohort study. BMC Cardiovasc Disord. (2016) 16:212. doi: 10.1186/s12872-016-0383-3
35. Chotechuang Y, Phrommintikul A, Kuanprasert S, Muenpa R, Ruengorn C, Patumanond J, et al. Grace score and cardiovascular outcomes prediction among the delayed coronary intervention after post-fibrinolytic stemi patients in a limited Pci-capable hospital. Open Heart. (2020) 7:e001133. doi: 10.1136/openhrt-2019-001133
36. Morrow DA, Antman EM, Charlesworth A, Cairns R, Murphy SA, de Lemos JA, et al. Timi risk score for St-elevation myocardial infarction: a convenient, bedside, clinical score for risk assessment at presentation: an intravenous npa for treatment of infarcting myocardium early Ii trial substudy. Circulation. (2000) 102:2031–7. doi: 10.1161/01.CIR.102.17.2031
37. Antman EM, Cohen M, Bernink PJLM, McCabe CH, Horacek T, Papuchis G, et al. The Timi risk score for unstable angina/Non–St elevation mia method for prognostication and therapeutic decision making. JAMA. (2000) 284:835–42. doi: 10.1001/jama.284.7.835
38. Amin ST, Morrow DA, Braunwald E, Sloan S, Contant C, Murphy S, et al. Dynamic timi risk score for stemi. J Am Heart Assoc. (2013) 2:e003269. doi: 10.1161/JAHA.112.003269
39. Scirica BM, Cannon CP, Antman EM, Murphy SA, Morrow DA, Sabatine MS, et al. Validation of the thrombolysis in myocardial infarction (Timi) risk score for unstable angina pectoris and non-st-elevation myocardial infarction in the timi Iii registry. Am J Cardiol. (2002) 90:303–5. doi: 10.1016/S0002-9149(02)02468-2
40. Pollack CV Jr, Sites FD, Shofer FS, Sease KL, Hollander JE. Application of the timi risk score for unstable angina and non-St elevation acute coronary syndrome to an unselected emergency department chest pain population. Acad Emerg Med. (2006) 13:13–8. doi: 10.1197/j.aem.2005.06.031
41. Littnerova S, Kala P, Jarkovsky J, Kubkova L, Prymusova K, Kubena P, et al. Grace score among six risk scoring systems (Cadillac, Pami, Timi, Dynamic Timi, Zwolle) demonstrated the best predictive value for prediction of long-term mortality in patients with St-elevation myocardial infarction. PLoS ONE. (2015) 10:e0123215. doi: 10.1371/journal.pone.0123215
42. Eagle KA, Lim MJ, Dabbous OH, Pieper KS, Goldberg RJ, Van de Werf F, et al. A validated prediction model for all forms of acute coronary syndrome: estimating the risk of 6-month postdischarge death in an international registry. JAMA. (2004) 291:2727–33. doi: 10.1001/jama.291.22.2727
43. Abu-Assi E, García-Acuña JM, Peña-Gil C, González-Juanatey JR. Validation of the grace risk score for predicting death within 6 months of follow-up in a contemporary cohort of patients with acute coronary syndrome. Rev Esp Cardiol. (2010) 63:640–8. doi: 10.1016/S1885-5857(10)70138-9
44. Bradshaw PJ, Katzenellenbogen JM, Sanfilippo FM, Hobbs MST, Thompson PL, Thompson SC. Validation study of grace risk scores in indigenous and non-indigenous patients hospitalized with acute coronary syndrome. BMC Cardiovasc Disord. (2015) 15:151. doi: 10.1186/s12872-015-0138-6
45. Kao YT, Hsieh YC, Hsu CY, Huang CY, Hsieh MH, Lin YK, et al. Comparison of the timi, grace, pami and cadillac risk scores for prediction of long-term cardiovascular outcomes in taiwanese diabetic patients with St-segment elevation myocardial infarction: from the registry of the taiwan society of cardiology. PLoS ONE. (2020) 15:e0229186. doi: 10.1371/journal.pone.0229186
46. van der Sangen NMR, Azzahhafi J, Chan Pin Yin DRPP, Peper J, Rayhi S, Walhout RJ, et al. External validation of the grace risk score and the risk–treatment paradox in patients with acute coronary syndrome. Open Heart. (2022) 9:e001984. doi: 10.1136/openhrt-2022-001984
47. Gong IY, Goodman SG, Brieger D, Gale CP, Chew DP, Welsh RC, et al. Grace risk score: sex-based validity of in-hospital mortality prediction in canadian patients with acute coronary syndrome. Int J Cardiol. (2017) 244:24–9. doi: 10.1016/j.ijcard.2017.06.055
48. Yan AT, Yan RT, Huynh T, Casanova A, Raimondo FE, Fitchett DH, et al. Understanding physicians' risk stratification of acute coronary syndromes: insights from the Canadian Acs 2 registry. Arch Intern Med. (2009) 169:372–8. doi: 10.1001/archinternmed.2008.563
49. Carvalho JF, Belo A, Congo K, Neves D, Santos AR, Piçarra B, et al. Left Main and/or three-vessel disease in patients with non-St-segment elevation myocardial infarction and low-risk grace score: prevalence, clinical outcomes and predictors. Rev Port Cardiol (Engl Ed). (2018) 37:911–9. doi: 10.1016/j.repc.2018.03.016
50. Poldervaart JM, Langedijk M, Backus BE, Dekker IMC, Six AJ, Doevendans PA, et al. Comparison of the grace, heart and timi score to predict major adverse cardiac events in chest pain patients at the emergency department. Int J Cardiol. (2017) 227:656–61. doi: 10.1016/j.ijcard.2016.10.080
51. Chen YH, Huang SS, Lin SJ. Timi and grace risk scores predict both short-term and long-term outcomes in chinese patients with acute myocardial infarction. Acta Cardiol Sin. (2018) 34:4–12. doi: 10.6515/acs.201801_34(1).20170730b
52. D'Ascenzo F, Biondi-Zoccai G, Moretti C, Bollati M, Omedè P, Sciuto F, et al. Timi, grace and alternative risk scores in acute coronary syndromes: a meta-analysis of 40 derivation studies on 216,552 patients and of 42 validation studies on 31,625 patients. Contemp Clin Trials. (2012) 33:507–14. doi: 10.1016/j.cct.2012.01.001
53. Fujii T, Suzuki T, Torii S, Murakami T, Nakano M, Nakazawa G, et al. Diagnostic accuracy of global registry of acute coronary events (Grace) risk score in St-elevation myocardial infarction for in-hospital and 360-day mortality in Japanese patients. Circ J. (2014) 78:2950–4. doi: 10.1253/circj.CJ-14-0808
54. Wang JJ, Fan Y, Zhu Y, Zhang JD, Zhang SM, Wan ZF, et al. Biomarkers enhance the long-term predictive ability of the kamir risk score in Chinese patients with St-elevation myocardial infarction. Chin Med J (Engl). (2019) 132:30–41. doi: 10.1097/CM9.0000000000000015
55. Gong X, Zhang T, Feng S, Song D, Chen Y, Yao T, et al. Association between N-terminal Pro-Bnp and 12 months major adverse cardiac events among patients admitted with nstemi. Ann Palliat Med. (2021) 10:5231–43. doi: 10.21037/apm-20-2538
56. Lin F-J, Tseng W-K, Yin W-H, Yeh H-I, Chen J-W, Wu C-C. Residual risk factors to predict major adverse cardiovascular events in atherosclerotic cardiovascular disease patients with and without diabetes mellitus. Sci Rep. (2017) 7:9179. doi: 10.1038/s41598-017-08741-0
57. Wessler BS, Lai Yh L, Kramer W, Cangelosi M, Raman G, Lutz JS, et al. Clinical prediction models for cardiovascular disease: tufts predictive analytics and comparative effectiveness clinical prediction model database. Circ Cardiovasc Qual Outcomes. (2015) 8:368–75. doi: 10.1161/CIRCOUTCOMES.115.001693
58. Buccheri S, D'Arrigo P, Franchina G, Capodanno D. Risk stratification in patients with coronary artery disease: a practical walkthrough in the landscape of prognostic risk models. Interv Cardiol. (2018) 13:112–20. doi: 10.15420/icr.2018.16.2
59. Di Tanna GL, Wirtz H, Burrows KL, Globe G. Evaluating risk prediction models for adults with heart failure: a systematic literature review. PLoS ONE. (2020) 15:e0224135. doi: 10.1371/journal.pone.0224135
60. Cai R, Wu X, Li C, Chao J. Prediction models for cardiovascular disease risk in the hypertensive population: a systematic review. J Hypertens. (2020) 38:1632–9. doi: 10.1097/HJH.0000000000002442
61. Gopal DP, Usher-Smith JA. Cardiovascular risk models for South Asian populations: a systematic review. Int J Public Health. (2016) 61:525–34. doi: 10.1007/s00038-015-0733-4
62. Pencina MJ, D'Agostino RB. Sr. Evaluating discrimination of risk prediction models: the C statistic. JAMA. (2015) 314:1063–4. doi: 10.1001/jama.2015.11082
63. Steyerberg EW, Vickers AJ, Cook NR, Gerds T, Gonen M, Obuchowski N, et al. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology. (2010) 21:128–38. doi: 10.1097/EDE.0b013e3181c30fb2
64. Cook NR. Use and misuse of the receiver operating characteristic curve in risk prediction. Circulation. (2007) 115:928–35. doi: 10.1161/CIRCULATIONAHA.106.672402
65. Crowson CS, Atkinson EJ, Therneau TM. Assessing calibration of prognostic risk scores. Stat Methods Med Res. (2016) 25:1692–706. doi: 10.1177/0962280213497434
66. Van Calster B, McLernon DJ, van Smeden M, Wynants L, Steyerberg EW, Bossuyt P, et al. Calibration: the achilles heel of predictive analytics. BMC Med. (2019) 17:230. doi: 10.1186/s12916-019-1466-7
67. Grant SW, Collins GS, Nashef SAM. Statistical primer: developing and validating a risk prediction model†. Eur J Cardiothorac Surg. (2018) 54:203–8. doi: 10.1093/ejcts/ezy180
68. Egger M, Smith GD. Meta-Analysis: potentials and promise. BMJ. (1997) 315:1371–4. doi: 10.1136/bmj.315.7119.1371
69. Riley RD, Higgins JPT, Deeks JJ. Interpretation of random effects meta-analyses. BMJ. (2011) 342:d549. doi: 10.1136/bmj.d549
70. Mahajan SM, Heidenreich P, Abbott B, Newton A, Ward D. Predictive models for identifying risk of readmission after index hospitalization for heart failure: a systematic review. Eur J Cardiovasc Nurs. (2018) 17:675–89. doi: 10.1177/1474515118799059
71. Carrillo-Larco RM, Altez-Fernandez C, Pacheco-Barrios N, Bambs C, Irazola V, Miranda JJ, et al. Cardiovascular disease prognostic models in latin america and the caribbean: a systematic review. Glob Heart. (2019) 14:81–93. doi: 10.1016/j.gheart.2019.03.001
72. Laureano-Phillips J, Robinson RD, Aryal S, Blair S, Wilson D, Boyd K, et al. Heart score risk stratification of low-risk chest pain patients in the emergency department: a systematic review and meta-analysis. Ann Emerg Med. (2019) 74:187–203. doi: 10.1016/j.annemergmed.2018.12.010
Keywords: risk prediction model, prognostic model, acute myocardial infarction, validation, Southeast Asia
Citation: Ismail SR, Khalil MKN, Mohamad MSF and Azhar Shah S (2022) Systematic review and meta-analysis of prognostic models in Southeast Asian populations with acute myocardial infarction. Front. Cardiovasc. Med. 9:921044. doi: 10.3389/fcvm.2022.921044
Received: 15 April 2022; Accepted: 22 June 2022;
Published: 26 July 2022.
Edited by:
Sebhat Erqou, Brown University, United StatesReviewed by:
HuiJun Chih, Curtin University, AustraliaJunsuk Josh Ko, Duke-NUS Medical School, Singapore
Copyright © 2022 Ismail, Khalil, Mohamad and Azhar Shah. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Shamsul Azhar Shah, drsham@ppukm.ukm.edu.my