Changing performance of surgical risk scores according to the endpoint of postoperative mortality in infective endocarditis patients

Background: The optimal endpoint for reporting early mortality after cardiac operations for infective endocarditis (IE), as well as the optimal mortality target for surgical risk scores, are unresolved questions.

Methods: Five risk scores created specifically to predict early mortality after cardiac operations for definite IE, and the European System for Cardiac Operative Risk Evaluation II, were assessed in terms of calibration, discrimination and accuracy in predicting early mortality following cardiac surgery for IE. The evaluation was based on five definite endpoints of postoperative mortality: In-hospital, 30-day, in-hospital/30-day, six-month, and one-year mortality. The six risk scores were tested in a population of 991 patients with definite IE who underwent 1,014 cardiac operations at five European university-affiliated centers.

Results: There were 133 (13.1%) hospital deaths after surgery. Overall, 10% of patients died within 30 days after surgery, 10.4% of survivors died between 30 days and six months after surgery, and another 5.5% between six months and one year after surgery. All risk scores showed good prediction accuracy and at least acceptable discrimination for all endpoints of postoperative mortality. However, only one (IE-specific) risk score exhibited acceptable calibration for every endpoint of postoperative mortality.

Conclusions: Since mortality decreases slowly throughout the first year after cardiac surgery for IE, it may be appropriate to report both in-hospital and one-year mortality (coupled endpoint) for this condition. For both endpoints, only one of the risk scores considered in this study showed acceptable calibration and discrimination.

Introduction

Clinical trials exploring early postoperative outcomes in subjects undergoing cardiac operations for infective endocarditis (IE) either use in-hospital or 30-day mortality as an endpoint (1–3); combination of the two, namely in-hospital/30-day mortality, is sometimes adopted (4–6), and six-month mortality is occasionally reported (7). Accordingly, risk scores specifically devised to predict early mortality after surgery in the subset of patients with IE are developed and validated using one of the parameters above as mortality target. These risk scores are usually based on the patient's comorbidities, the complexity of planned surgery, and the nature of involved pathogen (2–7). This variability, however, impacts clinical decision-making and counseling the patients and their family, complicates the interpretation of outcomes and research comparability, and hinders the development of standardized guidelines.

To date there are no studies indicating which should be the optimal endpoint for reporting early mortality after cardiac surgery for IE. Consequently, from a methodological point of view, the optimal target of mortality for the IE-specific surgical risk scores remains an unresolved and urgent question. This is a significant issue since IE is a life-threatening disease that may require challenging surgery and cause higher rates of early postoperative complications than other less complex cardiac pathologies, as well as longer hospital stays and higher costs of health care. In addition, the timing of onset of complications after surgery for IE can also be very different. It is known indeed that severe complications such as surgically related infections may appear late after surgery and that, unfortunately, recurrence of infection in patients operated on for IE is not such a rare event. By convention, every surgical site infection that occurs within one year from surgery should be considered related to that surgery, at least until proven otherwise (1). Therefore, the present authors wondered whether the same endpoints of mortality that are usually adopted for other heart diseases of surgical interest might also be used for IE.

The aim of the present short communication is to show any changes in the performance of six risk scores in predicting early mortality following cardiac operations for IE based on five definite endpoints of postoperative mortality: In-hospital, 30-day, in-hospital/30-day, six-month, and one-year mortality. The risk scores were evaluated about calibration, discrimination, and accuracy of prediction.

Methods

In a clinical investigation by the present authors, which has been published recently in The American Heart Journal (8), five risk scores created specifically and selectively to predict early mortality after cardiac surgery for definite IE—STS-IE (Society of Thoracic Surgeons for IE) (2); PALSUSE (Prosthetic valve, Age ≥70, Large intra-cardiac destruction, Staphylococcus spp, Urgent surgery, Sex [female], logistic EuroSCORE ≥10 (3); ANCLA (Anemia, New York Heart Association functional class IV, Critical state, Large intra-cardiac destruction, surgery on thoracic Aorta) (4); AEPEI II (Association pour l’Étude et la Prévention de l’Endocardite Infectieuse II) (5); and APORTEI (Análisis de los factores PROnósticos en el Tratamiento quirúrgico de la Endocarditis Infecciosa) (6) (Supplementary Tables S1–S3) –, as well as the European System for Cardiac Operative Risk Evaluation II (EuroSCORE II) (9), which has been widely and commonly used throughout Europe since 2012 to evaluate the mortality risk after any type of cardiac operation, were validated and compared in terms of calibration, discrimination and accuracy in predicting 30-day postoperative mortality. The above risk scores were selected regarding their acceptance in Literature, clinical applicability, and availability of the collected data (8).

In the present study, the same risk scores are now evaluated according to the endpoint of early postoperative mortality. The study patients, definitions and statistical methods are approximately the same as in the previously cited study from the authors (8). Baseline characteristics, surgical and IE-related features, and early postoperative deaths of 991 adults (20 years of age or older) with definite IE who underwent 1,014 cardiac operations at five European university-affiliated centers were prospectively recorded in computerized databases. Only the cases with known pathogen, without missing values for all considered variables, and having at least one-year of follow-up were retained for the present retrospective analyses. The patients underwent surgery for IE during different time periods depending on the center. The series of cases were consecutive for all but one center. The five series were merged and managed as one single cohort. Definite IE was defined according to the 2023 Duke-International Society for Cardiovascular Infectious Diseases criteria (1). For each risk score's computation, the definitions of variables used were those reported in the corresponding original paper (2–6, 9). A mix of definitions of variables deriving from the six risk scores considered was adopted for reporting data in the present manuscript. The in-hospital death was defined as death occurring before hospital discharge; the 30-day death was defined as death occurring within the postoperative day 30 (regardless of hospital discharge); the in-hospital/30-day death was defined as death occurring before hospital discharge, or within the postoperative day 30 for discharged patients; the six-month and one-year deaths were defined as deaths occurring within six months and one year after surgery, respectively. The causes of death were not investigated.

An informed consent for future retrospective studies was obtained preoperatively from each patient. The study protocol conforms to the ethical guidelines of the 1975 Declaration of Helsinki as reflected in a priori approval by the human research committee of each participating institution.

Statistical methods

Data is presented as absolute number, percentage, or mean ± standard deviation. Calibration of each risk score was assessed using calibration plot analysis. Initially, a regression line (linear trendline) was derived from plot data of each risk score. The regression line was estimated using the observed event rates as a dependent variable and, as an independent variable, the estimated probabilities by the score. Slope and intercept were derived from equation of each regression line and used to compare the risk scores to each other. For intercept-values closer to 0, the closer to 1 the slope-value, the higher the calibration of the risk score. For identical or very similar slope-values, the closer to 0 the intercept-value, the higher the risk score calibration. Calibration was defined as acceptable when the following conditions were satisfied simultaneously: (1) Slope-values range between 0.7 and 1.3; (2) Intercept-values range between −0.1 and 0.1. Later, a regression curve (polynomial trendline) was obtained to evaluate semi-quantitatively each risk score. The curve was estimated using the observed event rates as a dependent variable and, as an independent variable, the estimated probabilities by the score. The order of equation and value of intercept were arbitrarily set to 6 and 0, respectively. The square of the correlation coefficient r (R² or R-squared) was adopted to measure how well the trendline approximated the plot data (R²-values range from 0 to 1, with higher values indicating a better fit). Both linear and polynomial trendline approximated well the plot data for R²-values >0.7. Afterwards, a receiver-operating characteristic curve was plotted for each risk score and the area under the curve (AUC) was calculated as a measure of discrimination of the model (the possible values of AUC range from 0.5, no discrimination, to 1, perfect discrimination). Finally, accuracy of prediction of each risk score was evaluated with the ratio between the observed and the expected mortality (observed-to-expected ratio, O/E), and the Brier's score (scores are bound between 0 and 1, with 0 indicating perfect accuracy and 1 perfect inaccuracy). Acquisition of the data entries was performed using Microsoft Office Excel, version 2007. Data analysis was performed using the SPSS software package for Windows, version 13.0.

Results

Early postoperative mortality

There were 133 (13.1%) hospital deaths after surgery (only 100 within postoperative day 30). One patient died after hospital discharge but within the postoperative day 30. Out of discharged patients, 7.1% died within six months from surgery, and 12.3% within one year. Overall, 10% of patients died within 30 days after surgery, 10.4% of survivors died between 30 days and six months after surgery, and another 5.5% between six months and one year after surgery. There were 23 (2.3%) cases of reoperation due to recurrence of cardiac infection; two patients died within 30 days from surgery, one patient within six months, and another one within one year (Table 1).

Table 1

Table 1. Perioperative variables frequency in the overall series of cases (N = 1,014)^a,b.

Risk score performance

Calibration of the six risk scores for the five definite endpoints of postoperative mortality is summarized in Table 2. While the AEPEI II and APORTEI scores showed acceptable calibration for in-hospital mortality, the STS-IE, ANCLA and AEPEI II scores showed acceptable calibration for one-year mortality. Only the AEPEI score II showed acceptable calibration for all endpoints (Table 2, Figure 1, and Supplementary Figures S1, S2).

Table 2

Table 2. Calibration of the six risk scores for the five definite endpoints of postoperative mortality (N = 1,014)^a,b.

Figure 1

Figure 1. Calibration and discrimination of the six risk scores in study patients for in-hospital and one-year mortality (i.e., the two components of the proposed combined endpoint for reporting early mortality after cardiac surgery for IE) are shown here using calibration and ROC curves, respectively.

Four risk scores (EuroSCORE II, PALSUSE, ANCLA and APORTEI) showed the highest AUC values for 30-day mortality, and two risk scores (STS-IE and AEPEI II) for in-hospital/30-day mortality. For all risk scores, the lowest AUC values were observed for one-year mortality, though the lower limit of the 95% confidence interval of AUC was always above 0.5 (0.548 or greater) (Figure 1 and Supplementary Figures S1, S2).

Overall, O/E was always equal to, or greater than, 0.928 and equal to, or less than, 1.023, for all risk scores and for all mortality endpoints; Brier's score was 0.179 or less. For all but one risk score, the best predictive accuracy was observed for one-year mortality, at least based on O/E (O/E closer to 1); the AEPEI score II showed the best O/E for 30-day mortality; the PALSUSE score showed the highest predictive accuracy both for six-month and one-year mortality. According to Brier's score instead, the best predictive accuracy was observed for 30-day mortality for all risk scores (Supplementary Figure S1).

Comment

Six surgical risk scores were evaluated in this study. Their main characteristics are reported in Supplementary Tables S1–S3 and summarized below. EuroSCORE II consists of 18 variables and has been modeled from a contemporary surgical cohort of 22,381 patients, including 497 (2.2%) with active IE (9). It has been created to predict 30-day mortality after any cardiac operation. Since its publication (2012) it was quickly and widely introduced into clinical practice in Europe. The STS-IE score has been published in 2011 (2). It derives from a series of 13,619 IE-patients who had been operated on at 824 centers in North America between 2002 and 2008. It includes 13 variables: nine host-related factors, two heart-related factors, and two extracardiac events. It is routinely used in North America. The PALSUSE score has been published in 2014 (3). It derives from a series of 437 IE-patients who had been operated on at 26 Spanish hospitals between 2008 and 2010. It includes seven variables: three host-related, two heart-related, one pathogen-related, and logistic EuroSCORE ≥10%. The ANCLA score has been published in 2017 (4). It derives from a series of 138 IE-patients who had operated on at one-single Italian university hospital between 1999 and 2015. It includes two host-related variables, one heart-related variable, one laboratory finding, and one surgical data. The AEPEI score II also has been published in 2017 (5). It is the three-variable alternate model of AEPEI score I. Both risk scores derive from a series of 361 consecutive patients who had undergone surgery for IE in one Italian (period, 1999–2015) and seven French (throughout 2008) cardiac surgery centers. The AEPEI score II includes only two host-related variables and one extra-cardiac event. Unlike the three previous risk scores, which derive directly from a multivariable analysis, the APORTEI score derives from a meta-analysis performed on 16 selected studies (publication years, 2007–2018) comprising 7,484 IE-patients (6). The APORTEI score includes 11 variables: six host-related factors, three heart-related factors, one extra-cardiac event, and one pathogen-related factor.

According to the statistical analyses of the present study, (1) all risk scores showed good accuracy of prediction and acceptable discrimination for every index endpoint, though the highest discrimination was for death occurring before hospital discharge and/or within 30 days from surgery; (2) EuroSCORE II had a good discrimination for early postoperative mortality, though its calibration was poor (8); (3) The AEPEI score II showed an acceptable calibration for all endpoints of early postoperative mortality; (4) While the AEPEI II and APORTEI scores showed acceptable calibration for in-hospital mortality, the STS-IE, ANCLA and AEPEI II scores showed acceptable calibration for one-year mortality.

Hence, based on these results and those of previous studies (8, 10), EuroSCORE II should be the recommended risk model to discriminate mortality that occurs in IE-patients immediately after cardiac surgery. However, since its calibration is poor and generally lower than IE-specific risk scores, EuroSCORE II should be accompanied by specific risk scores such as the AEPEI II or the APORTEI score. In addition, the STS-IE, ANCLA, and AEPEI II scores, which showed acceptable discrimination and calibration one year after surgery, should be adopted to identify patients who need to be carefully monitored after hospital discharge due to the high risk of IE recurrence.

The authors of this study intend to propose a coupled endpoint—in-hospital & at one year from surgery—to report mortality after cardiac operations for IE. This original approach aims to not underestimate the true impact of cardiac surgery in IE-patients and derives from the following interdependent concepts: (1) Either in IE-patients treated with medical therapy alone or in combination with surgery, the mortality rate declines slowly throughout the first year after hospital discharge; (2) In IE-patients, it is reasonable to distinguish between immediate and early postoperative death, since the causes and mechanisms underlying the two types of death are different. In fact, complications after surgery for IE are time-dependent in nature. While immediate postoperative deaths are usually related to the host's preoperative comorbidity and nutritional status, heart failure due to extensive cardiac destruction that requires complex surgical reconstruction, pathogen virulence, and uncontrolled infection, early postoperative deaths are generally related to patient's failure to recover from surgical stress and recurrence of infection caused by incomplete surgical cleanup or failure of antibiotic treatment (7, 11, 12). These two peculiar aspects of IE-surgery explain at least partially why it can be very different from surgery for other heart diseases, pose more complex challenges and require original and diversified approaches.

In conclusion, since mortality decreased slowly throughout the first year after surgery, also for patients in this study, from a methodological point of view it may be appropriate for IE-patients undergoing cardiac surgery to use a coupled endpoint and report both in-hospital and one-year mortality. This original approach could provide a more accurate picture of the outcomes of surgery for IE.

Data availability statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics statement

The studies involving humans were approved by Comitato Etico Unico Regionale. The studies were conducted in accordance with the local legislation and institutional requirements. The participants provided their written informed consent to participate in this study.

Author contributions

GG: Conceptualization, Data curation, Formal Analysis, Investigation, Methodology, Project administration, Resources, Software, Supervision, Validation, Visualization, Writing – original draft, Writing – review & editing. AF: Conceptualization, Investigation, Validation, Visualization, Writing – original draft, Writing – review & editing, Methodology. MI: Data curation, Formal Analysis, Methodology, Visualization, Writing – original draft, Writing – review & editing. IV: Conceptualization, Investigation, Validation, Visualization, Writing – original draft, Writing – review & editing, Methodology. AM: Conceptualization, Data curation, Validation, Visualization, Writing – original draft, Writing – review & editing. GS: Conceptualization, Investigation, Supervision, Visualization, Writing – original draft, Writing – review & editing. AP: Conceptualization, Investigation, Validation, Visualization, Writing – original draft, Writing – review & editing, Methodology. EM: Conceptualization, Investigation, Supervision, Visualization, Writing – original draft, Writing – review & editing, Data curation, Methodology, Validation.

Funding

The author(s) declare that no financial support was received for the research and/or publication of this article.

Conflict of interest

The author(s) declared that they were an editorial board member of Frontiers, at the time of submission. This had no impact on the peer review process and the final decision.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fcvm.2025.1543049/full#supplementary-material

Supplementary Figure S1 | Changes in calibration, discrimination and accuracy in predicting in-hospital (In-H), 30-day (30-D), in-hospital/30-day (In-H/30-D), six-month (6-M) and one-year (1-Y) mortality after surgery for IE of six risk scores (N = 1,014).

Supplementary Figure S2 | Calibration curves for in-hospital, 30-day, in-hospital/30-day, six-month and one-year mortality after surgery for IE of six risk scores (N = 1,014).

Supplementary Figure S3 | Receiver-operating characteristics (ROC) curves for in-hospital, 30-day, in-hospital/30-day, six-month and one-year mortality after surgery for IE of six risk scores (N = 1,014).

References

1. Delgado V, Ajmone Marsan N, de Waha S, Bonaros N, Brida M, Burri H, et al. 2023 ESC guidelines for the management of endocarditis. Eur Heart J. (2023) 44(39):3948–4042. doi: 10.1093/eurheartj/ehad193

PubMed Abstract | Crossref Full Text | Google Scholar

2. Gaca JG, Sheng S, Daneshmand MA, O'Brien S, Rankin JS, Brennan JM, et al. Outcomes for endocarditis surgery in North America: a simplified risk scoring system. J Thorac Cardiovasc Surg. (2011) 141(1):98–106. doi: 10.1016/j.jtcvs.2010.09.016

PubMed Abstract | Crossref Full Text | Google Scholar

3. Martínez-Sellés M, Muñoz P, Arnáiz A, Moreno M, Gálvez J, Rodríguez-Roda J, et al. Valve surgery in active infective endocarditis: a simple score to predict in-hospital prognosis. Int J Cardiol. (2014) 175(1):133–7. doi: 10.1016/j.ijcard.2014.04.266

PubMed Abstract | Crossref Full Text | Google Scholar

4. Gatti G, Benussi B, Gripshi F, Della Mattia A, Proclemer A, Cannatà A, et al. A risk factor analysis for in-hospital mortality after surgery for infective endocarditis and a proposal of a new predictive scoring system. Infection. (2017) 45(4):413–23. doi: 10.1007/s15010-016-0977-9

PubMed Abstract | Crossref Full Text | Google Scholar

5. Gatti G, Perrotti A, Obadia JF, Duval X, Iung B, Alla F, et al. Simple scoring system to predict in-hospital mortality after surgery for infective endocarditis. J Am Heart Assoc. (2017) 6(7):e004806. doi: 10.1161/JAHA.116.004806

PubMed Abstract | Crossref Full Text | Google Scholar

6. Varela Barca L, Fernández-Felix BM, Navas Elorza E, Mestres CA, Muñoz P, Cuerpo-Caballero G, et al. Prognostic assessment of valvular surgery in active infective endocarditis: multicentric nationwide validation of a new score developed from a meta-analysis. Eur J Cardiothorac Surg. (2020) 57(4):724–31. doi: 10.1093/ejcts/ezz328

PubMed Abstract | Crossref Full Text | Google Scholar

7. Park LP, Chu VH, Peterson G, Skoutelis A, Lejko-Zupa T, Bouza E, et al. Validated risk score for predicting 6-month mortality in infective endocarditis. J Am Heart Assoc. (2016) 5(4):e003016. doi: 10.1161/JAHA.115.003016

PubMed Abstract | Crossref Full Text | Google Scholar

8. Gatti G, Fiore A, Ismail M, Dralov A, Saade W, Costantino V, et al. Prediction of 30-day mortality after surgery for infective endocarditis using risk scores: insights from a European multicenter comparative validation study. Am Heart J. (2024) 275:108–18. doi: 10.1016/j.ahj.2024.05.021

PubMed Abstract | Crossref Full Text | Google Scholar

9. Nashef SA, Roques F, Sharples LD, Nilsson J, Smith C, Goldstone AR, et al. EuroSCORE II. Eur J Cardiothorac Surg. (2012) 41(4):734–44. doi: 10.1093/ejcts/ezs043

PubMed Abstract | Crossref Full Text | Google Scholar

10. El Hatimi S, Erpelding ML, Selton-Suty C, Botros JB, Goehringer F, Berthelot E, et al. Predictive performance of surgical mortality risk scores in infective endocarditis. Eur J Cardiothorac Surg. (2024) 65(1):ezad433. doi: 10.1093/ejcts/ezad433

PubMed Abstract | Crossref Full Text | Google Scholar

11. Thuny F, Giorgi R, Habachi R, Ansaldi S, Le Dolley Y, Casalta JP, et al. Excess mortality and morbidity in patients surviving infective endocarditis. Am Heart J. (2012) 164(1):94–101. doi: 10.1016/j.ahj.2012.04.003

PubMed Abstract | Crossref Full Text | Google Scholar

12. Shih CJ, Chu H, Chao PW, Lee YJ, Kuo SC, Li SY, et al. Long-term clinical outcome of major adverse cardiac events in survivors of infective endocarditis. Circulation. (2014) 130(19):1684–91. doi: 10.1161/CIRCULATIONAHA.114.012717

PubMed Abstract | Crossref Full Text | Google Scholar