Development and Validation a Nomogram for Predicting Overall Survival in Patients With Intrahepatic Cholangiocarcinoma

Background: This study aims to establish an effective nomogram to predict the overall survival of patients with intrahepatic cholangiocarcinoma (ICC).

Patients and Methods: Data used to build the nomogram comes from the Surveillance, Epidemiology, and End Results (SEER) database. Patients diagnosed with ICC between 2005 and 2016 were retrospectively collected. Prediction accuracy and discrimination ability of the nomogram was evaluated by concordance index (C-index) and calibration curve. The area under receiver operating characteristic (ROC) curve (AUC) and decision curve analysis (DCA) were used to compare the precision of the 1-, 3-, and 5-year survival of the nomogram with 8th American Joint Commission on Cancer (AJCC) tumor–node–metastasis (TNM) staging system. Finally, it was verified in a prospective study of patients diagnosed with ICC in the Second Affiliated Hospital of Nanchang University from 2013 to 2020 by bootstrap resampling.

Result: The study contains two parts of data; we establish a nomogram using external data, and we conducted internal verification and external verification. The nomogram that we have established has good calibration, with a concordance index (C-index) of 0.75 (95% CI, 0.74–0.76) for overall survival (OS) prediction. The AUC value of the nomogram predicting 1-, 3-, and 5-year OS rates were 0.821, 0.828, and 0.836, which were higher than those of the 8th AJCC TNM staging systems. The calibration curve for the probability of survival between prediction by nomogram and actual observation shows good agreement. The nomogram showed better accuracy than the 8th edition AJCC TNM staging.

Conclusion: The nomogram established can provide a more accurate prognosis for patients with intrahepatic cholangiocarcinoma.

Introduction

Intrahepatic cholangiocarcinoma (ICC) originates from the epithelial cells of the intrahepatic bile duct, which can be a small intrahepatic bile duct or a large intrahepatic bile duct near the bifurcation of the hepatic duct (1). The incidence of ICC is second only to hepatocellular carcinoma (HCC) and accounts for ~5–30% of all liver malignancies (2, 3). In addition, the incidence and mortality of ICC have increased worldwide in recent years (4, 5). Unfortunately, the prognosis of patients with ICC, whether surgically or non-surgically, is not satisfactory (6, 7). ICC is significantly different from HCC in behavior, and the clinical features, imaging findings, and treatment methods of ICC are also different from HCC and distal cholangiocarcinoma (8). Therefore, ICC is a malignant tumor different from other tumors; it needs a unique prognostic prediction model of its own. A good predictive model can help doctors choose the best treatment to suit the individual prognosis of different patients. It is very important for our clinicians. At present, our most commonly used traditional staging system is the 8th edition American Joint Commission on Cancer (AJCC) system. A recent study (9) had proved that the AJCC system is not suitable for all ICC patients; it only considers tumor size, lymph node metastasis, and distant metastasis but does not consider other patient characteristics such as age, gender, and treatment methods. Therefore, we urgently need a staging system for the individual prognosis of ICC patients.

The purpose of this study is to develop a nomogram for predicting overall survival (OS) of ICC patients using a cohort from the Surveillance, Epidemiology, and End Results (SEER) database and to conduct internal and external verification. This nomogram can provide clinicians with a better tool for risk stratification, prognosis prediction, and therefore clinical decision.

Patients and Methods

Development Cohort

The SEER program of the National Cancer Institute provides data on cancer incidence and survival rates covering 30% of the US population. In this study, we collected patients diagnosed with ICC from 2005 to 2016 from SEER database, by using the SEER^*Stat (National Cancer Institute, Bethesda, MD, USA) software version 8.3.8. The data we collected were from the International Classification of Diseases for Oncology 3rd edition (ICD-O-3), primary site code C22.1 (intrahepatic bile duct), along with histological/behavior code 8160.3 (cholangiocarcinoma). The exclusion criteria were as follows: (1) diagnosed under 18 years old, (2) combined with other primary tumors, (3) incomplete clinical data, (4) unclear follow-up information, and (5) surgical methods that did not achieve the purpose of treatment. Patients' clinical characteristics were extracted from the SEER database, including age at diagnosis, gender, tumor size, tumor–node–metastasis (TNM) stage, and follow-up information. The staging system uses the 8th AJCC edition system. The AJCC TNM 8th edition stage was calculated from the 6th or 7th edition TNM stages and other characteristics like tumor size (10, 11). OS refers to the time from diagnosis to death or the last follow-up. The approval and informed consent of the institutional review committee were exempted because the SEER database is a public database, which is open access for anyone who has registered an account and signed the authorization.

External Validation Cohort

Eighty-eight patients diagnosed of ICC between 2013 and 2020 at the Second Hospital of Nanchang University were used as validation cohort. The criteria for the validation cohort and the inclusion criteria and exclusion criteria for the development cohort are fully chaired. The diagnosis of ICC patients who have not undergone surgery is based on clinical, radiographic, and serum markers (12). The clinical characteristics of all patients are collected from the electronic medical record. All procedures performed in this study involving human participants comply with the ethical standards of the institution and/or national research committee and with the 1964 Helsinki Declaration and amendment or similar ethical standards. External verification data have been approved by the Second Hospital of Nanchang University ethics committee.

Statistical Analysis

Continuous data were presented by median ± range and compared with Student's t-test or Mann–Whitney U-test. Categorical data were presented by frequency (proportion) and compared with chi-square test. Cox regression model was used for multivariate analysis, and the independent risk factors that affect OS were extracted. The associated 95% confidence interval (CI) and hazard ratio (HR) were also calculated. OS were calculated by Kaplan–Meier curves and compared by the log-rank tests. Independent risk factors were used to establish a nomogram for 1-, 3-, and 5-year OS by using rms package in R Studio (3). The accuracy of the nomogram is evaluated by the C-index and the calibration curve (13).

The verification of the nomogram includes two parts: internal verification and external verification. First, we use the caret package of R Studio to divide the data collected from the SEER database into 30 and 70%, and the 30% of data were used for internal verification by bootstrap with 1,000 resamples. Second, clinical data from our institution were collected for external verification using the established nomogram. Lastly, the area under receiver operating characteristic (ROC) curve (AUC) was used to evaluate and compare the precision of the 1-, 3-, and 5-year survival of the nomogram. All statistical analyses were analyzed using SPSS version 26 (SPSS, Inc., Chicago, IL, USA) and R Studio version 1.3.1056 with R packages survival, rms, caret, survival ROC, and foreign packages. A two-tailed P < 0.05 was considered statistically significant.

Result

Patient Characteristics

A total of 1933 patients diagnosed with ICC were included from the SEER database during 2005 and 2016. Another 88 patients diagnosed with ICC were included as validation cohort from our institution during 2013 and 2020. The median age was 67 years (20–85 range) in the SEER database and 66 years (30–86 range) in the external validation cohort. The clinical characteristics are summarized in Table 1.

TABLE 1

Table 1. Clinical characteristics of patients in the SEER database and external validation cohort.

Os Analysis

The median follow-up period was 11 months (0–83 range) in the development cohort and 14.5 months (1–79 range) in the validation cohort. The mortality rate was 77.9% (1505/1933), and the OS rates at 1, 3, and 5 years were 44.9, 9.1, and 3.1%. Kaplan–Meier curve analysis showed significant differences according to different characteristics. No surgical resection (P < 0.001, Figure 1F), tumor size >5 cm (P < 0.05, Figure 1I), older age (P < 0.001, Figure 1A), lymph node metastases (P < 0.001, Figure 1D), distant metastases (P < 0.001, Figure 1E), higher T stage (P < 0.001, Figure 1C), no radiotherapy (P < 0.001, Figure 1G), no chemotherapy (P < 0.001, Figure 1H), and male gender (P < 0.05, Figure 1B) showed poorer OS.

FIGURE 1

Figure 1. Overall survival rates according to patient characteristics: (A) Age; (B) Gender; (C) T stage; (D) LN metastasis; (E) Metastasis; (F) Surgery; (G) Radiation; (H) Chemotherapy; (I) Tumor size. LN, lymph node.

Univariate and Multivariate Analyses of Effects of Factors on OS

In the COX regression univariate analysis, age, gender, surgical treatment, radiation treatment, chemotherapy, T stage, N stage, M stage, and tumor size were significantly correlated to the OS of ICC patients. After adjusting for other risk factors, in the multivariate COX regression analysis, age, surgical treatment, T stage, N stage, M stage, radiation treatment, and chemotherapy were significantly correlated to the OS of ICC patients (Table 2).

TABLE 2

Table 2. Univariate and multivariate analyses for OS in patients with ICC.

Nomogram Construction and Validation for OS

All the independent risk factors that have a significant impact on OS were included in the nomogram for predicting 1-, 3-, and 5-year OS in the training set (Figure 2). By adding the variable scores corresponding to each patient, it is easy to get the survival probability of different individuals. The C-index shown in the nomogram was 0.754 (95% CI, 0.746–0.762), with good accuracy. During internal verification, the nomogram showed good accuracy with C-index of 0.761 (95% CI, 0.751–0.771). In external verification, the nomogram also showed good accuracy with a C-index of 0.767 (95% CI, 0.735–0.799). In the 8th AJCC TNM staging system, the C-index is 0.607 (95% CI, 0.598–0.612). In the internal and external verifications, the 1-, 3-, and 5-year calibration curves showed that the survival rates predicted by the nomogram were in good agreement with the actual survival rates (Figure 3).

FIGURE 2

Figure 2. Nomogram predicting 1-, 3-, and 5-year OS of patients with ICC. OS, overall survival.

FIGURE 3

Figure 3. Calibration plots of the nomogram for 1-, and 3-year OS prediction of the training set (A–C) and internal verification set (D–F) and external verification set (G–I). X-axis represents the nomogram-predicted probability of survival; Y-axis represents the actual OS probability. A perfectly accurate nomogram prediction model would result in a plot that the observed and predicted probabilities for given groups fall along the line. Dots with bars represent nomogram-predicted probabilities along with 95% confidence interval. OS, overall survival.

Survival Analysis According to the Risk Stratification Based on the Nomogram

We divided the probability scores of all patients into two parts according to the average number. Patients with higher scores than the average are defined as high risk, and those lower are defined as low risk. As shown in Figure 4, we could see that the survival rate of low-risk patients was significantly higher than that of high-risk patients (P < 0.001).

FIGURE 4

Figure 4. OS stratified by the risk levels of the nomogram-predicted survival probabilities. OS, overall survival.

Comparison of the Performance of the Nomogram and 8th AJCC TNM Stage System

Finally, we analyzed the value of AUC to compare the discriminative ability of the established nomogram and the 8th edition TNM staging system (Figure 5). For the entire development cohort, the AUC values of the nomogram used to predict 1-, 3-, and 5-year OS were, respectively, 0.821, 0.828, and 0.836. However, the AUC values of the 8th AJCC TNM staging system were 0.650, 0.722, and 0.752. In DCA, within a wide range of threshold probabilities, the established nomogram had a higher net benefit in predicting 1-, 3-, and 5-year OS compared with the 8th AJCC TNM staging system (Figure 6). In general, the nomogram we have established had a better recognition ability and precision than the 8th AJCC TNM staging system.

FIGURE 5

Figure 5. The ROC curves of the nomogram (A–C) and the 8th AJCC TNM staging system (D–F) for 1-, 3-, and 5-year OS prediction. OS, overall survival.

FIGURE 6

Figure 6. Decision curve analysis of nomograms and AJCC 8th edition staging system for predicting (A,D) 1-year OS; (B,E) 3-year OS; (C,F) 5-year OS.

Discussion

The current study used data from the SEER database to establish a nomogram for predicting patients diagnosed with ICC and verified both internally and externally. The internal verification data also came from the SEER database, while the external verification data came from an independent cohort collected by our hospital. The nomogram showed a good distinction and calibration function, which provided better clinical decision making for both patents and clinicians.

ICC is the second most common liver malignant tumor following hepatocellular carcinoma (14), and the incidence of ICC is increasing worldwide (15). It is significant for clinicians to make individualized prognostic judgments based on accurate patient information. Traditional staging system such as the 8th AJCC TNM staging system only included specific related variables and evaluated the prognosis of specific patients. A recent study has shown that the TNM staging of the 8th edition of AJCC had a moderate discrimination ability in predicting the OS of ICC patients, while there was no significant improvement in the overall prognosis compared with the 7th edition (16). Therefore, it is meaningful to establish a novel prediction system that can effectively predict the prognosis of ICC patients. Recent studies paid more attention to ICC patients who underwent surgery (2, 17, 18). The C-index of the nomogram established by us is higher than that of the 8th edition AJCC TNM staging and had better predictive ability.

Although the prognosis of ICC patients who have undergone surgery was much better than that of those who have not undergone surgery, most patients have lost the chance of radical surgery due to locally advanced or distant metastases at the time of diagnosis (19–21). In our research, there is a fact that most patients have not undergone surgery; therefore, it is very necessary to include patients who have not undergone surgery into the study. Moreover, in our research, it could be found that some advanced patients who have not received surgery have a considerable prognosis; this is not seen in other similar studies. Many recent studies have shown that tumor-related factors, such as tumor size, tumor invasion, and lymph node condition, have a certain impact on the prognosis of ICC patients (22–24). The AJCC system is developed based on these related factors, and the latest system has been updated to the eighth edition, with using TNM staging to represent the degree of tumor invasion, lymph node metastasis, and a distant metastasis. Radiotherapy and chemotherapy now play a very important role in tumor treatment. In our study, radiotherapy and chemotherapy showed a good correlation in the survival rate of patients.

In our study, we can see that higher-level TNM staging has a poor prognosis of OS. In addition, multivariate analysis showed that TNM stages are independent risk factors that affect the prognosis of ICC patients. As previously reported (25), lymph node metastasis will have an impact on the patient's postoperative review, so the lymph node affects the patient's prognosis to a certain extent.

In our time, for clinicians, individualized cancer treatment is particularly important, and on this basis, we established a nomogram. Our nomogram combines factors that are easily obtained from the clinic, which makes it easy to calculate the individualization of ICC patients. In our research, whether it is the nomogram, internal verification, or external verification, there is a relatively good C-index and calibration curve, and we compared it with the eighth edition of TNM staging with a higher C-index. The larger the C-index, to a certain extent, the more accurate the prognosis prediction (26). However, high prognostic prediction accuracy does not necessarily have good clinical applicability (27). Decision curve analysis is a novel way to evaluate models; it uses an estimated threshold probability distribution, and the weighted area under the net income curve is used as a summary metric to compare risk prediction models in the range of interest (28–30). Therefore, we quoted DCA to evaluate our nomogram and compared it with the 8th TNM staging system. The results show that our nomogram has better clinical applicability.

This study has several limitations. First, we cannot find in the SEER database the serological tests that may have an impact on OS in ICC patients, such as tumor markers and blood routines, and some related positive variables, such as surgical margins and vascular invasion, cannot be found either. These variables may be a supplement to our current stage, which will be the main part of our future research. Second, like other retrospective studies, both development and validation cohorts are affected by selection bias. Last, due to the lack of external verification data, factors that can be found in the SEER database cannot be included in the study, and the small amount of external verification samples by a single institution may lead to verification errors. More samples and multi-institution verification will be conducted to verify the accuracy of the nomogram. However, despite these limitations, we have established a nomogram with better clinical applicability and better than 8th TNM staging system.

Conclusion

All in all, we build a nomogram to predict 1-, 3-, and 5-year diagnosis ICC based on a large population. This nomogram integrates easily accessible factors and has been verified internally and externally, showing good accuracy and clinical applicability, which may help clinicians to implement individualized clinical decisions.

Data Availability Statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics Statement

Ethical review and approval was not required for the study on human participants in accordance with the local legislation and institutional requirements. Written informed consent for participation was not required for this study in accordance with the national legislation and the institutional requirements.

Author Contributions

SZ was responsible for conception, design, quality control of this study, reviewed, and edited the manuscript. CY, ZH, SZ, and KW performed the study selection, data extraction, statistical analyses, and were major contributors in writing the manuscript and contributed in classification criteria discussion. CY and ZH participated in studies selection and statistical analyses. All authors have read and approved the final version of the manuscript.

Funding

This study was supported by the Research Project of Science and Technology from Healthy Committee of Jiangxi Province (No. 20204247) and Research Project of Science and Technology from Education Bureau of Jiangxi Province (No. 2020).

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

References

1. Zhang H, Yang T, Wu M, Shen F. Intrahepatic cholangiocarcinoma: epidemiology, risk factors, diagnosis and surgical management. Cancer Lett. (2016) 379:198–205. doi: 10.1016/j.canlet.2015.09.008

PubMed Abstract | CrossRef Full Text | Google Scholar

2. Ma K, Dong B, Wang L, Zhao C, Fu Z, Che C, et al. Nomograms for predicting overall survival and cancer-specific survival in patients with surgically resected intrahepatic cholangiocarcinoma. Cancer Manag Res. (2019) 11:6907–29. doi: 10.2147/CMAR.S212149

PubMed Abstract | CrossRef Full Text | Google Scholar

3. Núñez E, Steyerberg EW, Núñez J. Regression modeling strategies. Rev Esp Cardiol. (2011) 64:501–7. doi: 10.1016/j.rec.2011.01.017

CrossRef Full Text | Google Scholar

4. Malhi H, Gores GJ. Cholangiocarcinoma: modern advances in understanding a deadly old disease. J Hepatol. (2006) 45:856–67. doi: 10.1016/j.jhep.2006.09.001

PubMed Abstract | CrossRef Full Text | Google Scholar

5. Welzel TM, McGlynn KA, Hsing AW, O'Brien TR, Pfeiffer RM. Impact of classification of hilar cholangiocarcinomas (Klatskin tumors) on the incidence of intra- and extrahepatic cholangiocarcinoma in the United States. J Natl Cancer Inst. (2006) 98:873–5. doi: 10.1093/jnci/djj234

PubMed Abstract | CrossRef Full Text | Google Scholar

6. Bergquist A, von Seth E. Epidemiology of cholangiocarcinoma. Best Pract Res Clin Gastroenterol. (2015) 29:221–32. doi: 10.1016/j.bpg.2015.02.003

CrossRef Full Text | Google Scholar

7. Mavros MN, Economopoulos KP, Alexiou VG, Pawlik TM. Treatment and prognosis for patients with intrahepatic cholangiocarcinoma: systematic review and meta-analysis. JAMA Surg. (2014) 149:565–74. doi: 10.1001/jamasurg.2013.5137

PubMed Abstract | CrossRef Full Text | Google Scholar

8. Paik KY, Jung JC, Heo JS, Choi SH, Choi DW, Kim YI. What prognostic factors are important for resected intrahepatic cholangiocarcinoma?. J Gastroenterol Hepatol. (2008) 23:766–70. doi: 10.1111/j.1440-1746.2007.05040.x

PubMed Abstract | CrossRef Full Text | Google Scholar

9. Meng ZW, Pan W, Hong HJ, Chen JZ, Chen YL. Macroscopic types of intrahepatic cholangiocarcinoma and the eighth edition of AJCC/UICC TNM staging system. Oncotarget. (2017) 8:101165–74. doi: 10.18632/oncotarget.20932

PubMed Abstract | CrossRef Full Text | Google Scholar

10. Amin MB, Greene FL, Edge SB, Compton CC, Gershenwald JE, Brookland RK, et al. The eighth edition AJCC cancer staging manual: continuing to build a bridge from a population-based to a more “personalized” approach to cancer staging. CA Cancer J Clin. (2017) 67:93–9. doi: 10.3322/caac.21388

PubMed Abstract | CrossRef Full Text | Google Scholar

11. Lee AJ, Chun YS. Intrahepatic cholangiocarcinoma: the AJCC/UICC 8th edition updates. Chin Clin Oncol. (2018) 7:52. doi: 10.21037/cco.2018.07.03

PubMed Abstract | CrossRef Full Text | Google Scholar

12. Endo I, Gonen M, Yopp AC, Dalal KM, Zhou Q, Klimstra D, et al. Intrahepatic cholangiocarcinoma: rising frequency, improved survival, and determinants of outcome after resection. Ann Surg. (2008) 248:84–96. doi: 10.1097/SLA.0b013e318176c4d3

PubMed Abstract | CrossRef Full Text | Google Scholar

13. He C, Zhang Y, Cai Z, Duan F, Lin X, Li S. Nomogram to predict cancer-specific survival in patients with pancreatic acinar cell carcinoma: a competing risk analysis. J Cancer. (2018) 9:4117–27. doi: 10.7150/jca.26936

PubMed Abstract | CrossRef Full Text | Google Scholar

14. Chinchilla-López P, Aguilar-Olivos NE, García-Gómez J, et al. Prevalence, Risk Factors, and Survival of Patients with Intrahepatic Cholangiocarcinoma. Ann Hepatol. (2017) 16:565–8. doi: 10.5604/01.3001.0010.0293

PubMed Abstract | CrossRef Full Text | Google Scholar

15. Chun YS, Javle M. Systemic and adjuvant therapies for intrahepatic cholangiocarcinoma. Cancer Control. (2017) 24:1073274817729241. doi: 10.1177/1073274817729241

PubMed Abstract | CrossRef Full Text | Google Scholar

16. Kim Y, Moris DP, Zhang XF, Bagante F, Spolverato G, Schmidt C, et al. Evaluation of the 8th edition American Joint Commission on Cancer (AJCC) staging system for patients with intrahepatic cholangiocarcinoma: a surveillance, epidemiology, and end results (SEER) analysis. J Surg Oncol. (2017) 116:643–50. doi: 10.1002/jso.24720

PubMed Abstract | CrossRef Full Text | Google Scholar

17. Wang Y, Li J, Xia Y, Gong R, Wang K, Yan Z, et al. Prognostic nomogram for intrahepatic cholangiocarcinoma after partial hepatectomy. J Clin Oncol. (2013) 31:1188–95. doi: 10.1200/JCO.2012.41.5984

PubMed Abstract | CrossRef Full Text | Google Scholar

18. Yeh CN, Wang SY, Chen YY, Chen MH, Chiang KC, Cheng CT, et al. A prognostic nomogram for overall survival of patients after hepatectomy for intrahepatic cholangiocarcinoma. Anticancer Res. (2016) 36:4249–58.

PubMed Abstract | Google Scholar

19. Squires MH, Cloyd JM, Dillhoff M, Schmidt C, Pawlik TM. Challenges of surgical management of intrahepatic cholangiocarcinoma. Expert Rev Gastroenterol Hepatol. (2018) 12:671–81. doi: 10.1080/17474124.2018.1489229

PubMed Abstract | CrossRef Full Text | Google Scholar

20. Ruzzenente A, Conci S, Valdegamberi A, Pedrazzani C, Guglielmi A. Role of surgery in the treatment of intrahepatic cholangiocarcinoma. Eur Rev Med Pharmacol Sci. (2015) 19:2892–900.

PubMed Abstract | Google Scholar

21. Poultsides GA, Zhu AX, Choti MA, Pawlik TM. Intrahepatic cholangiocarcinoma. Surg Clin North Am. (2010) 90:817–37. doi: 10.1016/j.suc.2010.04.011

CrossRef Full Text | Google Scholar

22. Bagante F, Spolverato G, Merath K, Weiss M, Alexandrescu S, Marques HP, et al. Intrahepatic cholangiocarcinoma tumor burden: a classification and regression tree model to define prognostic groups after resection. Surgery. (2019) 166:983–90. doi: 10.1016/j.surg.2019.06.005

PubMed Abstract | CrossRef Full Text | Google Scholar

23. Thelen A, Scholz A, Weichert W, Wiedenmann B, Neuhaus P, Gessner R, et al. Tumor-associated angiogenesis and lymphangiogenesis correlate with progression of intrahepatic cholangiocarcinoma. Am J Gastroenterol. (2010) 105:1123–32. doi: 10.1038/ajg.2009.674

PubMed Abstract | CrossRef Full Text | Google Scholar

24. Choi SB, Kim KS, Choi JY, Park SW, Choi JS, Lee WJ, et al. The prognosis and survival outcome of intrahepatic cholangiocarcinoma following surgical resection: association of lymph node metastasis and lymph node dissection with survival. Ann Surg Oncol. (2009) 16:3048–56. doi: 10.1245/s10434-009-0631-1

PubMed Abstract | CrossRef Full Text | Google Scholar

25. Zhou R, Lu D, Li W, Tan W, Zhu S, Chen X, et al. Is lymph node dissection necessary for resectable intrahepatic cholangiocarcinoma? A systematic review and meta-analysis. HPB (Oxford). (2019) 21:784–92. doi: 10.1016/j.hpb.2018.12.011

PubMed Abstract | CrossRef Full Text | Google Scholar

26. Huitzil-Melendez FD, Capanu M, O'Reilly EM, Duffy A, Gansukh B, Saltz LL, et al. Advanced hepatocellular carcinoma: which staging systems best predict prognosis?. J Clin Oncol. (2010) 28:2889–95. doi: 10.1200/JCO.2009.25.9895

PubMed Abstract | CrossRef Full Text | Google Scholar

27. Balachandran VP, Gonen M, Smith JJ, DeMatteo RP. Nomograms in oncology: more than meets the eye. Lancet Oncol. (2015) 16:e173–80. doi: 10.1016/S1470-2045(14)71116-7

PubMed Abstract | CrossRef Full Text | Google Scholar

28. Vickers AJ, Cronin AM, Elkin EB, Gonen M. Extensions to decision curve analysis, a novel method for evaluating diagnostic tests, prediction models and molecular markers. BMC Med Inform Decis Mak. (2008) 8:53. doi: 10.1186/1472-6947-8-53

PubMed Abstract | CrossRef Full Text | Google Scholar

29. Talluri R, Shete S. Using the weighted area under the net benefit curve for decision curve analysis. BMC Med Inform Decis Mak. (2016) 16:94. doi: 10.1186/s12911-016-0336-x

PubMed Abstract | CrossRef Full Text | Google Scholar

30. Rousson V, Zumbrunn T. Decision curve analysis revisited: overall net benefit, relationships to ROC curve analysis, and application to case-control studies. BMC Med Inform Decis Mak. (2011) 11:45. doi: 10.1186/1472-6947-11-45

PubMed Abstract | CrossRef Full Text | Google Scholar