Prognostic Nomogram Based on the Metastatic Lymph Node Ratio for T1-4N0-1M0 Pancreatic Neuroendocrine Tumors After Surgery

Purpose: This study aimed to investigate the prognostic significance of the metastatic lymph node ratio (LNR) in patients with pancreatic neuroendocrine tumors (pNETs) and to develop and validate nomograms to predict 5-, 7-, and 10-year overall survival (OS) and cancer-specific survival (CSS) rates for pNETs after surgical resection.

Methods: The demographics and clinicopathological information of T_1-4N_0-1M₀ pNET patients between 2004 and 2018 were extracted from the Surveillance, Epidemiology and End Results database. X-tile software was used to determine the best cutoff value for the LNR. Patients were randomly divided into the training and the validation groups. A Cox regression model was used in the training group to obtain independent prognostic factors to develop nomograms for predicting OS and CSS. The concordance index (C-index), calibration curves, area under the receiver operating characteristic curve (AUC) and decision curve analysis (DCA) were used to assess the nomograms. Patients were divided into four groups according to the model scores, and their survival curves were generated by the Kaplan–Meier method.

Results: A total of 806 patients were included in this study. The best cutoff value for the LNR was 0.16. The LNR was negatively correlated with both OS and CSS. Age, sex, marital status, primary site, grade, the LNR and radiotherapy were used to construct OS and CSS nomograms. In the training group, the C-index was 0.771 for OS and 0.778 for CSS. In the validation group, the C-index was 0.737 for OS and 0.727 for CSS. The calibration curves and AUC also indicated their good predictability. DCA demonstrated that the nomograms displayed better performance than the American Joint Committee on Cancer (AJCC) TNM staging system (8th edition). Risk stratification indicated that patients with higher risk had a worse prognosis.

Conclusions: The LNR is an independent negative prognostic factor for pNETs. The nomograms we built can accurately predict long-term survival for pNETs after surgery.

Introduction

Pancreatic neuroendocrine tumors (pNETs) are relatively rare tumors that originate from the pancreatic neuroendocrine system (1). According to the US epidemiology survey, its incidence is less than 1 in 100,000 people, representing nearly 10% of all pancreatic tumors (2, 3). Owing to the advancement of imaging and endoscopic techniques, the detection rate of pNETs has gradually increased in recent years (4). PNETs are classified as functional and nonfunctional types based on whether they secrete hormones associated with a clinical syndrome, while nonfunctional pNETs account for most of them (4, 5). One of the characteristics of pNETs is heterogeneity. Generally, they exhibit indolent clinical features; however, they may become invasive and transform rapidly in some circumstances (6).

Surgical resection is an effective treatment for pNETs without metastases (7). One SEER database study demonstrated that surgery might greatly improve survival compared with nonsurgery interventions (114 months vs. 35 months) (8). Other studies reported that the 5-year survival rate after surgical resection of pNETs was approximately 80% (6, 9). However, due to the rarity of this disease, studies on prediction models for pNETs after surgery are deficient.

Recently, the metastatic lymph node ratio (LNR), defined as the proportion of positive nodes to total examined nodes, has been an important prognostic factor for many tumors. You et al. (10) pointed out that the LNR might predict the prognosis of patients with pancreatic cancer. Zhang et al. (11) demonstrated that the LNR was a strong negative prognostic factor for patients with colorectal cancer. Two other studies also indicated that it was negatively related to the survival of patients with gastric and small intestinal neuroendocrine tumors (12, 13). However, there are no reports of nomograms of long-term survival after resection of pNETs that incorporate LNR data.

Therefore, based on the Surveillance, Epidemiology, and End Results (SEER) database, the present study attempted to explore the correlation between the LNR and the prognosis of pNET patients and to construct nomograms to predict 5-, 7-, and 10-year overall survival (OS) and cancer-specific survival (CSS) rates for pNETs after surgery.

Materials and Methods

Data Collection

The SEER database is an authoritative cancer statistics database in the United States, covering approximately 30% of the US population (9). Data were extracted from the SEER database with SEER*Stat Software (version 8.3.9.2), and the Incidence SEER Research Plus Data, 9 Registries, Nov 2020 Sub (1975-2018) dataset was selected for analysis (username for login: 15881-Nov2020). Patients diagnosed with pNETs from 2004 to 2018 were identified retrospectively. The corresponding selection formula in the software was as follows: Site and Morphology, “Site recode ICD-O-3/WHO 2008” (Pancreas) and “ICD-O-3 Hist/behav” (8013/3, 8150/3-8156/3, 8240/2, 8240/3-8246/3, 8246/2, 8249/3). The following variables were extracted from the database: patient ID, age, sex, race, marital status, year of diagnosis, primary site, histologic type, grade, diagnostic confirmation, tumor size, T stage, N stage, M stage, American Joint Committee on Cancer (AJCC) TNM staging system, surgery at the primary site, the scope of regional lymph node surgery, radiation recode, chemotherapy recode, regional nodes examined, regional nodes positive, survival months, vital status recode, SEER cause-specific death classification, first malignant primary indicator and sequence number.

Data Processing

Patients conforming to any of the following criteria were excluded: 1) surgery was not performed; 2) no regional nodes examined or the number of regional lymph nodes removed was unknown; 3) the AJCC TNM staging system belonged to M1; 4) not the first malignant tumor or multiple primary tumors; and 5) missing or unknown clinical information.

Age was regarded as a continuous variable, and other factors were treated as categorical variables. Patients who were widowed, divorced, single or separated were considered unmarried. (C25.4, islets of Langerhans), (C25.7, other specified parts of the pancreas), (C25.8, overlapping lesion of the pancreas) and (C25.9, pancreas, NOS) in the primary site were considered other. The staging system was adjusted according to the 8th edition AJCC. OS was defined as survival time until death by any cause or last follow-up, and CSS was defined as survival time until death due to pNETs (9).

Construction and Validation of the Nomograms

The overall patients were randomly divided into a training cohort and a validation cohort at a ratio of 7:3. For the training group, univariate and multivariate Cox regression analyses were used to screen out significant variables. After that, all the independent prognostic factors in multivariate analysis were used to construct nomograms using the package “rms” in R software (14). The validation cohort was used to perform an external validation of the nomograms.

To assess the accuracy of the nomograms, we calculated Harrell’s concordance index (C-index) and drew calibration curves of the training and validation groups for internal and external validation (15). The area under the receiver operating characteristic curve (AUC) was also used to evaluate the predictive value (15). In addition, we used decision curve analysis (DCA) to determine the 5-, 7-, and 10-year survival rates of the two groups (16). Furthermore, DCA was applied to compare the AJCC staging system (8th edition), the LNR and the nomograms.

Statistical Analysis

All statistical tests were performed using R statistical software (version 4.1.2, https://www.r-project.org, Vienna, Austria). Categorical variables were expressed as frequencies and percentages. Continuous variables were presented as the mean with standard deviation. The optimal cutoff value for the LNR was determined by the X-tile program (17). Survival curves were generated by the Kaplan–Meier method and analyzed by the log-rank test. Variables with P < 0.2 in univariate analysis were subjected to multivariate analysis. P < 0.1 in multivariate analysis was considered clinically significant. Pearson’s correlation was performed to detect collinearity among the variables. A correlation coefficient < 0.7 between two variables indicated no multicollinearity (18). Tolerance and variance inflation factor (VIF) were also used to evaluate multicollinearity between variables, with tolerance < 0.1 and VIF > 10 indicating multicollinearity (19). P < 0.05 was considered statistically significant.

Results

Patient Characteristics

A total of 806 patients with T_1-4N_0-1M₀ pNETs were enrolled in this study, of which 564 cases were randomly assigned to the training group, while 242 cases were assigned to the validation group (Figure 1). The clinicopathological characteristics of patients in the 2 cohorts are summarized in Table 1. In the training group, 75 patients (13.3%) died from pNETs, and 23 (4.1%) patients died from other causes, while in the validation group, the numbers were 27 (11.2%) and 5 (2.1%), respectively. The optimal cutoff value for the LNR was 0.16. Therefore, patients were divided into three groups (LNR1: 0, LNR2: ≤ 0.16 and LNR3: > 0.16). Patients in the high LNR group had poorer OS and CSS than those in the relatively low LNR group (Figures 2A, B).

FIGURE 1

Figure 1 This is the flow chart of enrolled patients.

TABLE 1

Table 1 Patient demographics and pathological characteristics.

FIGURE 2

Figure 2 Kaplan–Meier curves of OS (A) and CSS (B) for patients with different LNRs in the overall dataset. OS, overall survival; CSS, cancer-specific survival; LNR, lymph node ratio.

Variable Screening and Nomogram Construction

For the training cohort, the univariate analysis for OS revealed that age, sex, marital status, primary site, grade, tumor size, T stage, LNR, radiotherapy and chemotherapy were significantly associated with survival. Additionally, age, sex, marital status, primary site, grade, tumor size, T stage, LNR, radiotherapy, chemotherapy and lymph node dissection were significantly associated with CSS. The multivariate Cox regression analyses revealed that age, sex, marital status, primary site, grade, LNR and radiotherapy were significant for both OS and CSS (Tables 2, 3). Therefore, these factors were included in the construction of the nomograms. Every variable was given a score in these two nomograms. Users could obtain the total score based on the individual scores of those factors and estimate the probability of survival for 5, 7 and 10 years (Figures 3A, B).

TABLE 2

Table 2 Variables associated with overall survival (OS) according to the Cox proportional hazards regression model in the training cohort.

TABLE 3

Table 3 Variables associated with cancer-specific survival (CSS) according to the Cox proportional hazards regression model in the training cohort.

FIGURE 3

Figure 3 (A) Nomogram used to predict the 5-, 7- and 10-year OS rates of patients with T_1-4N_0-1M₀ pNETs after surgery. (B) Nomogram used to predict the 5-, 7- and 10-year CSS rates of patients with T_1-4N_0-1M₀ pNETs after surgery. OS, overall survival; pNETs, pancreatic neuroendocrine tumors; CSS, cancer-specific survival.

There was no significant correlation between the screened variables for the overall dataset, the training or the validation groups (Figure 4). Furthermore, the tolerance was >1, and the VIF was <10 for the three groups, proving no collinearity among the factors.

FIGURE 4

Figure 4 Correlations between variables in the overall dataset (A), the training group (B) and the validation group (C).

Nomogram Validation

The two nomograms were validated both internally and externally. In the internal validation, the C-index was 0.771 for OS and 0.778 for CSS. In the external validation, the C-index was 0.737 for OS and 0.727 for CSS. The calibration plots were close to the standard curves for OS (Figure 5A) and CSS (Figure 6A) in the training group and for OS (Figure 5E) and CSS (Figure 6E) in the validation group. The AUC values for predicting 5-, 7-, and 10-year OS rates were 0.804, 0.797 and 0.799 in the training group (Figure 7A) and 0.777, 0.770 and 0.822 in the validation group, respectively (Figure 7B). For the 5-, 7-, and 10-year CSS rates, the AUC values were 0.799, 0.790 and 0.786 in the training group (Figure 7C) and 0.740, 0.644 and 0.687 in the validation group (Figure 7D). For both the training and validation groups, the DCA curves for OS (Figures 5B–D, F–H) and CSS (Figures 6B–D, F–H) indicated that our nomograms displayed better performance than the AJCC staging system (8th edition), and the LNR was an important factor for predicting patient prognosis.

FIGURE 5

Figure 5 Calibration curve of the nomogram for OS prediction from the training group (A) and the validation group (E). Decision curve analysis of the AJCC 8th edition staging system, nomogram and the LNR for the 5- (B), 7- (C) and 10-year (D) OS rates of patients with T_1-4N_0-1M₀ pNETs from the training group. Decision curve analysis of the AJCC 8th edition staging system, nomogram and the LNR for the 5- (F), 7- (G) and 10-year (H) OS rates of patients with T_1-4N_0-1M₀ pNETs from the validation group. OS, overall survival; LNR, lymph node ratio; pNETs, pancreatic neuroendocrine tumors. For calibration curves, red, blue and green lines represent 5, 7, and 10 years, respectively.

FIGURE 6

Figure 6 Calibration curve of the nomogram for CSS prediction from the training group (A) and the validation group (E). Decision curve analysis of the AJCC 8th edition staging system, nomogram and the LNR for the 5- (B), 7- (C) and 10-year (D) CSS rates of patients with T_1-4N_0-1M₀ pNETs from the training group. Decision curve analysis of the AJCC 8th edition staging system, nomogram and the LNR for the 5- (F), 7- (G) and 10-year (H) CSS rates of patients with T_1-4N_0-1M₀ pNETs from the validation group. CSS, cancer-specific survival; LNR, lymph node ratio; pNETs, pancreatic neuroendocrine tumors. For calibration curves, red, blue and green lines represent 5, 7, and 10 years, respectively.

FIGURE 7

Figure 7 Receiver operating characteristic curves (ROCs) of the nomogram for OS prediction (A, training group; B, validation group) and CSS prediction (C, training group; D, validation group). OS, overall survival; CSS, cancer-specific survival; AUC, area under the receiver operating characteristic curve. For ROCs, red represents 5 years, blue represents 7 years and yellow represents 10 years.

Risk Stratification

To determine the performance of the established nomograms in stratifying the risk of pNETs patients, we divided them into four groups according to the scores for OS (Min-67.9, 67.9-83.1, 83.1-100.2, 100.2-Max) and the scores for CSS (Min-91.7, 91.7-117.2, 117.2-150.6, 150.6-Max). Statistically significant differences in OS (Figures 8A, B) and CSS (Figures 8C, D) were found among the four groups in both the training and validation groups.

FIGURE 8

Figure 8 Kaplan–Meier curves of risk stratification within the training group for OS (A) and CSS (C) and within the validation group for OS (B) and CSS (D). OS, overall survival; CSS, cancer-specific survival.

Discussion

PNETs are heterogeneous tumors with different prognoses. Therefore, predicting the outcomes of these patients is complicated. In this study, we screened data from the SEER database and studied the factors affecting the long-term survival of T_1-4N_0-1M₀ pNET patients after surgery. After that, we constructed nomograms to predict the 5-, 7-, and 10-year OS rates and CSS rates for these patients. Considering the importance of the LNR, we included it in the nomograms. The relatively high C-indices and values of AUC in the training and validation cohorts proved the good clinical predictive ability of the nomograms. The calibration plots also demonstrated their reasonable prediction probability.

The LNR has been increasingly recognized as a strong predictor of survival for many neoplasms. Teng et al. (20) noted that the LNR was a significant negative predictive variable for the OS of breast cancer patients. Tol et al. (21) reported that the LNR was an important predictor of poor survival in ampullary cancer. Using SEER data regarding pNETs, Liu et al. (22) suggested that the LNR, but not the total number of examined lymph nodes or lymph node metastasis, was an adverse prognostic factor for OS. Heidsma et al. (23) pointed out lymph node status (yes/no presence of positive node) was an independent factor to predict 5-year recurrence after resection of grade 1 and 2 nonfunctional pNETs. Ricci et al. (24) indicated that the LNR > 0.07 was associated with lower recurrence-free survival (RFS), whereas Boninsegna et al. (25) identified an LNR > 0.2 as an independent indicator of poor RFS. Another study showed that an LNR ≥ 0.5 was associated with worse CSS and that the LNR-based staging system was better than the 8th edition AJCC N staging (26). Similar to previous reports, this study showed that the LNR was a poor independent prognostic factor for both OS and CSS. Moreover, the LNR, being a ratio, eliminates the variability in lymph node sampling during surgery and is more suitable for evaluation of prognosis.

In addition to LNR, several other variables were included in the established nomograms, such as age, sex, marital status, primary site, grade and radiotherapy. Several studies have indicated that age significantly impacts the survival of pNET patients (9, 27, 28). A similar negative effect was observed in this study. There are two possible reasons for this. First, tumor resistance in elderly patients is poorer than that in young patients due to physical aging; thus, the CSS of these patients is poor (14). Second, there are far more age-related comorbid conditions in elderly patients than in young patients; therefore, the OS of elderly patients is poor. Female patients had more favorable prognoses than male patients, which was in agreement with the conclusion of Miao et al. (29). Marital status was also shown to affect both OS and CSS, consistent with previous reports (29, 30). This is because married patients usually have better psychological mentation and socioeconomic status, which may indicate a better prognosis. Similar to a previous study, the primary site of the tumor was associated with survival (29). Furthermore, histologic grade, which is an inherent characteristic of tumors, was a critical factor in predicting the prognosis of pNET patients in this study. Similar results were found in previous reports (9, 31, 32). Interestingly, this study showed that radiotherapy had some impact on survival. Iwata et al. (33) pointed out radiotherapy was an effective treatment for local disease control. However, the number of patients in that research was too small. The effect of radiotherapy on pNETs requires large sample studies.

For the first time, we developed and validated nomograms to predict 5-, 7-, and 10-year OS and CSS for T_1-4N_0-1M₀ pNET patients after surgery. The nomograms included not only clinicopathological factors but also demographic factors and were more precise than the conventional staging system. Furthermore, we compared the nomograms with the AJCC TNM staging system (8th edition). DCA curves proved that our nomograms prediction ability was better than the AJCC TNM staging system. In addition, the nomograms were composed of variables that are readily available in clinical practice. With this easily used system, doctors could evaluate the risk factors for patients more precisely. Risk stratification demonstrated good applicability for patients in different stages. Therefore, patients with high risks of poor prognosis may benefit from adjuvant therapy. These may promote more specialized individualized treatment for this heterogeneous neoplasm.

There were some limitations in this study. First, it was a retrospective study based on the SEER database, and this may lead to the risk of potential selection bias. Second, several important clinicopathological factors, such as carbohydrate antigen 19-9 (CA19-9), Ki-67 index, surgical margin status, chemotherapy regimens and radiation technology, were not included due to the limitation of the SEER database, which might affect the results.

In conclusion, the present study identified that the LNR is an independent prognostic factor for pNETs. We established nomograms based on the SEER database to predict long-term survival for pNETs after surgery. These nomograms could help clinicians make tailored treatment plans for patients.

Data Availability Statement

The datasets presented in this study can be found in online repositories. The names of the repository/repositories and accession number(s) can be found in the article/Supplementary Material.

Author Contributions

JS, JZ and YW: study design and revision of the manuscript. JS: performed the literature search and wrote the manuscript. JC and SS: data collection. SL and CR: statistical analysis. JZ and YW: study supervision. All authors read and approved the final manuscript.

Funding

This work was supported by the Tianjin Health Bureau under Grant (No. 2014KR04) and the Key Research Project of Tianjin Health and Family Planning Commission (No. 15KG114).

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Acknowledgments

The authors sincerely thank the Surveillance, Epidemiology, and End Results (SEER) program for providing high-quality open resources for researchers.

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fonc.2022.899759/full#supplementary-material.

References

1. Halfdanarson TR, Rabe KG, Rubin J, Petersen GM. Pancreatic Neuroendocrine Tumors (PNETs): Incidence, Prognosis and Recent Trend Toward Improved Survival. Ann Oncol (2008) 19:1727–33. doi: 10.1093/annonc/mdn351

PubMed Abstract | CrossRef Full Text | Google Scholar

2. Vaghaiwalla T, Keutgen XM. Surgical Management of Pancreatic Neuroendocrine Tumors. Surg Oncol Clin N Am (2020) 29:243–52. doi: 10.1016/j.soc.2019.11.008

PubMed Abstract | CrossRef Full Text | Google Scholar

3. Dasari A, Shen C, Halperin D, Zhao B, Zhou S, Xu Y, et al. Trends in the Incidence, Prevalence, and Survival Outcomes in Patients With Neuroendocrine Tumors in the United States. JAMA Oncol (2017) 3:1335–42. doi: 10.1001/jamaoncol.2017.0589

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Zhang Z, Liu M, Ji S, Luo G, Xu W, Liu W, et al. Prognostic Value and Clinical Predictors of Lymph Node Metastases in Pancreatic Neuroendocrine Tumors. Pancreas (2020) 49:381–6. doi: 10.1097/MPA.0000000000001493

PubMed Abstract | CrossRef Full Text | Google Scholar

5. Fathi AH, Romanyshyn J, Barati M, Choudhury U, Chen A, Sosa JA. Predicting Aggressive Behavior in Nonfunctional Pancreatic Neuroendocrine Tumors With Emphasis on Tumor Size Significance and Survival Trends: A Population-Based Analysis of 1787 Patients. Am Surg (2020) 86:458–66. doi: 10.1177/0003134820919730

PubMed Abstract | CrossRef Full Text | Google Scholar

6. Tanaka M, Heckler M, Mihaljevic AL, Probst P, Klaiber U, Heger U, et al. Systematic Review and Metaanalysis of Lymph Node Metastases of Resected Pancreatic Neuroendocrine Tumors. Ann Surg Oncol (2021) 28:1614–24. doi: 10.1245/s10434-020-08850-7

PubMed Abstract | CrossRef Full Text | Google Scholar

7. Rosenblum RE, Harris CK, Baeg KJ, Starr JA, Brais LK, Stashek KM, et al. Predictors of Recurrence and Survival in Patients With Surgically Resected Pancreatic Neuroendocrine Tumors. Pancreas (2020) 49:249–54. doi: 10.1097/MPA.0000000000001477

PubMed Abstract | CrossRef Full Text | Google Scholar

8. Hill JS, McPhee JT, McDade TP, Zhou Z, Sullivan ME, Whalen GF, et al. Pancreatic Neuroendocrine Tumors: The Impact of Surgical Resection on Survival. Cancer (2009) 115:741–51. doi: 10.1002/cncr.24065

PubMed Abstract | CrossRef Full Text | Google Scholar

9. Song Z, Wang S, Wu Y, Zhang J, Liu S. Prognostic Nomograms to Predict Overall Survival and Cancer-Specific Survival of Patients With Pancreatic Neuroendocrine Tumors: A Population-Based Study. Pancreas (2021) 50:414–22. doi: 10.1097/MPA.0000000000001779

PubMed Abstract | CrossRef Full Text | Google Scholar

10. You MS, Lee SH, Choi YH, Shin BS, Paik WH, Ryu JK, et al. Lymph Node Ratio as Valuable Predictor in Pancreatic Cancer Treated With R0 Resection and Adjuvant Treatment. BMC Cancer (2019) 19:952. doi: 10.1186/s12885-019-6193-0

PubMed Abstract | CrossRef Full Text | Google Scholar

11. Zhang CH, Li YY, Zhang QW, Biondi A, Fico V, Persiani R, et al. The Prognostic Impact of the Metastatic Lymph Nodes Ratio in Colorectal Cancer. Front Oncol (2018) 8:628. doi: 10.3389/fonc.2018.00628

PubMed Abstract | CrossRef Full Text | Google Scholar

12. Li J, Lin Y, Wang Y, Lin H, Lin F, Zhuang Q, et al. Prognostic Nomogram Based on the Metastatic Lymph Node Ratio for Gastric Neuroendocrine Tumour: SEER Database Analysis. ESMO Open (2020) 5:e000632. doi: 10.1136/esmoopen-2019-000632

PubMed Abstract | CrossRef Full Text | Google Scholar

13. Wu L, Chen F, Chen S, Wang L. The Lymph Node Ratio Optimizes Staging in Patients With Small Intestinal Neuroendocrine Tumors. Neuroendocrinology (2018) 107:209–17. doi: 10.1159/000491017

PubMed Abstract | CrossRef Full Text | Google Scholar

14. Liao Y, Yin G, Fan X. The Positive Lymph Node Ratio Predicts Survival in T₁-₄n_1-3m₀ Non-Small Cell Lung Cancer: A Nomogram Using the SEER Database. Front Oncol (2020) 10:1356. doi: 10.3389/fonc.2020.01356

PubMed Abstract | CrossRef Full Text | Google Scholar

15. Heagerty PJ, Zheng Y. Survival Model Predictive Accuracy and ROC Curves. Biometrics (2005) 61:92–105. doi: 10.1111/j.0006-341X.2005.030814.x

PubMed Abstract | CrossRef Full Text | Google Scholar

16. Fitzgerald M, Saville BR, Lewis RJ. Decision Curve Analysis. JAMA (2015) 313:409–10. doi: 10.1001/jama.2015.37

PubMed Abstract | CrossRef Full Text | Google Scholar

17. Camp RL, Dolled-Filhart M, Rimm DL. X-Tile: A New Bio-Informatics Tool for Biomarker Assessment and Outcome-Based Cut-Point Optimization. Clin Cancer Res (2004) 10:7252–9. doi: 10.1158/1078-0432.CCR-04-0713

PubMed Abstract | CrossRef Full Text | Google Scholar

18. Waldmann P. On the Use of the Pearson Correlation Coefficient for Model Evaluation in Genome-Wide Prediction. Front Genet (2019) 10:899. doi: 10.3389/fgene.2019.00899

PubMed Abstract | CrossRef Full Text | Google Scholar

19. Marcoulides KM, Raykov T. Evaluation of Variance Inflation Factors in Regression Models Using Latent Variable Modeling Methods. Educ Psychol Meas (2019) 79:874–82. doi: 10.1177/0013164418817803

PubMed Abstract | CrossRef Full Text | Google Scholar

20. Teng J, Abdygametova A, Du J, Ma B, Zhou R, Shyr Y, et al. Bayesian Inference of Lymph Node Ratio Estimation and Survival Prognosis for Breast Cancer Patients. IEEE J BioMed Health Inform (2020) 24:354–64. doi: 10.1109/JBHI.2019.2943401

PubMed Abstract | CrossRef Full Text | Google Scholar

21. Tol JA, Brosens LA, Van Dieren S, Van Gulik TM, Busch OR, Besselink MG, et al. Impact of Lymph Node Ratio on Survival in Patients With Pancreatic and Periampullary Cancer. Br J Surg (2015) 102:237–45. doi: 10.1002/bjs.9709

PubMed Abstract | CrossRef Full Text | Google Scholar

22. Liu P, Zhang X, Shang Y, Lu L, Cao F, Sun M, et al. Lymph Node Ratio, But Not the Total Number of Examined Lymph Nodes or Lymph Node Metastasis, is a Predictor of Overall Survival for Pancreatic Neuroendocrine Neoplasms After Surgical Resection. Oncotarget (2017) 8:89245–55. doi: 10.18632/oncotarget.19184

PubMed Abstract | CrossRef Full Text | Google Scholar

23. Heidsma CM, Van Roessel S, Van Dieren S, Engelsman AF, Strobel O, Buechler MW, et al. International Validation of a Nomogram to Predict Recurrence After Resection of Grade 1 and 2 Nonfunctioning Pancreatic Neuroendocrine Tumors. Neuroendocrinology (2021). doi: 10.1159/000518757

CrossRef Full Text | Google Scholar

24. Ricci C, Casadei R, Taffurelli G, Buscemi S, D’Ambra M, Monari F, et al. The Role of Lymph Node Ratio in Recurrence After Curative Surgery for Pancreatic Endocrine Tumours. Pancreatology (2013) 13:589–93. doi: 10.1016/j.pan.2013.09.001

PubMed Abstract | CrossRef Full Text | Google Scholar

25. Boninsegna L, Panzuto F, Partelli S, Capelli P, Delle Fave G, Bettini R, et al. Malignant Pancreatic Neuroendocrine Tumour: Lymph Node Ratio and Ki67 are Predictors of Recurrence After Curative Resections. Eur J Cancer (2012) 48:1608–15. doi: 10.1016/j.ejca.2011.10.030

PubMed Abstract | CrossRef Full Text | Google Scholar

26. Gaitanidis A, Patel D, Nilubol N, Tirosh A, Kebebew E. A Lymph Node Ratio-Based Staging Model Is Superior to the Current Staging System for Pancreatic Neuroendocrine Tumors. J Clin Endocrinol Metab (2018) 103:187–95. doi: 10.1210/jc.2017-01791

PubMed Abstract | CrossRef Full Text | Google Scholar

27. Liao T, Su T, Huang L, Li B, Feng LH. Development and Validation of a Novel Nomogram for Predicting Survival Rate in Pancreatic Neuroendocrine Neoplasms. Scand J Gastroenterol (2022) 57:85–90. doi: 10.1080/00365521.2021.1984571

PubMed Abstract | CrossRef Full Text | Google Scholar

28. Strosberg JR, Cheema A, Weber J, Han G, Coppola D, Kvols LK. Prognostic Validity of a Novel American Joint Committee on Cancer Staging Classification for Pancreatic Neuroendocrine Tumors. J Clin Oncol (2011) 29:3044–9. doi: 10.1200/JCO.2011.35.1817

PubMed Abstract | CrossRef Full Text | Google Scholar

29. Miao DL, Song W, Qian J, Zhu ZG, Wu Q, Lv CG, et al. Development and Validation of a Nomogram for Predicting Overall Survival in Pancreatic Neuroendocrine Tumors. Transl Oncol (2018) 11:1097–103. doi: 10.1016/j.tranon.2018.06.012

PubMed Abstract | CrossRef Full Text | Google Scholar

30. Zhou H, Zhang Y, Song Y, Tan W, Qiu Z, Li S, et al. Marital Status is an Independent Prognostic Factor for Pancreatic Neuroendocrine Tumors Patients: An Analysis of the Surveillance, Epidemiology, and End Results (SEER) Database. Clin Res Hepatol Gastroenterol (2017) 41:476–86. doi: 10.1016/j.clinre.2017.02.008

PubMed Abstract | CrossRef Full Text | Google Scholar

31. Zhang C, Wu Y, Zhuang H, Li D, Lin Y, Yin Z, et al. Establishment and Validation of an AJCC-Stage and Histologic Grade-Based Nomogram for Pancreatic Neuroendocrine Tumors After Surgical Resection. Cancer Manag Res (2019) 11:7345–52. doi: 10.2147/CMAR.S200340

PubMed Abstract | CrossRef Full Text | Google Scholar

32. Raj N, Valentino E, Capanu M, Tang LH, Basturk O, Untch BR, et al. Treatment Response and Outcomes of Grade 3 Pancreatic Neuroendocrine Neoplasms Based on Morphology: Well Differentiated Versus Poorly Differentiated. Pancreas (2017) 46:296–301. doi: 10.1097/MPA.0000000000000735

PubMed Abstract | CrossRef Full Text | Google Scholar

33. Iwata T, Ueno H, Itami J, Ito Y, Inaba K, Morizane C, et al. Efficacy of Radiotherapy for Primary Tumor in Patients With Unresectable Pancreatic Neuroendocrine Tumors. Jpn J Clin Oncol (2017) 47:826–31. doi: 10.1093/jjco/hyx081

PubMed Abstract | CrossRef Full Text | Google Scholar