Advanced Lung Cancer Inflammation Index Predicts Survival Outcomes of Patients With Oral Cavity Cancer Following Curative Surgery

Aim: The aim of our study was to investigate the prognostic value of preoperative advanced lung cancer inflammation index (ALI) and to establish prognostic nomograms for the prediction of survival outcomes in patients with oral cavity squamous cell carcinoma (OSCC).

Materials and Methods: A total of 372 patients who received primary curative surgery for OSCC during 2008–2017 at a tertiary referral center were enrolled. We used the receiver operating characteristic curve to determine the optimal cutoff point of ALI. Through a Cox proportional hazards model and Kaplan–Meier analysis, we elucidated the ALI–overall survival (OS) and ALI–disease-free survival (DFS) associations. Prognostic nomograms based on ALI and the results of multivariate analysis were created to predict the OS and DFS. We used the concordance indices (C-indices) and calibration plots to assess the discriminatory and predictive ability.

Results: The results revealed that the ALI cutoff was 33.6, and 105 and 267 patients had ALI values of <33.6 and ≥33.6, respectively. ALI < 33.6 significantly indicated lower OS (44.0% vs. 80.1%, p < 0.001) and DFS (33.6% vs. 62.8%; p < 0.001). In multivariate analysis, ALI < 33.6 was independently associated with poor OS and DFS (both p < 0.001). The C-indices of established nomograms were 0.773 and 0.674 for OS and DFS, respectively; moreover, the calibration plots revealed good consistency between nomogram-predicted and actual observed OS and DFS.

Conclusion: ALI is a promising prognostic biomarker in patients undergoing primary surgery for OSCC; moreover, ALI-based nomograms may be a useful prognostic tool for individualized OS and DFS estimations.

Introduction

Squamous cell carcinoma (SCC) that inherently affects the head and neck is ranked sixth among the most common types of cancer worldwide, with oral cavity SCC (OSCC) representing a majority of the cases (1). In Taiwan, OSCC incidence continues to increase even though betel quid use has declined recently; possible reasons for the mentioned increase are the longstanding carcinogenic effects engendered by betel quid use and the damaging influence induced by chronic inflammation, cigarette smoking, and alcohol consumption (2, 3). Currently, the main therapeutic modalities for OSCC include primary curative surgery followed by adjuvant therapy (if indicated) and the definitive chemoradiotherapy (4, 5). Despite the advances in diagnostic modalities and modern multidisciplinary treatments over the past two decades, no significant improvement has been noted in OSCC survival rates; moreover, some patients still experience treatment failure due to locoregional recurrence or distant metastasis (4, 6).

Systemic inflammation and malnutrition are responsible for cancer growth, tumorigenesis, and metastasis, and these can be assessed by routine laboratory examinations at the time of diagnosis (7, 8); therefore, various nutrition/inflammation-based biomarkers, such as C-reactive protein–albumin ratio (CAR), platelet-to-lymphocyte ratio (PLR), neutrophil-to-lymphocyte ratio (NLR), and albumin–globulin ratio (AGR), are used for early estimation of head and neck cancer (HNC) prognosis (9–12). In 2013, Jafri et al. first combined NLR, serum albumin level, and body mass index (BMI) into a unified advanced lung cancer inflammation index (ALI) and demonstrated its prognostic value in patients who had been diagnosed as having metastatic non–small-cell lung cancer (NSCLC) (13). Subsequently, ALI was found to be useful for predicting survival outcomes in various cancers (14–16), including large B-cell lymphoma (15), small-cell lung cancer (SCLC) (17), NSCLC (18), esophageal SCC (19), HNC (20), and colorectal cancer (CRC) (21). Jank et al. first reported that a low ALI was in association with poor survival of patients with HNC (20). However, they enrolled only 21 (22.6%) patients with OSCC and did not include some substantial prognostic factors of OSCC, such as depth of invasion (DOI) (22) and extranodal extension (ENE) (23), in their survival analysis.

We speculated that ALI could be a prognostic index for OSCC; however, to apply ALI as a prognostic biomarker for OSCC, a comprehensive evaluation is required owing to the lack of relevant robust evidence. Moreover, the use of a biomarker-based nomogram—in addition to the traditional, tumor-factor–based tumor–node–metastasis (TNM) staging system—may facilitate prognostic stratification and individualized treatment planning in patients with OSCC (24, 25). However, thus far, data have not indicated the prognostic value of nomograms based on preoperative ALI in predicting disease-free survival (DFS) as well as overall survival (OS) in patients with OSCC. Herein, we determined whether ALI is correlated with DFS as well as OS in such patients receiving primary surgery. Furthermore, for predicting individualized 3- and 5-year DFS and OS in patients with OSCC, we established nomograms integrating ALI and independent prognosticators determined through multivariate analysis.

Materials and Methods

Study Design and Patient Population

In our single-center observational cohort research, we retrospectively drew our study population from an OSCC cohort who underwent curative surgery as a first-line treatment followed by adjuvant therapy if indicated at Chang Gung Memorial Hospital’s Department of Otolaryngology—Head and Neck Surgery over the period spanning from January 2008 to December 2017. The data analysis was conducted from April to July 2020. We executed patient selection on the basis of the following inclusion criteria: (1) being aged > 18 years, (2) having received a pathological diagnosis of invasive OSCC, and (3) having undergone primary surgery for OSCC in our hospital. By contrast, the exclusion criteria were as follows: (1) being diagnosed as having inoperable cancer or having a condition that constituted a contraindication for surgery, (2) having undergone neoadjuvant treatment before curative surgery, (3) having second primary cancer or distant metastasis when OSCC was diagnosed, (4) having a history of malignancy or hematologic disease, (5) having blood test results and the clinical symptoms and signs indicated severe infection status, and (6) having missing preoperative or follow-up data. Of the 391 eligible patients, we excluded 19 with missing data on variable of interest. Finally, 372 patients were analyzed further.

As shown in Table 1, we recorded each patient’s baseline and outcome data, including gender; age at diagnosis; primary tumor location; personal habits; American Joint Committee on Cancer (AJCC) Staging Manual (Seventh Edition)–based cancer stage; DOI; cancer cell differentiation; perineural invasion (PNI) status; ENE; nearest surgical margin; presence and types of adjuvant therapy; and underlying comorbidities based on the Charlson comorbidity index (CCI) (26), laboratory test results, and survival duration through electronic patient charts review.

TABLE 1

Table 1 Baseline clinicopathological characteristics of patients with oral cavity squamous cell carcinoma (n = 372).

To explore the association between nutrition/inflammation indices and survival outcomes, the patients’ blood samples, height, and weight were routinely measured within a week preoperatively. We determined complete blood count and differential leukocyte count on an SE-9000 automated hematology analyzer (Sysmex, Kobe, Japan) and assayed serum albumin levels on a Cobas 8000 automated biochemistry analyzer (Roche Hitachi, Rotkreuz, Switzerland) according to the manufacturer’s instructions. BMI was derived using its established derivation formula: body weight (kg)/height squared (m²). The preoperative NLR was derived as follows: peripheral blood absolute neutrophil count divided by absolute lymphocyte count. In addition, ALI was derived using the following definition: [BMI (kg/m²) × serum albumin (g/dl)]/NLR (13).

Treatment

All patients received primary curative surgery: wide excision of the primary tumor with unilateral or bilateral neck dissection. Intraoperative frozen section for surgical margin control was applied, and the surgical defects were immediately reconstructed by a plastic surgeon by using a free, pedicled, or local flap. Postoperative adjuvant therapy was applied on the basis of our institutional guidelines: it was administered within 6 weeks postoperatively if indicated. In brief, patients diagnosed as having a pathologic T4 disease and one metastatic lymph node are provided adjuvant radiotherapy, whereas those diagnosed as having positive surgical margins, ENE, or multiple metastatic lymphadenopathies are administered adjuvant concurrent chemoradiotherapy. Here, 66 Gy was the total adjuvant radiation dose delivered to the tumor site, with 2 Gy being provided over 5 days per week. For concurrent chemotherapy, the chemotherapy regimen consisted of intravenous cisplatin 40 mg/m² weekly or 100 mg/m² triweekly, depending on the oncologist’s judgment and patient’s preference. More detailed adjuvant therapy guidelines in our institute have been reported by Lin et al. (27).

Follow-Up

The included study patients were all followed up at the outpatient clinic every 2, 3, and 6 months during the first, second, and third years after surgery, respectively. At every follow-up, the patients underwent complete physical examination, routine laboratory testing, and flexible endoscopy. Moreover, during the follow-up period, we executed magnetic resonance imaging or computed tomography at 6-month intervals during the first 2 years and annually thereafter. In this study, we defined OS as the period spanning from the date on which primary surgery was executed to the date on which the final follow-up was executed (i.e., December 31, 2019) or the date of censoring alive or death. We also defined DFS as the period spanning from the date on which primary surgery was executed to the date on which treatment failure (including locoregional recurrence, distant failure, censoring alive, or death) occurred or that on which the final follow-up was executed.

Statistical Analysis

The categorical variables are shown as a number and percentage of the total, and we used the Kolmogorov–Smirnov test to evaluate the normality of continuous variables, which are represented as mean ± SD if normally distributed. To derive each study variable’s cutoff value, we executed receiver operating characteristic (ROC) curve analysis to estimate the optimal area under the ROC curve (AUC), sensitivity, and specificity for all-cause mortality prediction. Next, we stratified our patients and their OSCC tumor characteristics according to various clinicopathological features, followed by evaluation through the Mann–Whitney test (for continuous variables that are not normally distributed), the chi-square test (for categorical variables), and correlation testing, as appropriate. Kaplan–Meier curves for DFS and OS curves were plotted subsequently, and the survival differences were assessed for statistical significance using the log-rank test. After testing the proportionality for using Cox proportional hazard assumption, we identified independent risk factors of OS and DFS through the Cox proportional hazards model. In the univariable analysis, the factors were assessed using the log-rank test; those that reached statistical significance (p < 0.1) were included in the multivariable analysis using the Cox proportional hazard regression. In addition to ALI (i.e., the variable of interest) and its components, we considered variables previously identified as covariates influencing OSCC prognosis for our Cox analyses. These factors included age, gender, tumor location, overall stage (AJCC 7th edition), cancer cell differentiation, ENE, PNI, DOI (≥10 or <10 mm), surgical margin (≥5 or <5 mm), CCI (0, 1, or ≥2), and chemotherapy (yes or no). All aforementioned statistical analyses were executed using SPSS (version 21.0; SPSS Inc., Chicago, IL, USA), except the test for proportionality assumption, which was conducted with SAS version 9.4 (SAS Institute Inc). We also set the statistical significance level in this study at a two-tailed p of < 0.05.

To examine the advantage of using ALI as prognostic marker in clinical practice, we established nomograms integrating preoperative ALI, with the endpoints being 3- and 5-year OS and DFS, by employing the “rms” package of R (version 5.1-0; Vanderbilt University, Nashville, TN, USA) (28). For evaluating the nomograms’ predictive accuracy regarding 3- and 5-year DFS and OS, we calculated the corresponding concordance indices (C-indices) (28, 29); here, C-index values of 0.5 and 1.0 were considered to indicate random predictability and perfect matching, respectively (30, 31). Finally, the consistency of survival rates between the nomogram-predicted and observed values was determined by creating calibration plots for the nomograms.

Results

Baseline Characteristics

In total, 372 patients [336 (90.3%) men; mean (SD) age, 58.7 (11.1) years] were enrolled. Table 1 lists the study cohort’s baseline demographic and clinicopathologic characteristics. The tongue was determined to constitute the most common primary tumor subsite (n = 143, 38.4%), followed by cheek mucosa (n = 120, 32.3%). A total of 207 patients (55.6%) had an advanced pathologic stage (stages III–IV) according to the AJCC staging system. History of cigarette smoking, alcohol consumption, and betel quid chewing were reported by 305 (82.1%), 248 (66.7%), and 292 (78.5%), respectively. In total, 188 (50.5%) patients underwent curative surgery alone, 135 (36.3%) underwent curative surgery followed by adjuvant chemoradiotherapy, and 49 (13.2%) underwent curative surgery with adjuvant radiotherapy. Comorbidities were recorded according to the CCI; 199 (53.5%), 144 (30.6%), and 59 (15.9%) patients had CCI scores of 0, 1, and ≥2 at the time of diagnosis, respectively. We determined the median follow-up duration for patients alive at the end of follow-up to be 58.5 (range: 2−126) months.

ROC Curves of ALI and Its Components

By analyzing the ROC curve, we determined 33.6 as the optimal ALI cutoff [AUC: 0.693, 95% confidence interval (CI): 0.631−0.755, p < 0.001, Figure 1]. On the basis of this ALI cutoff, we separated the patients into the following groups: high-ALI (≥33.6, n = 267) and low-ALI (<33.6, n = 105) groups. ROC curves for OS associated with ALI and its component factors, including serum albumin level, BMI, neutrophil and lymphocyte counts, and the NLR, were also generated. By analyzing the ROC curves, we determined the optimal cutoff value to be 23.1 for BMI (p = 0.009), 4.51 for NLR (p = 0.001), and 4.21 for albumin (p < 0.001). The AUC for each factor is illustrated in Table 2. Nearly all component factors (except for neutrophil count, p = 0.094) could predict poor OS, but the AUC for ALI was significantly higher than those for the serum albumin level, neutrophil and lymphocyte counts, BMI, and NLR (all p < 0.001).

FIGURE 1

Figure 1 ROC curve of ALI of patients with operable OSCC.

TABLE 2

Table 2 Comparison of the area under the curve values of the ALI and its components.

Clinicopathological Features Based on the Cutoff of ALI

The relationship of ALI with demographic and clinicopathological factors is presented in Table 3. Compared with the high-ALI group, the low-ALI group had higher proportions of patients with low BMI (p < 0.001), advanced overall stage (III−IV, p < 0.001), T3–T4 and N1–N3 classifications (both p < 0.001), PNI (p = 0.001), ENE (p = 0.001), DOI ≥ 10 mm (p < 0.001), adjuvant therapy requirement (p < 0.001), CCI ≥ 2 (p = 0.014), high NLR (p < 0.001), low serum albumin levels (p < 0.001), and short survival (p < 0.001). Nevertheless, no such significant differences were noted for gender (p = 0.951), age (p = 0.926), cell differentiation (p = 0.207), or surgical margin (p = 0.750).

TABLE 3

Table 3 Baseline clinicopathological characteristics according to the ALI.

Factors Associated With Poor OS and DFS

The low-ALI group had a shorter median OS period than the high-ALI group did (3.4 vs. 8.6 years). According to the Kaplan–Meier survival curve, the 5-year OS rates in the high- and low-ALI groups were 80.1% and 44.0%, respectively, signifying a remarkable difference, as revealed by the log-rank test (p < 0.001; Figure 2A). Moreover, the low-ALI group had a shorter median DFS period than did the high-ALI group (2.4 vs. 7.9 years). According to the Kaplan–Meier survival curve, the 5-year DFS rates in the high- and low-ALI groups were 62.8% and 33.6%, respectively, signifying a substantial difference, as demonstrated by the log-rank test (p < 0.001; Figure 2B).

FIGURE 2

Figure 2 Kaplan–Meier curves for (A) OS and (B) DFS of OSCC patients with ALI of ≥33.6 and <33.6.

Table 4 presents the results of the univariate analysis for each factor with regard to OS and DFS predictions. We noted significant correlations of OS and DFS with overall stage, ENE, cancer cell differentiation, DOI, chemotherapy, serum albumin level, NLR, and ALI. Because NLR and serum albumin level are component factors of ALI, we performed a separate multivariate analysis to prevent high collinearity; the results indicated advanced overall stage, the presence of ENE, poor cancer cell differentiation, serum albumin level of < 4.21, NLR of ≥ 4.51, and ALI of <33.6 to be independent prognostic factors for poor OS and DFS (Table 5).

TABLE 4

Table 4 Univariate analysis of poor prognostic factors for OS and DFS in OSCC patients.

TABLE 5

Table 5 Multivariate analysis of poor prognostic factors for OS and DFS in OSCC patients.

Subgroup Analysis for Discriminatory Ability of ALI

Figure 3 illustrates the subgroup analysis results: ALI was associated with OS with respect to different primary tumor sites [hazard ratio (HR): 3.13, 95% CI: 1.59–6.15, p = 0.001 for buccal cancer; HR: 4.21, 95% CI: 2.17–8.20, p < 0.001 for tongue cancer], early stage (I–II) disease (HR: 3.50, 95% CI: 1.29–9.49, p = 0.014), advanced-stage (III–IV) disease (HR: 2.49, 95% CI: 1.62–3.83, p < 0.001), early pT classification (HR: 3.33, 95% CI: 1.74–6.37, p < 0.001), late pT classification (HR: 2.42, 95% CI: 1.45–4.04, p = 0.001), and different pN classifications (HR: 2.99, 95% CI: 1.64–5.46, p < 0.001 for N0; HR: 3.34, 95% CI: 1.99–5.60, p < 0.001 for N1–N3).

FIGURE 3

Figure 3 HRs for ALI in subgroup analysis, stratified by primary tumor site, overall stage, and pT and pN classification.

Nomograms for Survival Prediction in OSCC

According to the multivariate analysis results, independent prognostic factors such as overall stage, ENE, cancer cell differentiation, and ALI were incorporated to establish the prognostic nomograms for the prediction of OS (Figure 4A) and DFS (Figure 5A) in patients with OSCC. The AUC of the nomograms was 0.81 (sensitivity: 70.3%, specificity: 76.1%) for OS prediction and 0.72 (sensitivity: 66.5%, specificity: 67.3%) for DFS prediction. The C-index (95% CI) of the nomograms was 0.773 (0.744–0.803) for OS and 0.674 (0.651–0.698) for DFS, suggesting the established nomograms to exhibit acceptable to good predictive accuracy and discrimination performance. We further used calibration plots to evaluate the consistency between the observed values and values predicted by the nomogram models. The calibration plots for the 3-year (Figure 4B) and 5-year (Figure 4C) OS rates predicted by the established nomograms were noted to be quite close to the 45° standard line, indicating that the nomograms had a good degree of calibration. Similarly, the calibration plots demonstrated that the 3- and 5-year DFS rates (Figures 5B, C, respectively) predicted by the nomograms were in good agreement with the actual observed values.

FIGURE 4

Figure 4 (A) Nomogram based on ALI and independent prognostic factors for OS prediction. Each parameter is included as a line segment on the nomogram, and the points on the line segment indicate the degree of risk contributed by this parameter. Addition of the points for all parameters yields the total points corresponding to the 3- and 5-year OS rates for the individual patient. (B, C) Calibration plots of the nomogram for (B) 3-year and (C) 5-year OS prediction. The light gray 45° line indicates the ideal prediction, and the blue line represents the value predicted by the nomogram model.

FIGURE 5

Figure 5 (A) Nomogram based on ALI and independent prognostic factors for DFS prediction. (B, C) Calibration plots of the nomogram for (B) 3-year and (C) 5-year DFS prediction. The predicted values were in good agreement with the observed values.

Discussion

The conventional AJCC and UICC staging systems, covering the tumor extent and status of neck nodal metastasis and distant metastasis, are currently the most widely used references for prognosis prediction and cancer treatment. Nevertheless, the AJCC staging system focuses only on tumor factors and lacks certain clinicopathological or demographic factors that potentially influence survival outcomes. Because the NLR reflects the systemic immunoinflammatory response (32) and because cachexia—a result of chronic systemic inflammation—is possibly reflected by BMI and serum albumin level (33), pretreatment ALI may reflect the underlying equilibrium of patient’s nutritional status and systemic inflammation and can predict survival outcomes for various malignant tumors (14–21) with superior discriminatory ability than the index based solely on inflammation (16, 34). In the present study, through a retrospective analysis of 372 patients’ clinicopathological features and survival outcomes, the prognostic ability of ALI in patients who underwent surgical treatment for OSCC was determined. According to the ROC curve analysis results, ALI demonstrated the highest AUC for BMI, albumin level, NLR, and neutrophil and lymphocyte counts, suggesting that ALI had a higher discriminatory ability for OS prediction than did its component facts because of its combination of body stature, host nutrition condition, and systemic inflammation patterns. Moreover, a low ALI (<33.6) was significantly associated with advanced-stage disease, advanced pT and pN classifications, PNI, ENE, need for adjuvant therapy, DOI >10 mm, and short survival period, indicating the importance of pretreatment screening for inflammatory status and malnutrition with regard to disease aggressiveness and prognosis. Our Kaplan–Meier analysis and log-rank test revealed that a low ALI was significantly associated with worse 5-year OS and DFS rates. Because ALI was calculated using BMI, the NLR, and albumin level, we conducted separate multivariate analyses to avoid the collinearity, and the results demonstrate that the association of advanced overall stage, presence of ENE, poor cell differentiation, low albumin, high NLR, and low ALI with poor DFS and OS was strong. These findings were consistent with previous studies, as the pretreatment serum albumin levels and NLR predicts survival outcomes in patients with HNC (35, 36).

Notably, the HRs of low ALI in predicting poor OS and DFS were 2.519 and 2.016, respectively, both of which were higher than that of albumin and NLR. In addition, our subgroup analysis revealed that the prognostic value of ALI for OS in patients with OSCC was consistent across different oral subsites, early and advanced-stage disease, and early and late T and N classifications. This observation not only highlights the general applicability of ALI in both metastatic lung cancer and OSCC but also suggests the consistent prognostic value of ALI in OSCC. In addition, we further integrated preoperative ALI and independent prognostic factors to derive prognostic nomograms for OS and DFS prediction in OSCC. The C-indices of the established nomograms were determined to be higher than that of the AJCC staging system–based nomogram model (0.773 vs. 0.699 for OS and 0.674 vs. 0.628 for DFS). After verifying the discriminatory ability, we created calibration plots for 3- and 5-year OS and DFS estimates and found that the nomograms had a good degree of calibration. Taken together, these findings suggest that preoperative ALI has potential value as a prognostic biomarker in patients with OSCC.

A low ALI has been revealed by previous studies to be associated with poor prognosis in various cancer types (14–16) because ALI integrates the factors essential in host immune, nutritional, and systemic inflammation statuses—namely, serum albumin level, BMI, and NLR. However, the definite mechanisms underlying the contribution of ALI to the survival outcomes of patients with OSCC have not been delineated yet. Thus far, serum albumin level, BMI, and NLR have been revealed to be significantly and independently associated with poorer survival outcomes in cancers, including OSCC. A meta-analysis of 25 studies including 6,847 patients with SCC of the head and neck concluded that a high pretreatment NLR strongly predicts poor progression-free survival, DFS, OS, and cancer-specific survival (12); a partial explanation for this may be that angiogenesis and extracellular matrix remodeling may potentially be promoted by cytokines and chemokines produced by neutrophils in order to provide a favorable microenvironment for cancer growth (37). Moreover, lymphocytes have a vital role in antitumor immunity; they destroy malignant cells by identifying tumor cell antigens (38). The relationship between malnutrition (reflected by BMI and serum albumin level) and cancer prognosis has also been extensively assessed for various cancers (39). Takenaka et al. indicated that a low pretreatment BMI is a prognostic factor for poor OS in patients with SCC of the head and neck who underwent definitive radiotherapy or chemoradiotherapy (40); this finding may be attributed to the cancer-related oxidative stress, chronic wasting, and relatively high protein metabolism caused by cancer cachexia (41).

Serum albumin may also serve as a surrogate for nutritional status. Lim et al. conducted a prospective cohort study on 338 patients with SCC of the head and neck who received definitive treatment and reported that pretreatment hypoalbuminemia constituted an independent risk factor for poor DFS, CSS, and OS, which may be explained by the fact that inflammation mediated by cancer may reduce serum albumin levels by expanding microvasculature permeability and the transcapillary passage of serum albumin and controlling albumin synthesis in the liver through cytokine mediation (36). Considering these results and the previous observations by Jank et al. in patients with HNC (20), we believed that a low ALI, which reflects a host’s poor immuno-nutritional status and overexpressed systemic inflammation overall, predicted worsened prognosis in patients with surgically treated OSCC. However, the definite mechanism underlying the ALI–survival outcome association in OSCC warrants further investigation.

In this study, ALI had the highest AUC for OS compared with its components. Because BMI and serum albumin levels are affected by various factors, such as cancer- or treatment-related malnutrition, body fluid volume changes, and hepatic insufficiency, considering BMI or albumin level alone may be insufficient for survival outcome prediction in patients with OSCC. Similarly, the NLR may be influenced by an indolent infection or chronic inflammation condition. Hence, ALI may provide a more comprehensive assessment with less measurement variability, making it a more stable indicator than BMI, serum albumin level, or NLR alone for the simultaneous demonstration of a patient’s immune, nutritional, and systemic inflammation status. BMI was also reported to be correlated with sarcopenia, a major component of cancer–cachexia syndrome and a negative predictor of prognosis of SCC of the head and neck (42). However, it may not be applicable for the precise interpretation of fat versus muscle composition (43). Some patients diagnosed as having sarcopenic obesity have low skeletal muscle mass but heavy body weight because of their high fat mass. Hence, Kim et al. used the L3 skeletal muscle index (SMI) measured through computed tomography to replace BMI and developed a modified ALI (14). Nevertheless, they found that the modified L3 SMI–based ALI had no additional prognostic value beyond the original BMI-based ALI in patients with SCLC and concluded that the original ALI was simple but robust for prognosis prediction. These findings confirm the relevance of our study results: ALI has high prognostic value in patients who are to undergo primary surgery for OSCC.

The AJCC staging system has always been an important reference with generalizability and applicability for cancer treatment and prognostic prediction globally. Nevertheless, it does not consider certain OSCC characteristics and demographic features that are frequently considered by clinicians when making treatment decisions. Nomograms are reliable and individualized prediction tools widely applied in oncology research and enable clinical physicians to conveniently execute practical assessments (44). Through the incorporation of diverse prognostic factors, nomograms can determine the probability of a clinical event (e.g., OS or DFS) and thus predict the prognosis of a single patient. Nomograms are necessary in the era of individualized oncological therapy, and several nomograms have been published as adjuncts in prognostic determination in different types of cancer, such as CRC (45), renal cancer (46), NSCLC (47), and gastric cancer (48). Regarding the HNC, the nomograms have also been constructed to estimate the risk of developing major surgical complications in OSCC patients (49) and predict the recurrence-free probability in patients with parotid cancer (50). As we determined after executing a literature review, the current study is the first to establish ALI-based nomograms; the main advantage of our established nomograms is their strong discriminatory ability in estimating individualized 3- and 5-year OS and DFS rates. Our multivariate analysis revealed advanced overall stage, presence of ENE, poor cell differentiation, and low ALI to be independent adverse predictors of OS and DFS in patients with OSCC; therefore, we constructed nomograms based on these independent prognostic factors with feasible results (C-index: 0.773 for OS and 0.674 for DFS). Calibration plots demonstrated high consistency between the OS and DFS predictions provided by the established nomograms and observed values. As illustrated in Figures 4 and 5, the calibration plots of the nomograms are quite close to the ideal 45° line with even distributions, suggesting that the incidence rates predicted by the nomograms were close to the actual observed incidence rates. These results confirm the high performance of the established ALI-based nomograms and verify that they can be used for the prediction of individualized 3- and 5-year OS and DFS rates in OSCC, possibly aiding surgeons in identifying patients who may benefit more from aggressive treatment and therefore influencing all aspects of cancer care, including the survival outcomes.

Although the ROC curve analysis revealed 33.6 to be the cutoff for ALI in our study, different ALI cutoff values have been adopted in other studies. Thus, the optimal threshold remains uncertain. In the first-ever study on ALI, Jafri et al. reported that the cutoff ALI value was 18, but they analyzed only 173 patients who were undergoing palliative chemotherapy for metastatic NSCLC (13). For patients with SCLC, Kim et al. (14) and He et al. (17) have reported cutoff values of 31.1 and 19.5, respectively. Moreover, Jank et al. indicated that ALI had prognostic value at a cutoff of 37.6 in 93 patients with HNC (20), similar to the cutoff value (i.e., 37.66) reported for resected NSCLC by Tomita et al. (18). Our analysis also provided a cutoff within the range of those reported previously. However, the lack of uniformity in the ALI cutoff values among the various cancers may impede the general applicability of pretreatment ALI in clinical practice; thus, further relevant investigation is warranted.

At present, the mainstay of treatment strategies should chiefly be based on the National Comprehensive Cancer Network (NCCN) guidelines, rather than inflammatory markers such as ALI. Of note, the indications of adjuvant therapy for patients with OSCC in our institute were not completely in line with the NCCN guidelines, and the results of their comparison have been reported by our colleague (27). However, our results reveal that preoperative ALI is a useful, convenient preoperative marker in patients with OSCC and that nomograms based on ALI may exhibit a high predictive accuracy for OS and DFS, both of which are meaningful findings. In general, in this study, the prognostic value of preoperative ALI was investigated by including a relatively large cohort of patients who underwent surgery for OSCC. We suggest that for convenience, preoperative ALI can be determined through routine blood test results and body stature measurements, and after surgery, nomograms can be derived by incorporating ALI and clinicopathological factors, thus developing a reliable and cost-effective tool for predicting OSCC prognosis and stratifying patients for suitable adjuvant therapy; this can ultimately facilitate the management of surgically treated OSCC in clinical settings. The current study’s limitations are as follows: First, our study design (single-institution and retrospective design) has inherent limitations that cannot be completed excluded, such as information bias. Second, our results could not be verified on an independent data set; thus, their external validity remains unconfirmed. The established nomogram may also suffer from over-optimism because it was developed and evaluated on the same database (51). Future large-scale, prospective cohort studies along with external validation are thus warranted.

Conclusions

The preoperative ALI may be applicable as a prognostic biomarker in patients with operable OSCC. Here, a low preoperative ALI was found to be associated with aggressive clinicopathological characteristics; it was also an independent risk factor for poor OS and DFS. Nomograms incorporating ALI into the conventional AJCC staging system might provide accurate prognostic information regarding OS and DFS to clinicians, thus enabling them to optimize adjuvant therapy and administer personalized treatment. Given the convenience and cost-effectiveness of this biomarker, ALI is a highly favorable candidate for use in clinical and oncology research.

Data Availability Statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics Statement

Chang Gung Memorial Hospital’s Institutional Review Board ratified our executed study, which followed the tenets of the Declaration of Helsinki. Furthermore, the hospital’s board exempted our study from the established requirement of receiving patient informed consent (approval no. 202000657B0).

Author Contributions

Y-TT and G-HC conceived, designed, and supervised the study. M-ST, Y-CL, and EH collected the data of patients and followed up. C-HL and K-HF analyzed the data. C-HL and G-HC provided technical assistance with the data analysis. Y-TT, K-HF, and C-MH wrote the manuscript. All authors contributed to the article and approved the submitted version.

Funding

This work was supported by a grant (CMRPG6J0202) from Chang Gung Memorial Hospital, Taiwan.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Acknowledgments

The authors would like to thank the HIE lab of Chiayi Chang Gung Memorial Hospital for their great support for the study.

References

1. Warnakulasuriya S. Global Epidemiology of Oral and Oropharyngeal Cancer. Oral Oncol (2009) 45(4-5):309–16. doi: 10.1016/j.oraloncology.2008.06.002

PubMed Abstract | CrossRef Full Text | Google Scholar

2. Liao CT, Wallace CG, Lee LY, Hsueh C, Lin CY, Fan KH, et al. Clinical Evidence of Field Cancerization in Patients With Oral Cavity Cancer in a Betel Quid Chewing Area. Oral Oncol (2014) 50(8):721–31. doi: 10.1016/j.oraloncology.2014.04.010

PubMed Abstract | CrossRef Full Text | Google Scholar

3. Moore SR, Johnson NW, Pierce AM, Wilson DF. The Epidemiology of Mouth Cancer: A Review of Global Incidence. Oral Dis (2000) 6(2):65–74. doi: 10.1111/j.1601-0825.2000.tb00104.x

PubMed Abstract | CrossRef Full Text | Google Scholar

4. De Paz D, Kao HK, Huang Y, Chang KP. Prognostic Stratification of Patients With Advanced Oral Cavity Squamous Cell Carcinoma. Curr Oncol Rep (2017) 19(10):65. doi: 10.1007/s11912-017-0624-3

PubMed Abstract | CrossRef Full Text | Google Scholar

5. Yao C, Chang EI, Lai SY. Contemporary Approach to Locally Advanced Oral Cavity Squamous Cell Carcinoma. Curr Oncol Rep (2019) 21(11):99. doi: 10.1007/s11912-019-0845-8

PubMed Abstract | CrossRef Full Text | Google Scholar

6. Lakshmaiah KC, Suresh TM, Babu KG, Sirsath NT, Dasappa L, Abraham LJ. Locally Advanced Oral Cavity Squamous Cell Carcinoma: Barriers Related to Effective Treatment. South Asian J Cancer (2015) 4(2):61–4. doi: 10.4103/2278-330X.155637

PubMed Abstract | CrossRef Full Text | Google Scholar

7. Hanahan D, Weinberg RA. Hallmarks of Cancer: The Next Generation. Cell (2011) 144(5):646–74. doi: 10.1016/j.cell.2011.02.013

PubMed Abstract | CrossRef Full Text | Google Scholar

8. Todoric J, Antonucci L, Karin M. Targeting Inflammation in Cancer Prevention and Therapy. Cancer Prev Res (Phila) (2016) 9(12):895–905. doi: 10.1158/1940-6207.CAPR-16-0209

PubMed Abstract | CrossRef Full Text | Google Scholar

9. Fang KH, Lai CH, Hsu CM, Huang E, Tsai MS, Chang GH, et al. A Retrospective Study on the Prognostic Value of Preoperative C-Reactive Protein to Albumin Ratio in Patients With Oral Cavity Squamous Cell Carcinoma. PeerJ (2020) 8:e9361. doi: 10.7717/peerj.9361

PubMed Abstract | CrossRef Full Text | Google Scholar

10. Takenaka Y, Oya R, Kitamiura T, Ashida N, Shimizu K, Takemura K, et al. Platelet Count and Platelet-Lymphocyte Ratio as Prognostic Markers for Head and Neck Squamous Cell Carcinoma: Meta-Analysis. Head Neck (2018) 40(12):2714–23. doi: 10.1002/hed.25366

PubMed Abstract | CrossRef Full Text | Google Scholar

11. Wang YT, Fang KH, Hsu CM, Chang GH, Lai CH, Lee YC, et al. Retrospective Study on the Potential of Albumin/Globulin Ratio as a Prognostic Biomarker for Oral Cavity Cancer Patients. Eur Arch Otorhinolaryngol (2021) 278(1):227–38. doi: 10.1007/s00405-020-06145-x

PubMed Abstract | CrossRef Full Text | Google Scholar

12. Yang L, Huang Y, Zhou L, Dai Y, Hu G. High Pretreatment Neutrophil-to-Lymphocyte Ratio as a Predictor of Poor Survival Prognosis in Head and Neck Squamous Cell Carcinoma: Systematic Review and Meta-Analysis. Head Neck (2019) 41(5):1525–35. doi: 10.1002/hed.25583

PubMed Abstract | CrossRef Full Text | Google Scholar

13. Jafri SH, Shi R, Mills G. Advance Lung Cancer Inflammation Index (Ali) at Diagnosis Is a Prognostic Marker in Patients With Metastatic Non-Small Cell Lung Cancer (Nsclc): A Retrospective Review. BMC Cancer (2013) 13:158. doi: 10.1186/1471-2407-13-158

PubMed Abstract | CrossRef Full Text | Google Scholar

14. Kim EY, Kim N, Kim YS, Seo JY, Park I, Ahn HK, et al. Prognostic Significance of Modified Advanced Lung Cancer Inflammation Index (Ali) in Patients With Small Cell Lung Cancer_ Comparison With Original Ali. PloS One (2016) 11(10):e0164056. doi: 10.1371/journal.pone.0164056

PubMed Abstract | CrossRef Full Text | Google Scholar

15. Park YH, Yi HG, Lee MH, Kim CS, Lim JH. Prognostic Value of the Pretreatment Advanced Lung Cancer Inflammation Index (Ali) in Diffuse Large B Cell Lymphoma Patients Treated With R-Chop Chemotherapy. Acta Haematol (2017) 137(2):76–85. doi: 10.1159/000452991

PubMed Abstract | CrossRef Full Text | Google Scholar

16. Shiroyama T, Suzuki H, Tamiya M, Tamiya A, Tanaka A, Okamoto N, et al. Pretreatment Advanced Lung Cancer Inflammation Index (Ali) for Predicting Early Progression in Nivolumab-Treated Patients With Advanced Non-Small Cell Lung Cancer. Cancer Med (2018) 7(1):13–20. doi: 10.1002/cam4.1234

PubMed Abstract | CrossRef Full Text | Google Scholar

17. He X, Zhou T, Yang Y, Hong S, Zhan J, Hu Z, et al. Advanced Lung Cancer Inflammation Index, a New Prognostic Score, Predicts Outcome in Patients With Small-Cell Lung Cancer. Clin Lung Cancer (2015) 16(6):e165–71. doi: 10.1016/j.cllc.2015.03.005

PubMed Abstract | CrossRef Full Text | Google Scholar

18. Tomita M, Ayabe T, Nakamura K. The Advanced Lung Cancer Inflammation Index Is an Independent Prognostic Factor After Surgical Resection in Patients With Non-Small-Cell Lung Cancer. Interact Cardiovasc Thorac Surg (2018) 26(2):288–92. doi: 10.1093/icvts/ivx329

PubMed Abstract | CrossRef Full Text | Google Scholar

19. Feng JF, Huang Y, Chen QX. A New Inflammation Index Is Useful for Patients With Esophageal Squamous Cell Carcinoma. Onco Targets Ther (2014) 7:1811–5. doi: 10.2147/OTT.S68084

PubMed Abstract | CrossRef Full Text | Google Scholar

20. Jank BJ, Kadletz L, Schnoll J, Selzer E, Perisanidis C, Heiduschka G. Prognostic Value of Advanced Lung Cancer Inflammation Index in Head and Neck Squamous Cell Carcinoma. Eur Arch Otorhinolaryngol (2019) 276(5):1487–92. doi: 10.1007/s00405-019-05381-0

PubMed Abstract | CrossRef Full Text | Google Scholar

21. Shibutani M, Maeda K, Nagahara H, Fukuoka T, Matsutani S, Kimura K, et al. The Prognostic Significance of the Advanced Lung Cancer Inflammation Index in Patients With Unresectable Metastatic Colorectal Cancer: A Retrospective Study. BMC Cancer (2019) 19(1):241. doi: 10.1186/s12885-019-5468-9

PubMed Abstract | CrossRef Full Text | Google Scholar

22. Caldeira PC, Soto AML, de Aguiar MCF, Martins CC. Tumor Depth of Invasion and Prognosis of Early-Stage Oral Squamous Cell Carcinoma: A Meta-Analysis. Oral Dis (2020) 26(7):1357–65. doi: 10.1111/odi.13194

PubMed Abstract | CrossRef Full Text | Google Scholar

23. Shaw RJ, Lowe D, Woolgar JA, Brown JS, Vaughan ED, Evans C, et al. Extracapsular Spread in Oral Squamous Cell Carcinoma. Head Neck (2010) 32(6):714–22. doi: 10.1002/hed.21244

PubMed Abstract | CrossRef Full Text | Google Scholar

24. Eltohami YI, Kao HK, Lao WW, Huang Y, Abdelrahman M, Liao CT, et al. The Prediction Value of the Systemic Inflammation Score for Oral Cavity Squamous Cell Carcinoma. Otolaryngol Head Neck Surg (2018) 158(6):1042–50. doi: 10.1177/0194599817751678

PubMed Abstract | CrossRef Full Text | Google Scholar

25. Lu Z, Yan W, Liang J, Yu M, Liu J, Hao J, et al. Nomogram Based on Systemic Immune-Inflammation Index to Predict Survival of Tongue Cancer Patients Who Underwent Cervical Dissection. Front Oncol (2020) 10:341. doi: 10.3389/fonc.2020.00341

PubMed Abstract | CrossRef Full Text | Google Scholar

26. Charlson ME, Charlson RE, Peterson JC, Marinopoulos SS, Briggs WM, Hollenberg JP. The Charlson Comorbidity Index Is Adapted to Predict Costs of Chronic Disease in Primary Care Patients. J Clin Epidemiol (2008) 61(12):1234–40. doi: 10.1016/j.jclinepi.2008.01.006

PubMed Abstract | CrossRef Full Text | Google Scholar

27. Lin CY, Fan KH, Lee LY, Hsueh C, Yang LY, Ng SH, et al. Precision Adjuvant Therapy Based on Detailed Pathologic Risk Factors for Resected Oral Cavity Squamous Cell Carcinoma: Long-Term Outcome Comparison of Cgmh and Nccn Guidelines. Int J Radiat Oncol Biol Phys (2020) 106(5):916–25. doi: 10.1016/j.ijrobp.2019.08.058

PubMed Abstract | CrossRef Full Text | Google Scholar

28. Harrell FE Jr., Lee KL, Mark DB. Multivariable Prognostic Models: Issues in Developing Models, Evaluating Assumptions and Adequacy, and Measuring and Reducing Errors. Stat Med (1996) 15(4):361–87. doi: 10.1002/(SICI)1097-0258(19960229)15:4<361::AID-SIM168>3.0.CO;2-4

PubMed Abstract | CrossRef Full Text | Google Scholar

29. Vander Poorten VL, Hart AA, van der Laan BF, Baatenburg de Jong RJ, Manni JJ, Marres HA, et al. Prognostic Index for Patients With Parotid Carcinoma: External Validation Using the Nationwide 1985-1994 Dutch Head and Neck Oncology Cooperative Group Database. Cancer (2003) 97(6):1453–63. doi: 10.1002/cncr.11254

PubMed Abstract | CrossRef Full Text | Google Scholar

30. Brentnall AR, Cuzick J, Field J, Duffy SW. A Concordance Index for Matched Case-Control Studies With Applications in Cancer Risk. Stat Med (2015) 34(3):396–405. doi: 10.1002/sim.6335

PubMed Abstract | CrossRef Full Text | Google Scholar

31. Mayr A, Schmid M. Boosting the Concordance Index for Survival Data–a Unified Framework to Derive and Evaluate Biomarker Combinations. PloS One (2014) 9(1):e84483. doi: 10.1371/journal.pone.0084483

PubMed Abstract | CrossRef Full Text | Google Scholar

32. Takenaka Y, Oya R, Kitamiura T, Ashida N, Shimizu K, Takemura K, et al. Prognostic Role of Neutrophil-to-Lymphocyte Ratio in Head and Neck Cancer: A Meta-Analysis. Head Neck (2018) 40(3):647–55. doi: 10.1002/hed.24986

PubMed Abstract | CrossRef Full Text | Google Scholar

33. Vanhoutte G, van de Wiel M, Wouters K, Sels M, Bartolomeeussen L, De Keersmaecker S, et al. Cachexia in Cancer: What Is in the Definition? BMJ Open Gastroenterol (2016) 3(1):e000097. doi: 10.1136/bmjgast-2016-000097

PubMed Abstract | CrossRef Full Text | Google Scholar

34. Ozyurek BA, Ozdemirel TS, Ozden SB, Erdogan Y, Ozmen O, Kaplan B, et al. Does Advanced Lung Inflammation Index (Ali) Have Prognostic Significance in Metastatic Non-Small Cell Lung Cancer? Clin Respir J (2018) 12(6):2013–19. doi: 10.1111/crj.12768

PubMed Abstract | CrossRef Full Text | Google Scholar

35. Kindlova M, Mrklas L, Broukal Z, Povysilova V. The Influence of Age and the Plaque Microbes Spectrum on the Development and Healing of Periodontal Lesions in Rats. J Periodontal Res (1985) 20(6):621–4. doi: 10.1111/j.1600-0765.1985.tb00846.x

PubMed Abstract | CrossRef Full Text | Google Scholar

36. Lim WS, Roh JL, Kim SB, Choi SH, Nam SY, Kim SY. Pretreatment Albumin Level Predicts Survival in Head and Neck Squamous Cell Carcinoma. Laryngoscope (2017) 127(12):E437–E42. doi: 10.1002/lary.26691

PubMed Abstract | CrossRef Full Text | Google Scholar

37. Felix K, Gaida MM. Neutrophil-Derived Proteases in the Microenvironment of Pancreatic Cancer -Active Players in Tumor Progression. Int J Biol Sci (2016) 12(3):302–13. doi: 10.7150/ijbs.14996

PubMed Abstract | CrossRef Full Text | Google Scholar

38. Mantovani A, Allavena P, Sica A, Balkwill F. Cancer-Related Inflammation. Nature (2008) 454(7203):436–44. doi: 10.1038/nature07205

PubMed Abstract | CrossRef Full Text | Google Scholar

39. Zhang X, Tang T, Pang L, Sharma SV, Li R, Nyitray AG, et al. Malnutrition and Overall Survival in Older Adults With Cancer: A Systematic Review and Meta-Analysis. J Geriatr Oncol (2019) 10(6):874–83. doi: 10.1016/j.jgo.2019.03.002

PubMed Abstract | CrossRef Full Text | Google Scholar

40. Takenaka Y, Takemoto N, Nakahara S, Yamamoto Y, Yasui T, Hanamoto A, et al. Prognostic Significance of Body Mass Index Before Treatment for Head and Neck Cancer. Head Neck (2015) 37(10):1518–23. doi: 10.1002/hed.23785

PubMed Abstract | CrossRef Full Text | Google Scholar

41. Jeevanandam M, Horowitz GD, Lowry SF, Brennan MF. Cancer Cachexia and Protein Metabolism. Lancet (1984) 1(8392):1423–6. doi: 10.1016/s0140-6736(84)91929-9

PubMed Abstract | CrossRef Full Text | Google Scholar

42. Wong A, Zhu D, Kraus D, Tham T. Radiologically Defined Sarcopenia Affects Survival in Head and Neck Cancer: A Meta-Analysis. Laryngoscope (2020) 131(2):333–41. doi: 10.1002/lary.28616

PubMed Abstract | CrossRef Full Text | Google Scholar

43. Levolger S, van Vugt JL, de Bruin RW, JN IJ. Systematic Review of Sarcopenia in Patients Operated on for Gastrointestinal and Hepatopancreatobiliary Malignancies. Br J Surg (2015) 102(12):1448–58. doi: 10.1002/bjs.9893

PubMed Abstract | CrossRef Full Text | Google Scholar

44. Montero PH, Yu C, Palmer FL, Patel PD, Ganly I, Shah JP, et al. Nomograms for Preoperative Prediction of Prognosis in Patients With Oral Cavity Squamous Cell Carcinoma. Cancer (2014) 120(2):214–21. doi: 10.1002/cncr.28407

PubMed Abstract | CrossRef Full Text | Google Scholar

45. Kattan MW, Gonen M, Jarnagin WR, DeMatteo R, D'Angelica M, Weiser M, et al. A Nomogram for Predicting Disease-Specific Survival After Hepatic Resection for Metastatic Colorectal Cancer. Ann Surg (2008) 247(2):282–7. doi: 10.1097/SLA.0b013e31815ed67b

PubMed Abstract | CrossRef Full Text | Google Scholar

46. Yan H, Wei X, Wu A, Sha Y, Li X, Qi F. Nomograms for Predicting Overall and Cancer-Specific Survival in Patients With Papillary Renal Cell Carcinoma: A Population-Based Study Using Seer Database. Transl Androl Urol (2020) 9(3):1146–58. doi: 10.21037/tau-19-807

PubMed Abstract | CrossRef Full Text | Google Scholar

47. Xie D, Allen MS, Marks R, Jiang G, Sun Z, Nichols F, et al. Nomogram Prediction of Overall Survival for Patients With Non-Small-Cell Lung Cancer Incorporating Pretreatment Peripheral Blood Markers. Eur J Cardiothorac Surg (2018) 53(6):1214–22. doi: 10.1093/ejcts/ezx462

PubMed Abstract | CrossRef Full Text | Google Scholar

48. Wang CY, Yang J, Zi H, Zheng ZL, Li BH, Wang Y, et al. Nomogram for Predicting the Survival of Gastric Adenocarcinoma Patients Who Receive Surgery and Chemotherapy. BMC Cancer (2020) 20(1):10. doi: 10.1186/s12885-019-6495-2

PubMed Abstract | CrossRef Full Text | Google Scholar

49. Awad MI, Palmer FL, Kou L, Yu C, Montero PH, Shuman AG, et al. Individualized Risk Estimation for Postoperative Complications After Surgery for Oral Cavity Cancer. JAMA Otolaryngol Head Neck Surg (2015) 141(11):960–8. doi: 10.1001/jamaoto.2015.2200

PubMed Abstract | CrossRef Full Text | Google Scholar

50. Peeperkorn S, Meulemans J, Van Lierde C, Laenen A, Valstar MH, Balm AJM, et al. Validated Prognostic Nomograms for Patients With Parotid Carcinoma Predicting 2- and 5-Year Tumor Recurrence-Free Interval Probability. Front Oncol (2020) 10:1535. doi: 10.3389/fonc.2020.01535

PubMed Abstract | CrossRef Full Text | Google Scholar

51. Van Houwelingen JC, Le Cessie S. Predictive Value of Statistical Models. Stat Med (1990) 9(11):1303–25. doi: 10.1002/sim.4780091109

PubMed Abstract | CrossRef Full Text | Google Scholar