Preoperative Prediction of Lymph Node Metastasis in Patients With Early-T-Stage Non-small Cell Lung Cancer by Machine Learning Algorithms

Background: Lymph node metastasis (LNM) is difficult to precisely predict before surgery in patients with early-T-stage non-small cell lung cancer (NSCLC). This study aimed to develop machine learning (ML)-based predictive models for LNM.

Methods: Clinical characteristics and imaging features were retrospectively collected from 1,102 NSCLC ≤ 2 cm patients. A total of 23 variables were included to develop predictive models for LNM by multiple ML algorithms. The models were evaluated by the receiver operating characteristic (ROC) curve for predictive performance and decision curve analysis (DCA) for clinical values. A feature selection approach was used to identify optimal predictive factors.

Results: The areas under the ROC curve (AUCs) of the 8 models ranged from 0.784 to 0.899. Some ML-based models performed better than models using conventional statistical methods in both ROC curves and decision curves. The random forest classifier (RFC) model with 9 variables introduced was identified as the best predictive model. The feature selection indicated the top five predictors were tumor size, imaging density, carcinoembryonic antigen (CEA), maximal standardized uptake value (SUV_max), and age.

Conclusions: By incorporating clinical characteristics and radiographical features, it is feasible to develop ML-based models for the preoperative prediction of LNM in early-T-stage NSCLC, and the RFC model performed best.

Introduction

Lung cancer remains the leading cause of global cancer death (1). Early-T-stage non-small cell lung cancer (NSCLC) has been detected more frequently following the rapid development and employment of radiographical technology (2). An accurate nodal stage is critical for treatment decision-making (3). Currently, there are several evaluation methods, such as computed tomography (CT), positron emission tomography/CT (PET/CT), mediastinoscopy and endobronchial ultrasound transbronchial needle aspiration (EBUS-TBNA), that can be used to classify the nodal stage before operation. However, performing mediastinoscopy or EBUS-TBNA is not cost-effective for patients with early-stage NSCLC. Furthermore, although CT and PET/CT have been widely used for the preoperative evaluation of lung cancer, the incidence of occult lymph node metastasis (LNM) in early-T-stage NSCLC remains high and cannot be ignored (4, 5). Therefore, new reliable methods for the preoperative prediction of LNM are highly required.

Machine learning (ML) is an emerging computer-based method that has been widely used for data analysis in medicine during the past decade (6, 7). It learns from data and finds the dataset pattern to identify the outcome (7, 8). Supervised ML is a process in which the model is trained with fully labeled and classified data. Compared with conventional statistical methods such as logistic regression (LR), which relies on predetermined models, ML can deeply detect the interactions among variations and iteratively learn from data to update algorithms (9).

A number of predictive models have been made based on ML algorithms. Several studies have reported effective ML-based models for the prediction of LNM in other carcinomas, such as breast cancer (10, 11). It was reported that radiomics could be used to predict LNM by analyzing radiological images in NSCLC (12). However, few reports have incorporated clinical characteristics and radiographical features as in our study. This study aimed to develop and validate effective ML-based models for the prediction of LNM in patients with early-T-stage NSCLC.

Materials and Methods

Study Population

Between January 2013 and June 2019, 1,102 patients who underwent surgical resection for NSCLC at Peking Union Medical College Hospital were included in this study. The inclusion criteria were as follows: (1) single NSCLC lesion; (2) tumor maximum diameter ≤ 2 cm on CT; and (3) receiving lung resection with systematic lymph node dissection. The exclusion criteria were as follows: (1) small cell lung cancer (SCLC); (2) multiple lung cancer; (3) receiving radiotherapy or chemotherapy before surgery; (4) distant metastasis; and (5) incomplete clinical records. The pathological classification of carcinomas was based on the 2015 World Health Organization (WHO) classification (13). The clinical and pathological staging was performed according to the 8th edition of the TNM staging system (14). This study was approved by the Ethics Committee of Peking Union Medical College Hospital. All patients signed informed consent before operation.

Clinical Characteristics and Radiographical Features

A total of 23 variables were analyzed in this study. The patients' clinical characteristics included age, sex, smoking status and serum tumor biomarkers. All preoperative serum tumor biomarkers were measured within 3 months before surgery, including carbohydrate antigen 24-2 (CA242), squamous cell carcinoma antigen (SCCAg), carcinoembryonic antigen (CEA), carbohydrate antigen 19-9 (CA199), carbohydrate antigen 12-5 (CA125), carbohydrate antigen 72-4 (CA724), carbohydrate antigen 15-3 (CA153), neuron-specific enolase (NSE), tissue polypeptide-specific antigen (TPS), cytokeratin 19-fragments (Cyfra211) and pro-gastrin-releasing peptide (proGRP). CT features were reviewed by one radiologist and two thoracic surgeons independently, including tumor location side, tumor maximum size, spiculation, vessel convergence, lobulation, pleural indentation, calcification, and imaging density. If disagreement occurred, the final result was reached by consensus. Based on imaging density on CT, the cancer lesions were divided into pure ground-glass opacity (pGGO), mixed GGO (mGGO) and solid nodules. The mGGO was further divided into two groups according to different percentages of solid components, whose cut-off value was 50% (the ratio between the maximal diameter of the solid component at the mediastinal window and the maximal tumor diameter at the lung window). In addition, the maximal standardized uptake value (SUV_max) on PET/CT was also included. However, PET scan was not routinely performed in early-T-stage NSCLC. All patients underwent CT or PET scan within 60 days at our hospital before the operation.

Construction of ML-Based Models

All patients were randomly divided into training and testing groups at a ratio of 8:2, keeping the distribution of node-positive and node-negative data in both groups consistent. To construct more reliable ML-based predictive models, all continuous variables were preprocessed by z-score normalization except for multinomial naïve Bayes (MNB) in which min-max normalization is preferred (15). Some continuous variables with missing data (Table S1), such as SUV_max and tumor biomarkers, were imputed by median value (16, 17).

Eight algorithms were applied to predict LNM, including adaptive boosting (AdaBoost), artificial neural network (ANN), decision tree (DT), gradient boosting decision tree (GBDT), logistic regression (LR), MNB, random forest classifier (RFC), and extreme gradient boosting (XGBoost) (18–23). Among all 8 algorithms, LR and MNB are considered conventional methods, and the others are representative supervised ML-based algorithms. Only DT, LR, and MNB were interpretable, in which users were able to recognize function between variable and predictive outcome.

The prediction ability of the 8 models was first evaluated by the receiver operating characteristic (ROC) curve, which is a conventional diagnostic test method that only pays attention to the sensitivity and specificity but ignores the clinical utility of predictive information. Decision curve analysis (DCA) was performed to calculate the clinical values of these models, which is a novel method to assess the information value between diagnostic models by considering the possible range of a patient's risk and benefit preferences without actually measuring these preferences for one particular patient (24).

Validation Strategy and Feature Selection

Overfitting is a common problem in ML, especially with high dimensions (number of variables). To minimize the negative influence of overfitting, some strategies, such as the preselection of variables and cross-validation, were feasible (25, 26). Therefore, 5-fold cross-validation and feature selection were performed in this study. The 5-fold cross-validation randomly split the dataset into 5 subsets. For each repeated time, four subsets were used as the training group and the remaining subset was used as the testing data. This procedure was repeated 5 times, and each subset should be used exactly once as the testing group. To rank and select meaningful variables, a classifier-specific evaluator was used, returning a ranked list of variables for each algorithm. The ranks of each variable in different algorithms were compared, and the variables with high ranks were identified.

Statistical Analysis

Univariate analysis was performed using IBM SPSS 25.0 (SPSS Inc; Chicago, IL, USA). Quantitative data were first tested for normality by the Shapiro-Wilk test. Normal data are expressed as the mean ± standard deviation (SD), while non-normal data are expressed as the median with interquartile range (IQR). Student's t-test was used to compare normal quantitative parameters, while the Mann-Whitney U test was used to compare non-normal quantitative parameters. For categorical data, Pearson's chi square test or Fisher's exact test was applied. Python programming language (version 3.7, Python Software Foundation) was used for the construction of ML models and DCA. Student's t-test was also used for the comparison of different ML models (AUCs). A P-value < 0.05 was considered statistically significant.

Results

Patient Characteristics

All 1,102 patients' clinical characteristics and radiographical features are listed in Table 1. Univariate analysis was performed for data without a median value imputed. LNM occurred in 10.5% (116/1102) of patients with NSCLC ≤ 2 cm. In total, 699 (63.4%) patients were female, and LNM occurred more frequently in smokers (P = 0.026). The maximum tumor size on CT in patients with positive nodes was significantly larger than that in patients with negative nodes (P < 0.001). All patients had a maximal diameter no smaller than 4 mm. Tumor imaging density (P < 0.001) and pleural indentation (P = 0.006) also presented significant differences between node-positive and node-negative patients. None of the patients with positive nodes in this study had a pGGO cancer nodule. Moreover, patients with LNM were significantly different from those without LNM in 4 serum tumor biomarkers: CEA (P < 0.001), CA125 (P = 0.001), CA153 (P = 0.030), and Cyfra211 (P = 0.013).

TABLE 1

Table 1. Univariate analysis of patients' clinical characteristics and image features.

Predictive Performance and Clinical Utility of ML-Based Models

A total of 23 preoperative variables were used to develop predictive models for LNM based on 8 algorithms. The predictive performance of all models is shown in Figure 1 and Table 2. The best performance was observed in the GBDT model (AUC = 0.899, SD = 0.048), which performed similarly to RFC (AUC = 0.890, SD = 0.045, P = 0.773), XGBoost (AUC = 0.883, SD = 0.047, P = 0.627), AdaBoost (AUC = 0.873, SD = 0.048, P = 0.432), and ANN (AUC = 0.868, SD = 0.049, P =0.341). All ML-based models except DT (AUC = 0.802, SD = 0.057) were better than the two conventional methods, LR (AUC = 0.867, SD = 0.049, P = 0.338) and MNB (AUC = 0.784, SD = 0.058, P = 0.002). Moreover, all models performed significantly better than using only tumor size (AUC = 0.753, SD = 0.023, P < 0.001; the cut-off value was 1.5 cm), SUV_max (AUC = 0.734, SD = 0.024, P < 0.001; the cut-off value was 2.8) or CEA (AUC = 0.720, SD = 0.026, p < 0.001; the cut-off value was 2.98 ng/ml).

FIGURE 1

Figure 1. Receiver operating characteristic (ROC) curve for 8 models. AdaBoost, adaptive boosting; ANN, artificial neural network; DT, decision tree; GBDT, gradient boosting decision tree; LR, logistic regression; MNB, multinomial naïve Bayes; RFC, random forest classifier; XGBoost, extreme gradient boosting.

TABLE 2

Table 2. Predictive performance (AUC) of 8 models and using several variables alone.

Furthermore, the decision curve showed the clinical values of these models (Figure 2). The net benefits of 8 models at each threshold probability are shown in Table S2. Most of these models presented better net benefits than two control models that were represented by positive and negative line, respectively. The negative line represents the net benefit is zero when none of patients receive lobectomy with systematic lymph node dissection (SND), assuming that all patients have no positive nodes. On the contrary, the positive line represents the net benefits at the time when all patients have positive nodes and receive lobectomy with SND. Four models (RFC, XGBoost, GBDT, and LR) performed significantly better than the others at most of threshold points. At the range of 0.2–0.5, the LR model was less beneficial than RFC, XGBoost and GBDT on most occasions. The RFC model with 9 variables introduced, which achieved a very high AUC (0.890) and had the highest net benefits almost across the entire range of threshold probabilities, was regarded as the best predictive model in this study, although its AUC value was slightly lower than that of GBDT (P = 0.773).

FIGURE 2

Figure 2. Decision curve for 8 models. AdaBoost, adaptive boosting; ANN, artificial neural network; DT, decision tree; GBDT, gradient boosting decision tree; LR, logistic regression; MNB, multinomial naïve Bayes; RFC, random forest classifier; XGBoost, extreme gradient boosting.

Variable Importance

By feature selection, the 23 variables for each algorithm were ranked by their predictive importance (Table S3). The top 10 variables are shown in Figure 3. The five top-ranked predictors were tumor size, imaging density, CEA, SUV_max, and age. The relationship between the AUCs of models and the number of variables were evaluated in Figure 4. The AUCs of most models reached a plateau when 7 variables were introduced, while those of ANN, DT, and MNB started to drop down when they reached the highest points. The AUCs of RFC for each number of variables are shown in Figure 5. Its AUC value reached a plateau when 9 variables were introduced and reached the highest value when 13 variables were introduced, but it did not increase significantly with the change from 9 variables (AUC = 0.886) to 13 variables (AUC = 0.890) introduced. Considering the clinical utility, the 9 top-ranked variables were identified to construct the optimal predictive model, which included tumor size, SUV_max, imaging density, vessel convergence sign, CEA, CA125, sex, age, and spiculation sign.

FIGURE 3

Figure 3. Ranks of the top 10 variables for the prediction of lymph node metastasis. Variables were ranked using a classifier-specific evaluator based on machine learning algorithms. Each variable was ordered according to their mean ranks. The lower rank represents more contributions to the prediction of lymph node metastasis. For example, SUV_max was ranked 2nd, 3rd, 3^rd, and 5th in RFC, GBDT, LR, and XGB, respectively. TS, tumor size; ID, imaging density; CEA, carcinoembryonic antigen; SUVmax, maximal standardized uptake value; VCS, vessel convergence sign on CT; CA125, carbohydrate antigen 12-5; Cyfra211, cytokeratin 19-fragments; proGRP, pro-gastrin-releasing peptide.

FIGURE 4

Figure 4. Predictive performance (AUCs) of 8 models as number of variables increases. AdaBoost, adaptive boosting; ANN, artificial neural network; DT, decision tree; GBDT, gradient boosting decision tree; LR, logistic regression; MNB, multinomial naïve Bayes; RFC, random forest classifier; XGBoost, extreme gradient boosting.

FIGURE 5

Figure 5. Predictive performance (AUCs) of the random forest classifier (RFC) model at each number of variables.

Discussion

Lobectomy with systematic lymph node dissection remains the standard treatment for patients with early-T-stage NSCLC (≤ 2 cm) (27). However, sublobar resection, including segmentectomy and wedge resection, has been proposed to achieve more precise intervention with the advancement of imaging techniques in recent years. In addition, the reasonable extent of lymph node dissection remains controversial. An exact nodal status is critical for treatment selection and prognosis.

In this study, using ML algorithms, we developed 8 models to predict LNM in 1,102 patients with NSCLC ≤ 2 cm, incorporating their clinical characteristics and radiographical features. ROC analysis and DCA were used to evaluate the predictive performance and clinical values of the models, respectively. Most of 8 models maintained high AUCs and All ML-based models (with AUCs ranging from 0.868 to 0.899) except DT performed better than two models using conventional statistical methods (LR and MNB) in the prediction of LNM (Figure 1 and Table 2).

DCA has been used for many medical studies and has shown great clinical utility (28, 29). In the decision curve, most of these models performed better than positive line and negative line, indicating that the overall net benefit of giving lobectomy with SND to patients identified by the models to have high risk of LNM was higher than that of giving the same surgical procedures to all patients or no patient. Four models (RFC, XGBoost, GBDT, and LR) performed better than the others at most of threshold points (Figure 2). Thus, these four potential models were used to identify variable importance by feature selection (Figure 3). The other four models, AdaBoost, MNB, DT, and ANN, had lower net benefits in the decision curve (Figure 2), although they possessed high AUCs in the ROC curve. This indicated that models with high predictive accuracy might not be clinically practical and require further evaluation by other methods, such as DCA.

Using conventional univariate analysis, previous studies reported the risk factors associated with LNM in NSCLC ≤ 2 cm, including tumor size, serum CEA and imaging density (30, 31). In addition, SUV_max was also thought to be a risk factor in patients with cT1 NSCLC (32). Thus, the AUCs when using tumor size (AUC = 0.753), SUV_max (AUC = 0.734), or CEA (AUC = 0.720) alone were also calculated, which were significantly lower than those of ML-based models (Table 2). Thus, previous studies might not provide precise predictive information for LNM. Reliable predictive models for LNM in patients with NSCLC are needed. To our knowledge, our study was the first to provide potential models for the prediction of LNM in patients with NSCLC by incorporating clinical characteristics and radiographical features.

Although most of the ML-based models in our study cannot demonstrate the connection between the predictive variables and the outcomes, the contribution of each variable to the models could be inferred by feature selection. Tumor size, imaging density, serum CEA, SUV_max, and age were indicated to be the most contributive risk factors of LNM (Figure 3), which was similar to the results of univariate analysis (Table 1). Since none of the patients with pGGO NSCLC had positive nodes in our and previous studies (30, 31), it could be inferred that pGGO might be predictive of node-negative status in early-T-stage NSCLC. It was also reported that a higher serum CEA level was significantly associated with a higher incidence of LNM (31, 33). Although only 611 patients' SUV_max values (pN+: n = 62, pN0: n = 549; p > 0.05) were available because some patients did not undergo PET scans, SUV_max was ranked at 4 among the four potential models (Figure 3) and was ranked at 2 in the RFC model (Figure 4). Meanwhile, a high AUC (0.734) for SUV_max was also obtained. Above all, SUV_max might be one of the most important predictive factors, which was consistent with previous studies (32, 34). Surprisingly, age showed no significance in univariate analysis (p = 0.382) but was ranked at the top 5 (Figure 3). This might be attributed to the surprising superiority of ML-based models in data mining, which could find more relations between the variables and the outcomes than conventional methods.

According to the ROC curve (Figure 1) and decision curve (Figure 2), the RFC model with 9 variables introduced (AUC = 0.890) was identified as the optimal model. By considering the clinical utility, an application based on the RFC algorithm with 9 variables (AUC = 0.886) should be developed in the future. These 9 variables were tumor size, SUV_max, imaging density, vessel convergence sign, CEA, CA125, sex, age, and spiculation sign. Thus, clinicians from other hospitals could benefit from our study.

In addition to the clinical values, there were several methodological indications in our study. First, although there were several studies of machine learning involving NSCLC, few of them have reported predictive models for LNM using ML algorithms by incorporating clinical characteristics and radiographical features. Most of them performed image analysis by radiographical data (12) or histological slides (35). This is the first study to predict LNM in NSCLC ≤ 2 cm, indicating the feasibility and potential of ML algorithms applied in NSCLC. More predictive models of NSCLC may be developed using ML algorithms to solve clinical problems in the future. Second, based on ROC analysis and DCA, multiple supervised ML algorithms performed better than conventional methods. Thus, the ML algorithms would play an important role in the analysis of large medical datasets. Third, in addition to the ROC curve, a decision curve was used to evaluate the clinical utility of these models. Some models performed worse in the decision curve, although they had very high AUCs. This provides a method to further evaluate the clinical values of ML-based models.

There were also some limitations in our study. First, there were some patients who received sublobar resection (wedge resection or segmentectomy), and thus, the incidence of LNM in this population might have been underestimated. Second, missing data were inevitable. This is because not all patients with early-T-stage NSCLC receive PET scans or tumor biomarker tests. Except for SUV_max and serum biomarkers, the clinical records of other variables were complete. The median value was imputed to solve this problem (16, 17). Third, this is a retrospective study that could not completely avoid data selection and measurement biases. More prospective studies or multicenter studies may be needed to develop predictive models in the future.

Conclusions

ML-based models are effective in the prediction of LNM in NSCLC ≤ 2 cm by incorporating clinical and radiographical characteristics. Based on ROC analysis and DCA, some ML-based models performed better than models using conventional methods, and the RFC model performed best. The feature selection approach identified that tumor size, imaging density, CEA, SUV_max, and age were the most important predictive risk factors for LNM.

Data Availability Statement

All datasets generated for this study are included in the article/Supplementary Material.

Ethics Statement

The studies involving human participants were reviewed and approved by Ethics Committee of Peking Union Medical College Hospital. Written informed consent to participate in this study was provided by the participants' legal guardian/next of kin.

Author Contributions

SL, NL, YW, and JL: conceptualization. YW and JL: methodology. YW, JL, CH, XL, and YC: formal analysis. YW, ZW, LG, JZ, XG, and CG: investigation. YW and JL: writing—original draft preparation. YW, SL, and NL: writing—review and editing. SL: supervision.

Funding

This research was funded by (1) Foundation for Key Program of Ministry of Education, China (Grant No. 311037); (2) CAMS Innovation Fund for Medical Sciences (CIFMS), (2017-12M-1-009; 2019-I2M-1-001); (3) Beijing Natural Science Foundation (7182132); (4) Special Data Service for Oncology, The National Population and Health Scientific Data Sharing Platform (NCMI-ABD02-201809; NCMI-YF02N-201906), supported by Ministry of Science and Technology of the People's Republic of China (MOST); 5) CSCO-CSCO Y-2019GENECAST-051.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Acknowledgments

We would like to give our sincere thanks to Professor Hongsheng Liu, Yushang Cui, Zhijun Han and Zhili Cao for their contributions to the clinical works and we also would like to thank American Journal Experts (www.aje.com) for its linguistic assistance during the preparation of this manuscript.

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fonc.2020.00743/full#supplementary-material

References

1. Siegel RL, Miller KD, Jemal A. Cancer statistics, 2019. CA Cancer J Clin. (2019) 69:7–34. doi: 10.3322/caac.21551

CrossRef Full Text | Google Scholar

2. Aberle DR, DeMello S, Berg CD, Black WC, Brewer B, Church TR, et al. Results of the two incidence screenings in the National Lung Screening Trial. N Engl J Med. (2013) 369:920–31. doi: 10.1056/NEJMoa1208962

PubMed Abstract | CrossRef Full Text | Google Scholar

3. Krantz SB, Lutfi W, Kuchta K, Wang CH, Kim KW, Howington JA. Improved lymph node staging in early-stage lung cancer in the national cancer database. Ann Thorac Surg. (2017) 104:1805–14. doi: 10.1016/j.athoracsur.2017.06.066

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Smeltzer MP, Faris N, Yu X, Ramirez RA, Ramirez LE, Wang CG, et al. Missed intrapulmonary lymph node metastasis and survival after resection of non-small cell lung cancer. Ann Thorac Surg. (2016) 102:448–53. doi: 10.1016/j.athoracsur.2016.03.096

PubMed Abstract | CrossRef Full Text | Google Scholar

5. Hung JJ, Yeh YC, Jeng WJ, Wu YC, Chou TY, Hsu WH. Factors predicting occult lymph node metastasis in completely resected lung adenocarcinoma of 3 cm or smaller. Eur J Cardiothorac Surg. (2016) 50:329–36. doi: 10.1093/ejcts/ezv485

CrossRef Full Text | Google Scholar

6. Thrall JH, Li X, Li Q, Cruz C, Do S, Dreyer K, et al. Artificial intelligence and machine learning in radiology: opportunities, challenges, pitfalls, and criteria for success. J Am Coll Radiol. (2018) 15(3 Pt B):504–8. doi: 10.1016/j.jacr.2017.12.026

PubMed Abstract | CrossRef Full Text | Google Scholar

7. Deo RC. Machine learning in medicine. Circulation. (2015) 132:1920–30. doi: 10.1161/CIRCULATIONAHA.115.001593

PubMed Abstract | CrossRef Full Text | Google Scholar

8. Shouval R, Bondi O, Mishan H, Shimoni A, Unger RA. Application of machine learning algorithms for clinical predictive modeling: a data-mining approach in SCT. Bone Marrow Transplant. (2014) 49:332–7. doi: 10.1038/bmt.2013.146

PubMed Abstract | CrossRef Full Text | Google Scholar

9. Waljee AK, Higgins PD. Machine learning in medicine: a primer for physicians. Am J Gastroenterol. (2010) 105:1224–6. doi: 10.1038/ajg.2010.173

PubMed Abstract | CrossRef Full Text | Google Scholar

10. Dihge L, Vallon-Christersson J, Hegardt C, Saal LH, Hakkinen J, Larsson C, et al. Prediction of lymph node metastasis in breast cancer by gene expression and clinicopathological models: development and validation within a population-based cohort. Clin Cancer Res. (2019) 25:clincanres.0075.2019. doi: 10.1158/1078-0432.CCR-19-0075

PubMed Abstract | CrossRef Full Text | Google Scholar

11. Ehteshami Bejnordi B, Veta M, Johannes van Diest P, van Ginneken B, Karssemeijer N, Litjens GJ, et al. Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer. JAMA. (2017) 318:2199–210. doi: 10.1001/jama.2017.14585

PubMed Abstract | CrossRef Full Text | Google Scholar

12. Zhong Y, Yuan M, Zhang T, Zhang YD, Li H, Yu TF. Radiomics approach to prediction of occult mediastinal lymph node metastasis of lung adenocarcinoma. AJR Am J Roentgenol. (2018) 211:109–13. doi: 10.2214/AJR.17.19074

PubMed Abstract | CrossRef Full Text | Google Scholar

13. Travis WD, Brambilla E, Nicholson AG, Yatabe Y, Austin JHM, Beasley MB, et al. The 2015 World Health Organization classification of lung tumors: impact of genetic, clinical and radiologic advances since the 2004 classification. J Thorac Oncol. (2015) 10:1243–60. doi: 10.1097/JTO.0000000000000630

PubMed Abstract | CrossRef Full Text | Google Scholar

14. Goldstraw P, Chansky K, Crowley J, Rami-Porta R, Asamura H, Eberhardt WE, et al. The IASLC lung cancer staging project: proposals for revision of the TNM stage groupings in the forthcoming (Eighth) edition of the TNM classification for lung cancer. J Thorac Oncol. (2016) 11:39–51. doi: 10.1016/j.jtho.2015.09.009

PubMed Abstract | CrossRef Full Text | Google Scholar

15. Shalabi LA, Shaaban Z, Kasasbeh B. Data mining: a preprocessing engine. J Comput Sci. (2006) 2:735–9. doi: 10.3844/jcssp.2006.735.739

CrossRef Full Text | Google Scholar

16. Wei R, Wang J, Su M, Jia E, Chen S, Chen T, et al. Missing value imputation approach for mass spectrometry-based metabolomics data. Sci Rep. (2018) 8:663. doi: 10.1038/s41598-017-19120-0

PubMed Abstract | CrossRef Full Text | Google Scholar

17. Zhou XH, Eckert GJ, Tierney WM. Multiple imputation in public health research. Stat Med. (2001) 20:1541–9. doi: 10.1002/sim.689

PubMed Abstract | CrossRef Full Text | Google Scholar

18. Ngiam KY, Khor IW. Big data and machine learning algorithms for health-care delivery. Lancet Oncol. (2019) 20:e262–73. doi: 10.1016/S1470-2045(19)30149-4

PubMed Abstract | CrossRef Full Text | Google Scholar

19. Gonzalez GH, Tahsin T, Goodale BC, Greene AC, Greene CS. Recent advances and emerging applications in text and data mining for biomedical discovery. Brief Bioinform. (2016) 17:33–42. doi: 10.1093/bib/bbv087

PubMed Abstract | CrossRef Full Text | Google Scholar

20. Breiman L, Cutler A. Random forests. Mach Learn. (2001) 45:5–32. Available online at: http://www.stat.berkeley.edu/~breiman/RandomForests/cc_home.htm (accessed June 12, 2011).

Google Scholar

21. Freund Y, Schapire RE. A short introduction to boosting. Jinko Chino Gakkaishi. (1999) 14:771–80. doi: 10.1109/CICC.1996.510579

CrossRef Full Text | Google Scholar

22. Freund Y, Mason L. The alternating decision tree learning algorithm. ICML. (1999) 99:124–33.

Google Scholar

23. Chen T, Guestrin C. XGBoost: A Scalable Tree Boosting System. In: Acm Sigkdd International Conference on Knowledge Discovery & Data Mining. (2016). doi: 10.1145/2939672.2939785

CrossRef Full Text | Google Scholar

24. Vickers AJ, Elkin EB. Decision curve analysis: a novel method for evaluating prediction models. Med Decis Making. (2006) 26:565–74. doi: 10.1177/0272989X06295361

PubMed Abstract | CrossRef Full Text | Google Scholar

25. Jung Y. Multiple predicting K-fold cross-validation for model selection. J Nonparametr Stat. (2018) 30:197–215. doi: 10.1080/10485252.2017.1404598

CrossRef Full Text | Google Scholar

26. Cook JA, Ranstam J. Overfitting. BJS. (2016) 103:1804–14. doi: 10.1002/bjs.10244

PubMed Abstract | CrossRef Full Text | Google Scholar

27. Ginsberg RJ, Rubinstein LV. Randomized trial of lobectomy versus limited resection for T1 N0 non-small cell lung cancer. Lung Cancer Study Group. Ann Thorac Surg. (1995) 60:615–22; discussion 622–3. doi: 10.1016/0003-4975(95)00537-U

PubMed Abstract | CrossRef Full Text | Google Scholar

28. Cadrin-Tourigny J, Bosman LP, Nozza A, Wang W, Tadros R, Bhonsale A, et al. A new prediction model for ventricular arrhythmias in arrhythmogenic right ventricular cardiomyopathy. Eur Heart J. (2019) 40:1850–8. doi: 10.1093/eurheartj/ehz103

PubMed Abstract | CrossRef Full Text | Google Scholar

29. Hijazi Z, Oldgren J, Lindback J, Alexander JH, Connolly SJ, Eikelboom JW, et al. The novel biomarker-based ABC (age, biomarkers, clinical history)-bleeding risk score for patients with atrial fibrillation: a derivation and validation study. Lancet. (2016) 387:2302–11. doi: 10.1016/S0140-6736(16)00741-8

PubMed Abstract | CrossRef Full Text | Google Scholar

30. Pani E, Kennedy G, Zheng X, Ukert B, Jarrar D, Gaughan C, et al. Factors associated with nodal metastasis in 2-centimeter or less non-small cell lung cancer. J Thorac Cardiovasc Surg. (2020) 159:1088–96.e1. doi: 10.1016/j.jtcvs.2019.07.089

CrossRef Full Text | Google Scholar

31. Yu X, Li Y, Shi C, Han B. Risk factors of lymph node metastasis in patients with non-small cell lung cancer ≤ 2 cm in size: A monocentric population-based analysis. Thoracic Cancer. (2018) 9:3–9. doi: 10.1111/1759-7714.12490

PubMed Abstract | CrossRef Full Text | Google Scholar

32. Park HK, Jeon K, Koh WJ, Suh GY, Kim H, Kwon OJ, et al. Occult nodal metastasis in patients with non-small cell lung cancer at clinical stage IA by PET/CT. Respirology. (2010) 15:1179–84. doi: 10.1111/j.1440-1843.2010.01793.x

PubMed Abstract | CrossRef Full Text | Google Scholar

33. Song CY, Kimura D, Sakai T, Tsushima T, Fukuda I. Novel approach for predicting occult lymph node metastasis in peripheral clinical stage I lung adenocarcinoma. J Thorac Dis. (2019) 11:1410–20. doi: 10.21037/jtd.2019.03.57

PubMed Abstract | CrossRef Full Text | Google Scholar

34. Li L, Ren S, Zhang Y, Guan Y, Zhao J, Liu J, et al. Risk factors for predicting the occult nodal metastasis in T1-2N0M0 NSCLC patients staged by PET/CT: potential value in the clinic. Lung Cancer. (2013) 81:213–7. doi: 10.1016/j.lungcan.2013.04.012

PubMed Abstract | CrossRef Full Text | Google Scholar

35. Takamatsu M, Yamamoto N, Kawachi H, Chino A, Saito S, Ueno M, et al. Prediction of early colorectal cancer metastasis by machine learning using digital slide images. Comput Methods Programs Biomed. (2019) 178:155–61. doi: 10.1016/j.cmpb.2019.06.022

PubMed Abstract | CrossRef Full Text | Google Scholar