Application of the Machine-Learning Model to Improve Prediction of Non-Sentinel Lymph Node Metastasis Status Among Breast Cancer Patients

Background: Performing axillary lymph node dissection (ALND) is the current standard option after a positive sentinel lymph node (SLN). However, whether 1–2 metastatic SLNs require ALND is debatable. The probability of metastasis in non-sentinel lymph nodes (NSLNs) can be calculated using nomograms. In this study, we developed an individualized model using machine-learning (ML) methods to select potential variables, which influence NSLN metastasis.

Materials and Methods: Cohorts of patients with early breast cancer who underwent SLN biopsy and ALND between 2012 and 2021 were created (training cohort, N 157 and validation cohort, N 58) for the development of the nomogram. Three ML methods were trained in the training set to create a strong predictive model. Finally, the multiple iterations of the least absolute shrinkage and selection operator regression method were used to determine the variables associated with NSLN status.

Results: Four independent variables (positive SLN number, absence of lymph node hilum, lymphovascular invasion (LVI), and total number of SLNs harvested) were combined to generate the nomogram. The area under the receiver operating characteristic curve (AUC) value of 0.759 was obtained in the entire set. The AUC values for the training set and the test set were 0.782 and 0.705, respectively. The Hosmer-Lemeshow test of the model fit accuracy was identified with p = 0.759.

Conclusion: This study developed a nomogram that incorporates ultrasound (US)-related variables using the ML method and serves to clinically predict the non-metastatic status of NSLN and help in the selection of the appropriate treatment option.

Introduction

In 2020, breast cancer was surpassed lung cancer as the world's most commonly diagnosed cancer. This is despite being arguably negligible in men (1). The most common route of breast cancer metastasis is lymphatic spread within the axilla. Axillary lymph node dissection (ALND) and sentinel lymph node biopsy (SLNB) are the main axillary surgeries for breast cancer (2). ALND could completely remove the metastatic lymph nodes, clarify the TNM stage of the cancer, and inform the prognosis (3). However, ALND can cause many complications, such as lymphedema, hematoma, sensory abnormalities, and limitation of upper limb movement (4, 5). Furthermore, approximately 50% of patients with positive sentinel lymph nodes (SLNs) are found to have no additional nodal metastases (6).

The possibility of exempting ALND in early breast cancer (cT1-2N0) has been widely explored in several clinical trials (7–10). This suggestion has essentially achieved good follow-up data reports. According to St. Gallen guidelines of 2017, women with 1 or 2 positive SLNs who have had breast conservation can avoid ALND and receive whole breast radiation and adjuvant systemic therapies only (11). However, with the limited randomized, multicenter clinical trials and strict inclusion criteria, proper selection of axillary surgeries for patients who fail to meet the criteria has become a priority of many clinicians.

The prediction of the risk of non-sentinel lymph nodes (NSLNs) metastasis determines the selection of axillary surgery. Previous reports show that nomograms have been developed and are validated as the commonly used method of predicting cancer prognosis. The Memorial Sloan-Kettering Cancer Center (MSKCC) nomogram (12) is undoubtedly the most authoritative nomogram. Different cancer centers have validated and confirmed MSKCC nomogram as a robust method of predicting NSLNs metastasis (13, 14).

Multivariate logistic regression is the most common method of incorporating variables into cancer prognostication. Machine learning (ML) is an emerging tool for predicting cancer prognosis that is making significant contributions in different cancer fields (15, 16). It is a learning process, which utilized techniques, such as decision trees (DTs), artificial neural networks (ANNs), and support vector machines (SVMs). Ayer et al. applied the ANN technique in the prediction of breast cancer susceptibility (17) while Zeng et al. predicted the breast cancer recurrence through SVMs (18). Further, Madekivi et al. filtered the variables by a gradient-boosted trees model to develop a final model for predicting NSLNs (19). All these studies show ML as a feasible and superior cancer prediction method. The aim of our study was to employ ML-based statistical methods to select variables with potential influence on NSLN metastasis status. The study ultimately developed an individualized prediction model that could guide clinicians for a better choice of cancer treatment options for different patients.

Method

Patients and Data Collection

The clinical data of patients who underwent surgery between January 2012 and May 2021 at Zhongshan Hospital (an affiliate of Fudan University) and Shanghai Public Health Center (Zhongshan Hospital South Branch) were collected and retrospectively analyzed. A total of 532 patients were screened (n = 532). The inclusion criteria were postoperative pathologically confirmed diagnosis of primary breast cancer, no history of other tumors, and that the patient has received both SLNB and ALND.

The exclusion criteria were lack of preoperative breast ultrasound (US) or pathological information, patients were not preoperatively staging as clinical T1-2N0 or had received neoadjuvant therapy, only axillary surgery without breast tumor resection in our hospital, and a negative pathological result of SLNs. Only 215 patients were retained by the exclusion criteria. The working protocol of our study is as shown in Figure 1. The ethical approval of this study was granted by the ethics committee of Zhongshan Hospital and Shanghai Public Health Center. There was no additional informed consent required from the patients because this was a retrospective study.

FIGURE 1

Figure 1. The working flow of this study.

The SLNs were identified before surgery using Methylene blue dye, and blue-stained nodes were removed and sent for frozen pathology to the Pathology Department. It was stained with H&E and microscopically examined by an experienced pathologist. Routine H&E analysis was performed for all additional nodes identified by ALND.

The collected clinical and medical information of patients included the patient age, breast tumor location, postoperative pathological features [histology, estrogen receptor (ER) status, proliferation index (Ki-67), progesterone receptor (PR) status, human epidermal growth factor receptor 2 (HER-2) overexpression, lymphovascular invasion (LVI), Scarff-Bloom-Richardson (SBR) grade, total number of SLNs harvested, T stage, number of positive SLNs, and number of NSLNs metastasis] and ultrasonic parameters of tumor and axillary lymph nodes [sizes, mass echogenicity, regular or irregular tumor margin, presence or absence of lymph node hilum, and color Doppler flow imaging (CDFI)].

Statistical Analysis

Clinical and pathological variables associated with the risk of lymph node metastasis were assessed on the basis of their clinical importance and predictors identified in previously published articles (20, 21). Categorical variables were reported as integers and proportions. The continuous variables were described as means [±standard deviation (SD)]. Collinearity for all explanatory variables was assessed using a correlation matrix and plausible interaction terms were also tested. Therefore, interaction terms were excluded in the multivariate analysis. To relax the assumption of a linear relationship between continuous predictors and the risk of NSLN metastasis status, continuous predictors that include the patient age, tumor size, and number of SLNs, etc., were categorized after evaluation using restricted cubic splines (22). Regarding the strong U- or S-shaped relation between continuous predictors and NSLNs metastasis results, the value of the turning in the graph was used as the dividing point (Supplementary Figure S1).

Patients were randomly sampled into the training and validation sets in the ratio of 7:3. To select the strongest predictive model, three ML methods were trained in the training set. These ML methods were random forest (RF) (23), SVMs (24), multiple iterations of the least absolute shrinkage, and least absolute shrinkage and selection operator (LASSO) regression (25). The best hyper-parameter for ML models was 10-fold cross-validation to avoid overfitting. The best classification model was selected to compare the performance of the ML methods.

We created a nomogram that could make a linear predictor in patients who were easily accessible to clinicians on the basis of the best-performing model. Further, we assessed the discriminating ability and predictive accuracy in both the validation and entire sets using the ROC curves (C-index and calibration curves) (26, 27). Finally, the decision curve analysis (DCA) was used to support the clinical decisions of the prepared prediction model (28). All the statistical analyses were carried out using the R software (version 3.6.3, http://www.r-project.org). The R-software packages used for statistical data analysis were “caret,” “rms,” “glmnet,” “randomForest,” “e1071,” “kernlab,” “pROC,” “rmda,” and “ResourceSelection.” A two-sided p < 0.05 was considered statistically significant.

Results

Demographic and Clinical Characteristics

Clinicopathological information of 215 patients with breast cancer after surgical operation was assessed between January 2012 and May 2021. Patients who participated were grouped into two groups based on the presence or absence of NSLNs metastasis. The observed clinicopathological characteristics of both metastasis and non-metastasis patients are presented in Table 1. All patients are Asian. The median age of patient was 56 years. The patients with tumors located in the upper-outer quadrant were 51.2%. Largely, all participating patients (97.6%) were at the T1-T2 stages and showed no difference in histology, ER, PR, and ki-67 or HER-2 status. A small proportion of patients in stage T3 were included as the preoperative assessment was T1-T2 but the postoperative pathology confirmed stage T3. However, the ultrasonic features (longitudinal diameter of lymph node, lymphatic echogenicity, and absence of lymph node hilum) and pathological features (LVI, number of positive SLNs, and proportion of positive SLNs) in the two groups were statistically different (p < 0.05).

TABLE 1

Table 1. Differences in clinicopathological characteristics between the patients with and without NSLNs metastasis.

Predictive Model and Factors Selection

All participants were randomly divided into two groups (training and validation cohorts) in the ratio of 7:3. The explanatory variables were transformed into categorical forms. The US transverse and longitudinal diameter of tumor or lymph were highly correlated, so only the largest diameter was retained. No other significant interaction was found. There were no statistical differences between the variables in both the training and validation sets (p > 0.05; Supplementary Table S1). Three ML algorithms were performed in the training set. The best SVM model was obtained when nine candidate variables were selected, as shown in Figure 2A. RF was effective in feature selection and the removal of redundant features. The RF model obtained the highest accuracy (0.689) with five predictive features (Figure 2B). The LASSO could select significantly predictive features but the results may not be identical each time. In this study, we conducted 500 times iterations and selected the features with more than 300 repeated occurrences. Then, these features were sequentially introduced into the logistic regression model to calculate the AUC values. The results showed that the final model with four predictive factors had the highest AUC of 0.705 in the validation set (Figure 2C). Comparisons of the predictive performance of validation sets among the three algorithms models (with each optimal variable and tuning parameter) are shown in Figure 2D and Supplementary Table S2. It turned out that the LASSO regression model demonstrated the highest performance. In detail, the relative weights of the final variables in the LASSO-based logistic regression model are displayed in Figure 2E.

FIGURE 2

Figure 2. Predictive model and factors selection. (A) The line graph shows the relationship between the number of candidate features and the accuracy in support vector machines (SVMs) model. (B) The line graph shows the relationship between the number of selected features and the accuracy in random forest (RF) model. (C) The line graph shows the relationship between the number of features and the area under the curve (AUC) values in the least absolute shrinkage and selection operator (LASSO)-based logistical model. (D) ROC curve analysis of machine-learning algorithms for prediction of non-sentinel lymph nodes (NSLN) without metastasis in the validation set. (E) The dot plot shows the coefficients of variables in the final model. LVI: lymphovascular invasion; totalSLN: total number of SLNs harvested; LnHilum: absence of lymph node hilum; SLNp: number of positive SLNs.

Nomograms and Model Performance

The four independent factors used to create a predictive nomogram were the number of positive SLNs (1–2, 3–4, or ≥5), the total number of SLNs harvested (≤2, 3–5, or ≥6), absence of lymph node hilum (no/not described or yes), and LVI (no or yes). According to the sum of the assigned points for each factor in the nomogram, a higher total score was associated with the absence of NSLN metastasis (Figure 3). The c-index in the logistic regression was equal to the area under the ROC curve. In Figure 4A, an AUC value of 0.759 is achieved in the entire set, while AUC values of 0.782 and 0.705 are obtained in the training and validation sets, respectively. The Hosmer-Lemeshow test was used to assess the accuracy of model fit and no departure from perfect fit was identified (p = 0.759). The sample bootstrapped calibration plot for the prediction is also presented in Figure 4B.

FIGURE 3

Figure 3. Nomogram for prediction of the absence of non-sentinel lymph nodes (NSLN) metastasis. SLNp: number of positive SLNs; LVI: lymphovascular invasion; LnHilum: absence of lymph node hilum; totalSLN: total number of SLNs harvested.

FIGURE 4

Figure 4. Nomogram prediction model validation. (A) ROC curves are used for all sets. AUC values for training set (red), validation set (green), and entire set (blue) are 0.782, 0.705, and 0.759. (B) The bootstrapped calibration plot and Hosmer-Lemeshow test for the training set.

Clinical Application Evaluation

Decision curve analysis showed that using this nomogram provides an additional benefit when the threshold probability of the entire set is between 0 and 87% (Figure 5). A similar observation was also reported in entire and test cohorts. Therefore, the nomogram model can predict the probability of NSLN metastasis in patients with breast cancer to facilitate early clinical intervention and support personalized postoperative cancer rehabilitation.

FIGURE 5

Figure 5. The decision curve analysis (DCA) curve shows the decision analysis for the entire set, the training set, and the validation set.

Discussion

The axillary surgery has been de-escalating since the awareness grows in breast cancer (29). Systemic treatments, such as chemotherapy therapies, are recognized as important control measures of cancer recurrence rather than the local therapy, such as the extent of surgical excision. The National Surgical Adjuvant Breast and Bowel Project (NSABP) b-32 trial (7) first investigated the necessity of ALND in patients with negative SLN. This trial found that SLNB alone without further axillary dissection is an appropriate therapy for the targeted patients. This report has been widely accepted in clinical practice to the point where it has become a surgical routine. Nevertheless, breast surgeons still explore the surgical indications. The After Mapping of the Axilla, Radiotherapy or Surgery (AMAROS) trial (8) revealed that ALND and axillary radiotherapy after a positive SLN provide comparable axillary control for patients with early breast cancer. The American College of Surgeons Oncology Group (ACOSOG) Z0011 (9) suggested that early breast cancer patients with 1 or 2 SLN metastases, breast-conserving lumpectomy, and whole-breast irradiation can be exempted from ALND. Furthermore, International Breast Cancer Study Group (IBCSG) 23-01 (10) trial argued the necessity of ALND in patients with micrometastatic SLN (metastases <2 mm). The discussion for sparing ALND is rooted in resultant series of complications that include range of motion, lymphoedema, pain, and sensory defects (4, 5, 30). In contrast, SLNB can significantly lower the morbidity of such complications (31). Furthermore, clinical data indicated that the majority of patients with positive SLNs had no additional nodes metastasis (6), which is consistent with our finding that roughly 55% of all patients had no metastasis. Despite the various clinical trials that explored the necessity of ALND, it is still apparent that the inclusion of patients is relatively stringent and the precise individualization of the choice is still pondering.

Previous studies have been conducted to predict NSLN metastasis (12, 20, 32–36). The MSKCC model is a widely acknowledged tool that incorporates eight variables (12). The variables combined in the MSKCC model were pathological size, ER status, multifocality, tumor type, tumor nuclear grade, LVI, method of detection, and the number of positive and negative SLNs with an AUC of 0.77 for the validation cohort. That the number of positive SLNs in the MSKCC model had the highest weight that is consistent with the findings of our study. The MSKCC model has been validated in various countries, for instance, in Australia where an AUC of 0.66 was obtained from the inclusion of 526 patients (14). The model of MD Anderson Cancer Center (33) is another frequently mentioned model. This model added two variables of SLN metastasis size and extracapsular extension, which are tied with the emphasis on SLN micrometastasis status. These variables were excluded in this study because some patients did not have SLN metastatic size as a result of the limitations of our pathology department. In contrast, the Helsinki University model (20) included a prediction variable of HER-2 status instead of PR status. Despite the fact that HER-2 positive is generally associated with NSLN positivity, the relationship between HER-2 status and NSLN positivity remains controversial (37), and the current study yielded no statistically significant differences.

Preoperative assessment of axillary lymph burden in breast cancer is routinely performed using different imaging techniques. US is considered to be the most recommended imaging technique owing to its inexpensive, convenience, and absence of radiation exposure. The Sentinel Node vs. Observation after Axillary Ultrasound (SOUND) trial (38) is exploring the potential possibility of US as a replacement for SLNB. Previous studies have also suggested that the inclusion of US parameters in the model could improve its predictive capacity (39, 40). Zhu (40) suggested that the Doppler resistance index and the extent of extension of the enhancing lesion were correlated with lymph node metastasis. Qiu et al. (39), on the other hand, incorporated three US-based variables of cortical thickness of SLN, transverse diameter of SLN, and lymph node hilum status in their nomogram, with an AUC of 0.864. This is consistent with the findings obtained in this study. However, the validity of the model is challenged by its dependence on adjustable parameters by the operator.

Apart from predicting NSLN status based on imaging features, such as ultrasound or clinical features, the role of molecular markers has also been explored. Metalloprotease-1 (41) and cytokeratin 19 mRNA copy (42, 43) have been suggested to be highly correlated with NSNL metastasis. Prediction models based on molecular markers usually showed a high specificity and sensitivity, but the time taken for intraoperative measurements and the high cost may be the reasons why they are difficult to extend.

Our nomogram provides an individual prediction of the probability of having negative NSLNs for patients with positive SLNs. A woman with 2 positive SLNs, out of 5 SLNs harvested, the preoperative US showed the presence of lymph node hilum and pathology revealed tumor LVI, might be considered to have a 20% risk of having negative NSLNs, which implies that the patient is at high risk of additional nodes metastasis and ALND should be recommended clinically. However, the study result is limited and requires much more validation before it can be applied to clinical reasoning.

Machine learning has been applied to different tumors as an innovative method for cancer prediction and prognosis (15, 16). It provides excellent accuracy through a continuous ML approach. According to Ayer et al. (17), a prediction model that enrolled 48,774 patients yielded an AUC of 0.965 and could distinguish malignant from benign mammographic findings. Such precision is unfathomable in alternative models. Typically, an AUC above 0.7 is regarded as reasonable. Madekivi et al. (19) narrowed down to seven variables by utilizing XG-Boost's capabilities for self-learning and eventually developed a nomogram with an AUC of 0.80. This study compared the performance of three ML methods, and the best classification model was selected. To reduce the number of variables, LASSO conducted 500 times iterations and selected the features with more than 300 repeated occurrences. This showed a well predictive effectiveness in the validation cohort.

The inclusion of US-related variables and the application of the ML approach are the two aspects of innovation in our prediction model. Eventually, the four variables included in the development of the nomogram were the number of positive SLNs, total number of SLNs harvested, LVI, and lymph node hilum status. The AUC values of 0.759 and 0.705 were used for the training and validation cohorts, respectively. Therefore, in comparison with other previous studies, the validation cohort in this study had a higher AUC value for the fewer variables. The continuous predictors were categorized using restricted cubic splines, which are better suited for the daily practice of clinicians. However, some limitations of the study were noted. First, the number of patients enrolled in this study is far from satisfaction. The inclusion criteria entailed only the patient who received both SLNB and ALND and this dramatically reduced the number. The nomogram lacked external validation due to the limitation in numbers, which meant that patients had to be studied as a single center although we included patients in multicenter. This study was short of a large sample that is technically required for ML to support a more convincing result. On the other hand, our pathology department was failed to accurately depict the size of SLN metastases at the beginning, owing to the inclusion of patients over a long time interval. This was significantly regretted because the existing literature already supports a relationship between SLN metastasis size and NSLN status.

Overall, the trend of de-escalation of axillary surgery is inevitable. However, the selection of appropriate patients remains a crucial issue to be addressed. It is essential to have some multicenter, prospective trials, such as Z0011, to upgrade the guidelines. This study revealed more areas that need future research to validate our findings.

Data Availability Statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Author Contributions

QW collected the data and was a major contributor to writing the manuscript. LD analyzed and interpreted the patient data. YJ made substantial contributions to the conception. HZ designed the work and revised it. All authors contributed to the article and approved the submitted version.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fsurg.2022.797377/full#supplementary-material

Supplementary Figure S1. The line graph shows the relationship between the continuous variables and the risk of non-sentinel lymph nodes (NSLNs) metastasis status by using restricted cubic splines analysis. Y-axis means risk ratio and the X-axis means values of variables. (A) age; (B) tumor color Doppler flow imaging (CDFI); (C) total number of SLNs harvested; (D) number of positive SLNs; (E) longitudinal/transverse axis ratio of tumor; (F) longitudinal/transverse axis ratio of lymph nodes; (G) proportion of positive SLNs; (H) transverse axis of tumor; (I) transverse axis of lymph nodes.

Supplementary Table S1. Differences in clinicopathological characteristics between the training and validation sets.

Supplementary Table S2. Comparisons of the predictive performance of validation sets among the three algorithms models.

References

1. Sung H, Ferlay J, Siegel RL. Global cancer statistics 2020: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA Cancer J Clin. (2021) 71:209–49. doi: 10.3322/caac.21660

PubMed Abstract | CrossRef Full Text | Google Scholar

2. Veronesi U, Paganelli G, Viale G. A randomized comparison of sentinel-node biopsy with routine axillary dissection in breast cancer. N Engl J Med. (2003) 349:546–53. doi: 10.1056/NEJMoa012782

PubMed Abstract | CrossRef Full Text | Google Scholar

3. Magnoni F, Galimberti V, Corso G. Axillary surgery in breast cancer: an updated historical perspective. Semin Oncol. (2020) 47:341–52. doi: 10.1053/j.seminoncol.2020.09.001

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Soares EW, Nagai HM, Bredt LC. Morbidity after conventional dissection of axillary lymph nodes in breast cancer patients. World J Surg Oncol. (2014) 12:67. doi: 10.1186/1477-7819-12-67

PubMed Abstract | CrossRef Full Text | Google Scholar

5. Dinas K, Kalder M, Zepiridis L. Axillary web syndrome: incidence, pathogenesis, and management. Curr Probl Cancer. (2019) 43:100470. doi: 10.1016/j.currproblcancer.2019.02.002

PubMed Abstract | CrossRef Full Text | Google Scholar

6. Abdessalam SF, Zervos EE, Prasad M. Predictors of positive axillary lymph nodes after sentinel lymph node biopsy in breast cancer. Am J Surg. (2001) 182:316–20. doi: 10.1016/S0002-9610(01)00719-X

PubMed Abstract | CrossRef Full Text | Google Scholar

7. Krag DN, Anderson SJ, Julian TB. Sentinel-lymph-node resection compared with conventional axillary-lymph-node dissection in clinically node-negative patients with breast cancer: overall survival findings from the NSABP B-32 randomised phase 3 trial. Lancet Oncol. (2010) 11:927–33. doi: 10.1016/S1470-2045(10)70207-2

PubMed Abstract | CrossRef Full Text | Google Scholar

8. Donker M, van Tienhoven G, Straver ME. Radiotherapy or surgery of the axilla after a positive sentinel node in breast cancer (EORTC 10981-22023 AMAROS): a randomised, multicentre, open-label, phase 3 non-inferiority trial. Lancet Oncol. (2014) 15:1303–10. doi: 10.1016/S1470-2045(14)70460-7

PubMed Abstract | CrossRef Full Text | Google Scholar

9. Giuliano AE, Ballman KV, McCall L. Effect of axillary dissection vs no axillary dissection on 10-year overall survival among women with invasive breast cancer and sentinel node metastasis: the ACOSOG Z0011 (Alliance) randomized clinical trial. JAMA. (2017) 318:918–26. doi: 10.1001/jama.2017.11470

PubMed Abstract | CrossRef Full Text | Google Scholar

10. Galimberti V, Cole BF, Viale G. Axillary dissection versus no axillary dissection in patients with breast cancer and sentinel-node micrometastases (IBCSG 23-01): 10-year follow-up of a randomised, controlled phase 3 trial. Lancet Oncol. (2018) 19:1385–93.

PubMed Abstract | Google Scholar

11. Curigliano G, Burstein HJ, Winer EP. De-escalating and escalating treatments for early-stage breast cancer: the St. Gallen International Expert Consensus Conference on the Primary Therapy of Early Breast Cancer 2017. Ann Oncol. (2017) 28:1700–12. doi: 10.1093/annonc/mdx308

PubMed Abstract | CrossRef Full Text | Google Scholar

12. Van Zee KJ, Manasseh DM, Bevilacqua JL. A nomogram for predicting the likelihood of additional nodal metastases in breast cancer patients with a positive sentinel node biopsy. Ann Surg Oncol. (2003) 10:1140–51. doi: 10.1245/ASO.2003.03.015

PubMed Abstract | CrossRef Full Text | Google Scholar

13. Nadeem RM, Gudur LD, Saidan ZA. An independent assessment of the 7 nomograms for predicting the probability of additional axillary nodal metastases after positive sentinel lymph node biopsy in a cohort of British patients with breast cancer. Clin Breast Cancer. (2014) 14:272–9. doi: 10.1016/j.clbc.2014.02.006

PubMed Abstract | CrossRef Full Text | Google Scholar

14. Tapia G, Ying V, Di Re A. Predicting non-sentinel lymph node metastasis in Australian breast cancer patients: are the nomograms still useful in the post-Z0011 era? Anz J Surg. (2019) 89:712–7. doi: 10.1111/ans.15173

PubMed Abstract | CrossRef Full Text | Google Scholar

15. Stojadinovic A, Nissan A, Eberhardt J. Development of a bayesian belief network model for personalized prognostic risk assessment in colon carcinomatosis. Am Surg. (2011) 77:221–30. doi: 10.1177/000313481107700225

PubMed Abstract | CrossRef Full Text | Google Scholar

16. Lee JH, Ha EJ, Kim D. Application of deep learning to the diagnosis of cervical lymph node metastasis from thyroid cancer with CT: external validation and clinical utility for resident training. Eur Radiol. (2020) 30:3066–72. doi: 10.1007/s00330-019-06652-4

PubMed Abstract | CrossRef Full Text | Google Scholar

17. Ayer T, Alagoz O, Chhatwal J. Breast cancer risk estimation with artificial neural networks revisited: discrimination and calibration. Cancer-Am Cancer Soc. (2010) 116:3310–21. doi: 10.1002/cncr.25081

PubMed Abstract | CrossRef Full Text | Google Scholar

18. Zeng Z, Espino S, Roy A. Using natural language processing and machine learning to identify breast cancer local recurrence. BMC Bioinformatics. (2018) 19:498. doi: 10.1186/s12859-018-2466-x

PubMed Abstract | CrossRef Full Text | Google Scholar

19. Madekivi V, Boström P, Karlsson A. Can a machine-learning model improve the prediction of nodal stage after a positive sentinel lymph node biopsy in breast cancer? Acta Oncol. (2020) 59:689–95. doi: 10.1080/0284186X.2020.1736332

PubMed Abstract | CrossRef Full Text | Google Scholar

20. Meretoja TJ, Leidenius MH, Heikkilä PS. International multicenter tool to predict the risk of nonsentinel node metastases in breast cancer. J Natl Cancer Inst. (2012) 104:1888–96. doi: 10.1093/jnci/djs455

PubMed Abstract | CrossRef Full Text | Google Scholar

21. Al-Hilli Z, Hieken TJ, Boughey JC. Axillary ultrasound in the management of the newly diagnosed breast cancer patient. Breast J. (2015) 21:634–41. doi: 10.1111/tbj.12497

PubMed Abstract | CrossRef Full Text | Google Scholar

22. Harrell FJ, Lee KL, Mark DB. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med. (1996) 15:361–87. doi: 10.1002/(SICI)1097-0258(19960229)15:4<361::AID-SIM168>3.0.CO;2-4

PubMed Abstract | CrossRef Full Text | Google Scholar

23. Buxton RT, McKenna MF, Clapp M. Efficacy of extracting indices from large-scale acoustic recordings to monitor biodiversity. Conserv Biol. (2018) 32:1174–84. doi: 10.1111/cobi.13119

PubMed Abstract | CrossRef Full Text | Google Scholar

24. Huang ML, Hung YH, Lee WM. SVM-RFE based feature selection and Taguchi parameters optimization for multiclass SVM classifier. Sci World J. (2014) 2014:795624. doi: 10.1155/2014/795624

PubMed Abstract | CrossRef Full Text | Google Scholar

25. Sveen A, Ågesen TH, Nesbakken A. ColoGuidePro: a prognostic 7-gene expression signature for stage III colorectal cancer patients. Clin Cancer Res. (2012) 18:6001–10. doi: 10.1158/1078-0432.CCR-11-3302

PubMed Abstract | CrossRef Full Text | Google Scholar

26. Harrell FJ, Califf RM, Pryor DB. Evaluating the yield of medical tests. JAMA. (1982) 247:2543–6. doi: 10.1001/jama.247.18.2543

CrossRef Full Text | Google Scholar

27. Kramer AA, Zimmerman JE. Assessing the calibration of mortality benchmarks in critical care: the hosmer-lemeshow test revisited. Crit Care Med. (2007) 35:2052–6. doi: 10.1097/01.CCM.0000275267.64078.B0

PubMed Abstract | CrossRef Full Text | Google Scholar

28. Van Calster B, Wynants L, Verbeek J. Reporting and interpreting decision curve analysis: a guide for investigators. Eur Urol. (2018) 74:796–804. doi: 10.1016/j.eururo.2018.08.038

PubMed Abstract | CrossRef Full Text | Google Scholar

29. Yan M, Abdi MA, Falkson C. Axillary management in breast cancer patients: a comprehensive review of the key trials. Clin Breast Cancer. (2018) 18:e1251–9. doi: 10.1016/j.clbc.2018.08.002

PubMed Abstract | CrossRef Full Text | Google Scholar

30. Roses DF, Brooks AD, Harris MN. Complications of level I and II axillary dissection in the treatment of carcinoma of the breast. Ann Surg. (1999) 230:194–201. doi: 10.1097/00000658-199908000-00009

PubMed Abstract | CrossRef Full Text | Google Scholar

31. Killelea BK, Long JB, Dang W. Associations between sentinel lymph node biopsy and complications for patients with ductal carcinoma in situ. Ann Surg Oncol. (2018) 25:1521–9. doi: 10.1245/s10434-018-6410-0

PubMed Abstract | CrossRef Full Text | Google Scholar

32. Degnim AC, Reynolds C, Pantvaidya G. Nonsentinel node metastasis in breast cancer patients: assessment of an existing and a new predictive nomogram. Am J Surg. (2005) 190:543–50. doi: 10.1016/j.amjsurg.2005.06.008

PubMed Abstract | CrossRef Full Text | Google Scholar

33. Mittendorf EA, Hunt KK, Boughey JC. Incorporation of sentinel lymph node metastasis size into a nomogram predicting nonsentinel lymph node involvement in breast cancer patients with a positive sentinel lymph node. Ann Surg. (2012) 255:109–15. doi: 10.1097/SLA.0b013e318238f461

PubMed Abstract | CrossRef Full Text | Google Scholar

34. Pal A, Provenzano E, Duffy SW. A model for predicting non-sentinel lymph node metastatic disease when the sentinel lymph node is positive. Br J Surg. (2008) 95:302–9. doi: 10.1002/bjs.5943

PubMed Abstract | CrossRef Full Text | Google Scholar

35. Kohrt HE, Olshen RA, Bermas HR. New models and online calculator for predicting non-sentinel lymph node status in sentinel lymph node positive breast cancer patients. BMC Cancer. (2008) 8:66. doi: 10.1186/1471-2407-8-66

PubMed Abstract | CrossRef Full Text | Google Scholar

36. Wang XY, Wang JT, Guo T. Risk factors and a predictive nomogram for non-sentinel lymph node metastases in Chinese breast cancer patients with one or two sentinel lymph node macrometastases and mastectomy. Curr Oncol. (2019) 26:e210–5. doi: 10.3747/co.26.4295

PubMed Abstract | CrossRef Full Text | Google Scholar

37. Lale A, Yur M, Özgül H. Predictors of non-sentinel lymph node metastasis in clinical early stage (cT1-2N0) breast cancer patients with 1-2 metastatic sentinel lymph nodes. Asian J Surg. (2020) 43:538–49. doi: 10.1016/j.asjsur.2019.07.019

PubMed Abstract | CrossRef Full Text | Google Scholar

38. Gentilini O, Veronesi U. Abandoning sentinel lymph node biopsy in early breast cancer? A new trial in progress at the European Institute of Oncology of Milan (SOUND: Sentinel node vs Observation after axillary UltraSouND). Breast. (2012) 21:678–81. doi: 10.1016/j.breast.2012.06.013

PubMed Abstract | CrossRef Full Text | Google Scholar

39. Qiu SQ, Zeng HC, Zhang F. A nomogram to predict the probability of axillary lymph node metastasis in early breast cancer patients with positive axillary ultrasound. Sci Rep. (2016) 6:21196. doi: 10.1038/srep21196

PubMed Abstract | CrossRef Full Text | Google Scholar

40. Zhu Y, Lv W, Wu H. A preoperative nomogram for predicting the risk of sentinel lymph node metastasis in patients with T1-2N0 breast cancer. Jpn J Radiol. (2022). doi: 10.1007/s11604-021-01236-z. [Epub ahead of print].

PubMed Abstract | CrossRef Full Text | Google Scholar

41. Eiró N, González LO, Atienza S. Prediction of metastatic breast cancer in non-sentinel lymph nodes based on metalloprotease-1 expression by the sentinel lymph node. Eur J Cancer. (2013) 49:1009–17. doi: 10.1016/j.ejca.2012.09.019

PubMed Abstract | CrossRef Full Text | Google Scholar

42. Pina H, Salleron J, Gilson P. Intraoperative prediction of non-sentinel lymph node metastases in breast cancer using cytokeratin 19 mRNA copy number: a retrospective analysis. Mol Clin Oncol. (2022) 16:58. doi: 10.3892/mco.2022.2491

PubMed Abstract | CrossRef Full Text | Google Scholar

43. Shimazu K, Miyake T, Okuno J. One-step nucleic acid amplification can identify sentinel node-negative breast cancer patients with excellent prognosis. Anticancer Res. (2019) 39:1447–54. doi: 10.21873/anticanres.13261

PubMed Abstract | CrossRef Full Text | Google Scholar