Multiparametric Microstructural MRI and Machine Learning Classification Yields High Diagnostic Accuracy in Amyotrophic Lateral Sclerosis: Proof of Concept

The potential of multiparametric quantitative neuroimaging has been extensively discussed as a diagnostic tool in amyotrophic lateral sclerosis (ALS). In the past, the integration of multimodal, quantitative data into a useful diagnostic classifier was a major challenge. With recent advances in the field, machine learning in a data driven approach is a potential solution: neuroimaging biomarkers in ALS are mainly observed in the cerebral microstructure, with diffusion tensor imaging (DTI) and texture analysis as promising approaches. We set out to combine these neuroimaging markers as age-corrected features in a machine learning model with a cohort of 502 subjects, divided into 404 patients with ALS and 98 healthy controls. We calculated a linear support vector classifier (SVC) which is a very robust model and then verified the results with a multilayer perceptron (MLP)/neural network. Both classifiers were able to separate ALS patients from controls with receiver operating characteristic (ROC) curves showing an area under the curve (AUC) of 0.87–0.88 (“good”) for the SVC and 0.88–0.91 (“good” to “excellent”) for the MLP. Among the coefficients of the SVC, texture data contributed the most to a correct classification. We consider these results as a proof of concept that demonstrated the power of machine learning in the application of multiparametric quantitative neuroimaging data to ALS.

Introduction

Amyotrophic lateral sclerosis (ALS) is a clinically and genetically heterogeneous, multidomain neurodegenerative syndrome of motor and extra-motor systems with multiple different clinical subphenotypes and alterations in several brain regions, most prominently the corticospinal tract (CST) (1). The potential of multiparametric, quantitative magnetic resonance imaging (MRI) in the diagnostic procedures of ALS is widely recognized (2, 3). The development of current machine learning algorithms introduced the opportunity to combine biomarkers from different MRI metrics into a single classifier, even in more complex settings. Previous approaches using resting state functional MRI (rs-fMRI), T1 weighted imaging (T1w), and diffusion tensor imaging (DTI) data have achieved diagnostic accuracy of well over 70% (4–6), recently summarized in a systematic review (7). The challenge in designing a suitable machine learning model remains the appropriate selection of features and the collection of a meaningful amount of data to build a reliable model (8). In addition to traditional techniques, texture analysis is among the most promising approaches for disease classification in ALS (9, 10). The combination of texture analysis with traditional diffusion metrics makes a multiparametric microstructural assessment possible that might enhance diagnostic accuracy in machine learning classifiers. In the present study, a retrospective data analysis with T1w and DTI data was conducted. To this end, we extracted diffusion metrics of the most important tracts in ALS as well as texture data from the motor segment of the corpus callosum. We combined these data in a linear support vector classifier (SVC) which is a robust model that can give feedback on feature importance, with the prospect of providing a proof of concept model with high accuracy and unraveling underlying patterns that could help to build future classifiers. Furthermore, we used a state-of-the-art artificial neural network to verify the results. As a proof of concept study, we primarily focused on the general feasibility of the approach, not the optimization of the ML algorithm.

Materials and Methods

Data Collection and Preprocessing

The data were collected from the MRI data archive of the Dept of Neurology, University of Ulm, Germany and included 1.5 T imaging data sets that contained a high-resolution T1w sequence and also a DTI sequence with at least 39 gradients. The search resulted in 404 data sets from ALS patients (237 male; mean age 63 ± 11 years) and 98 data sets from healthy control subjects (49 male; mean age 57 ± 16 years), recorded between May 2010 and February 2021 (Table 1). Magnetic resonance imaging scanning of all data sets was performed on the same 1.5 Tesla Magnetom Symphony (Siemens Medical, Erlangen, Germany); the study protocol consisted of a T1w scan with 144 slices [256 × 256 pixels, slice thickness 1.2 mm, pixel size 1.0 × 1.0 mm, echo time (TE) 4.2 ms, repetition time (TR) 1,640 ms] and a DTI study protocol with 52 volumes (64 slices, 128 × 128 pixels, slice thickness 2.8 mm, pixel size 2.0 × 2.0 mm, 39 control subjects, 306 ALS subjects), 48 gradient directions (b = 1,000 s/mm²), and four scans with b = 0, TE = 95 ms, TR = 8,000 ms (DTI protocol A). Alternatively, DTI protocol B consisted of 39 gradients including three b0 gradient directions (b = 1,000 s/mm², voxel size 2.0 × 2.0 × 2.8 mm, 128 × 128 × 64 matrix, TE = 95 ms, TR = 8,000 s, 0 control subjects, 33 ALS subjects); DTI protocol C consisted of 2 × 31 gradient directions including two b0 gradient directions (b = 1,000 s/mm², voxel size 2.5 × 2.5 × 2.5 mm, 64 slices, 128 ×128 in-plane matrix, TE = 102 ms, TR = 8,700 ms, 59 control subjects, 62 ALS subjects). The study was performed according to institutional guidelines in accordance with the Declaration of Helsinki and was approved by the Ethical Committee of the University of Ulm (reference # 19/12).

TABLE 1

Table 1. Demographic and clinical features of the subjects.

The software Tensor Imaging and Fiber Tracking (TIFT) was used for data analysis (12). Prior to data analysis, data underwent a standardized quality control. The DTI data were stereotaxically normalized to the Montreal Neurological Institute (MNI) stereotaxic space using a group-specific template (13); in a consecutive step, fractional anisotropy (FA) maps were corrected for the covariate age (14). For the harmonization of FA differences resulting from different acquisition protocols, FA maps were harmonized according to a previously reported protocol by a linear first order correction (15) and have been successfully applied to DTI data even from different scanners (16, 17). Finally, an 8 mm (FWHM) Gaussian filter was applied for smoothing of FA maps in order to achieve a good balance between sensitivity and specificity (13). For tractwise fractional anisotropy statistics (TFAS), the following tracts of interest (TOI) were isolated (18): the CST, the corticopontine tract, the corticorubral tract, the corticostriatal pathway, the proximal portion of the perforant path, and tracts originating in segments II and III of the corpus callosum (19–21). Using a threshold of 0.2, the mean FA-value within each TOI was calculated (22). The texture data calculation of the corpus callosum was described in detail in a previous study (10). In brief, after alignment to the AC-PC line (anterior and posterior commissure) and rigid brain transformation, the corpus callosum was automatically segmented from median and paramedian sagittal T1w slices. For the segment III, the texture was analyzed and several texture parameters were calculated, notably texture homogeneity and entropy, each age-corrected. The texture and the FA data were registered for consecutive machine learning operations.

Machine Learning Classifiers

According to previous studies, the following variables were defined as features in the applied machine learning classifiers (10, 19): age-corrected mean FA of the CST, the corticopontine tract, the corticorubral tract, the corticostriatal pathway, the proximal portion of the perforant path, and tracts originating in the segments II and III of the corpus callosum, as well as age-corrected texture homogeneity and entropy from segment III of the corpus callosum. Age-correction was important, due to significant differences between groups (see Table 1). For the diffusion metrics, FA was exclusively chosen out of the DTI metrics, since it has been identified as the most robust DTI-based parameter, according to a recent systematic review (7). All parameters were z-transformed prior to further analyses. For the dichotomous classification ALS patients vs. controls, two output classes were defined.

Two machine learning classification models were employed, i.e., first, a linear support vector machine (SVM) (23), which is a very robust model that also allowed us to analyze the linear coefficients of the model for later interpretation and second, a multilayer perceptron (MLP) classifier (24) in order to verify the results from the SVM. Each hyperparameter was chosen a priori. In the majority of the hyperparameters, this was done by using default values or referring to prior work (24). Choosing the hyperparameters a priori was a design choice, as we focused on the integration of multiparametric MRI data, not on optimizing or comparing ML models. This approach also had the consequence that the hyperparameters were not fitted to the validation set, except for the hidden layer sizes in the MLP. Calculations were done using the scikit-learn 0.23.2 library for python (25). To assess the diagnostic power of our models, receiver operating characteristics (ROC) curves were used to calculate the area under the curve (AUC) (26, 27).

Linear Support Vector Machine

For the SVM, we decided to use the “svc” estimator from the “svm” class with the following (hyper) parameters: kernel = “linear” for a linear SVM, class_weight = “balanced” to account for the sample size imbalance, and max_iter = −1 to ensure convergence during training (19, 21). For all other parameters, the default value was used, including the regularization parameter C = 1. The class weights were implemented as coefficients to the regularization parameter C for each class. This way, inaccurate classification for the smaller control group were penalized more heavily than for the larger ALS group, addressing the issue of imbalanced class sizes. By implementing class weights, we favored the benefit of having more examples over the downside of class imbalance, in the case of the SVM. The model was fit using all 404 ALS subjects and all 98 control subjects. For validation, a leave-one-out cross-validation (LOOCV) was performed (23, 25). For the interpretation of the input parameter weights, the “coef_” attribute was called (25). The entire procedure was repeated for each MRI modality, effectively resulting in three models, i.e., one with all features, one with only FA data, and one with only texture data.

Multilayer Perceptron

To verify the results from the SVM classifier, the MLPClassifier estimator from the “neural-network” class was chosen. We modeled this estimator in close resemblance to a previous model by van der Burgh et al. (24), using the following hyperparameters: activation = “logistic” for a logistic unit activation function, alpha = 0.1 to set the L2 penalty parameter to 0.1 in order to prevent overfitting, and max_iter = 1,000 to ensure convergence during training. The hidden layer sizes [hidden_layer_sizes = (23,23)] were determined by an exhaustive grid search within a 1–500 interval. For all other parameters, the default value was used. Because the MLPClassifier struggles with unbalanced sample sizes, we undersampled the ALS subject group at random to match the control group in size (n = 98). The model was then fit using these 98 ALS subjects and all 98 control subjects. For validation, a LOOCV was performed. Finally, the model was tested on the ALS patients' data that had been excluded during the undersampling process (n = 306, referred to as “holdout test sample”) (see Figure 1).

FIGURE 1

Figure 1. (A) Training, validation, and test samples in the multilayer perceptron classifier. The ALS data were undersampled for the training and validation samples. The leftover data were used as a holdout test sample. (B) Graphical representation of the multilayer perceptron/neural network. There were nine features in the input layer and two classes in the output layer (ALS, controls). The 23 nodes in the two hidden layers were determined by an extensive grid search.

Results

Linear Support Vector Machine

Within the training sample, the linear SVC achieved 81% sensitivity and 82% specificity with an AUC of 88% in the ROC analysis (Figure 2A). The LOOCV analysis confirmed these results with 80% sensitivity, 80% specificity, and an AUC of 87% in the ROC analysis. Each modality (FA maps, texture data) on its own underperformed the combined model (Table 2). Analysis of the (linear) coefficients demonstrated that the corpus callosum texture homogeneity contributed the most to accurate classification, followed by the CST FA and corpus callosum FA (Table 3).

FIGURE 2

Figure 2. (A) ROC curve and AUC from the support vector classifier. Left, training sample; right, leave-one-out cross-validation. (B) ROC curve and AUC from the multilayer perceptron classifier. Left, training sample; right, leave-one-out cross-validation.

TABLE 2

Table 2. Sensitivity and specificity of the linear support vector classifier.

TABLE 3

Table 3. Linear coefficients of the support vector classifiers.

Multilayer Perceptron

Within the training sample, the MLP classifier achieved 81% sensitivity and 86% specificity with an AUC of 0.91 (“excellent”) in the ROC analysis (Figure 2B). The LOOCV confirmed these results with 79% sensitivity, 84% specificity, and an AUC of 0.88 (“good”) in the ROC analysis. In the holdout test-sample (Figure 1A), the MLP classifier reached 72% sensitivity.

Discussion

The present study demonstrated the potential of a multiparametric approach in the diagnostic classification of ALS, as a proof of concept.

Classifiers

The SVM classifier delivered the most robust results with 80–81% sensitivity, 80–82% specificity, and an AUC of 0.87–0.88 (“good”) in the ROC analysis. Analyses of the coefficients revealed that texture homogeneity of the callosal area III was most important in distinguishing ALS from controls, followed by CST FA and corpus callosum FA. The high diagnostic accuracy of texture data alone was surprising, as we expected diffusion metrics to be superior in describing neurodegeneration in ALS, especially in the CST, as demonstrated in a study using a Random Forest approach to assess the clinical utility of DTI by combining FA, mean diffusivity (MD), and radial diffusivity (RD) measured along tracts between the cerebral peduncle and the corona radiata, reaching a mean five-fold cross validation accuracy of 80% in discriminating ALS from controls (28). Yet, prior studies reported similar results using texture (10) and DTI data (29). In addition, we noticed that the stage-defining tracts besides the CST did not contribute much to the classification, especially when texture data were present. Since adding a feature whose information is already (fully) represented by another does not increase accuracy, some DTI information might be redundant in the use of diffusion data of the CST or the corpus callosum or texture data. In a previous approach, Fekete and co-workers have addressed multicollinearity by using a multiple kernel learning (MKL) approach and implementing a nifty optimization method for the weights of the kernel matrix sum (30). Also, late-stage tracts will most probably only be affected in a subset of ALS patients—this might be why these data did not distinguish well-enough between patients and controls in our model (31). Pruning these features in future models might increase performance, as the subject to feature ratio increases without necessarily losing predictive power. The neural network classifier achieved higher performance metrics compared to the SVM, with an AUC of 0.87–0.91 (“good” to “excellent”) in the ROC curve. However, only 72% sensitivity was reached in the holdout test-sample. This result is probably due to overfitting, as well as some misrepresentation of the cohort by the random undersampling of ALS subjects. If the model overfits the training sample or the test sample differs substantially from the training sample, loss of predictive performance is expected. Still, we consider these results to be a proof of concept of what can be achieved with sound feature selection and a reasonable amount of multiparametric data.

The present study used only microstructural data, in a line of agreement with a previous study that reached a mean five-fold cross validation accuracy of 80% in discriminating ALS from controls from tract-based DTI metrics (28). Future studies should extend on this by incorporating additional MRI parameters like gray matter volume and functional connectivity. For functional connectivity data, machine learning has been already applied to independent component analysis, demonstrating an accuracy of 71.5% (4). The combination of different models, for example in a stacking ensemble machine learning classification, might further increase diagnostic accuracy (9). In a recent study on longitudinal neurodegeneration in the brain in ALS, data from different modalities were combined using deep learning (32). In this study, a random walker model did not only predict disease propagation in ALS, but also contributed to correct survival prediction. Applying deep learning techniques to disease classification might increase diagnostic accuracy in future models.

Limitations

Although the number of subjects with ALS was rather high, the machine learning models were trained with a comparatively low number of control subjects. This sample size hampered our model's performance in two respects, first by the limited data quantity of n <100 control samples and second by class imbalance. In addition, age differed significantly between ALS patients and controls; however, data analysis included age correction in order to compensate for this difference. Given that this is a proof-of-concept study, we advise against using this machine learning model in a clinical setting, as the field is still developing and scientific consensus regarding diagnostic utility of machine learning in ALS has not been reached yet. In addition, a proper clinical classifier has to incorporate mimic disorders, as only these are (falsely) suspected to be ALS cases. It can be argued that many ALS mimics are peripheral neuropathies with normal cranial MRI, hence, training a model with healthy control data and fine-tuning it with mimic disorders might suffice.

Conclusion

The integration of multimodal microstructural neuroimaging data into an appropriate diagnostic classifier demonstrated the power of machine learning for multiparametric, quantitative neuroimaging, and this proof of concept allowed for an accurate dichotomous classification of ALS patients vs. controls. Extending this concept beyond microstructural data might further enhance diagnostic accuracy in future models.

Data Availability Statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics Statement

The study was performed according to institutional guidelines in accordance with the Declaration of Helsinki and was approved by the Ethical Committee of the University of Ulm (reference # 19/12). The patients/participants provided their written informed consent to participate in this study.

Author Contributions

TK: study concept and design, analyses and interpretation of data and drafting of manuscript. JK: study concept and design, interpretation of data, and critical revision of manuscript for intellectual content. H-PM: study concept and design, analyses and interpretation of data, and critical revision of manuscript for intellectual content. AL and AB: interpretation of data and critical revision of manuscript for intellectual content. All authors contributed to the article and approved the submitted version.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Acknowledgments

The authors would like to thank the Ulm University Center for Translational Imaging MoMAN for its support.

References

1. van Es MA, Hardiman O, Chio A, Al-Chalabi A, Pasterkamp RJ, Veldink JH, et al. Amyotrophic lateral sclerosis. Lancet. (2017) 390:2084–98. doi: 10.1016/S0140-6736(17)31287-4

PubMed Abstract | CrossRef Full Text | Google Scholar

2. Kassubek J, Müller HP. Advanced neuroimaging approaches in amyotrophic lateral sclerosis: refining the clinical diagnosis. Expert Rev Neurother. (2020) 20:237–49. doi: 10.1080/14737175.2020.1715798

PubMed Abstract | CrossRef Full Text | Google Scholar

3. Agosta F, Spinelli EG, Filippi M. Neuroimaging in amyotrophic lateral sclerosis: current and emerging uses. Expert Rev Neurother. (2018) 18:395–406. doi: 10.1080/14737175.2018.1463160

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Welsh RC, Jelsone-Swain LM, Foerster BR. The utility of independent component analysis and machine learning in the identification of the amyotrophic lateral sclerosis diseased brain. Front Hum Neurosci. (2013) 7:251. doi: 10.3389/fnhum.2013.00251

PubMed Abstract | CrossRef Full Text | Google Scholar

5. Schuster C, Hardiman O, Bede P. Development of an automated MRI-based diagnostic protocol for amyotrophic lateral sclerosis using disease-specific pathognomonic features: a quantitative disease-state classification study. PLoS ONE. (2016) 11:e0167331. doi: 10.1371/journal.pone.0167331

PubMed Abstract | CrossRef Full Text | Google Scholar

6. Ferraro PM, Agosta F, Riva N, Copetti M, Spinelli EG, Falzone Y, et al. Multimodal structural MRI in the diagnosis of motor neuron diseases. Neuroimage Clin. (2017) 16:240–7. doi: 10.1016/j.nicl.2017.08.002

PubMed Abstract | CrossRef Full Text | Google Scholar

7. Kocar TD, Müller H-P, Ludolph AC, Kassubek J. Feature selection from magnetic resonance imaging data in ALS: a systematic review. Ther Adv Chronic Dis. (2021) 12:20406223211051002

PubMed Abstract | Google Scholar

8. Grollemund V, Pradat PF, Querin G, Delbot F, Le Chat G, Pradat-Peyre JF, et al. Machine learning in amyotrophic lateral sclerosis: achievements, pitfalls, and future directions. Front Neurosci. (2019) 13:135. doi: 10.3389/fnins.2019.00135

PubMed Abstract | CrossRef Full Text | Google Scholar

9. E Elahi GMM, Kalra S, Zinman L, Genge A, Korngut L, Yang YH. Texture classification of MR images of the brain in ALS using M-CoHOG: a multi-center study. Comput Med Imaging Graph. (2020) 79:101659. doi: 10.1016/j.compmedimag.2019.101659

PubMed Abstract | CrossRef Full Text | Google Scholar

10. Müller HP, Dreyhaupt J, Roselli F, Schlecht M, Ludolph AC, Huppertz HJ, et al. Focal alterations of the callosal area III in primary lateral sclerosis: an MRI planimetry and texture analysis. Neuroimage Clin. (2020) 26:102223. doi: 10.1016/j.nicl.2020.102223

PubMed Abstract | CrossRef Full Text | Google Scholar

11. Cedarbaum JM, Stambler N, Malta E, Cedarbaum JM, Stambler N, Malta E, et al. The ALSFRS-R: a revised ALS functional rating scale that incorporates assessments of respiratory function. BDNF ALS study group (phase III). J Neurol Sci. (1999) 169:13–21. doi: 10.1016/s0022-510x(99)00210-5

PubMed Abstract | CrossRef Full Text | Google Scholar

12. Müller HP, Unrath A, Ludolph AC. Kassubek, J. Preservation of diffusion tensor properties during spatial normalisation by use of tensor imaging and fibre tracking on a normal brain database. Phys Med Biol. (2007) 52:N99–N109. doi: 10.1088/0031-9155/52/6/N01

PubMed Abstract | CrossRef Full Text | Google Scholar

13. Unrath A, Müller HP, Riecker A, Unrath A, Müller HP, Riecker A. Whole brain-based analysis of regional white matter tract alterations in rare motor neuron diseases by diffusion tensor imaging. Hum Brain Mapp. (2010) 31:1727–40. doi: 10.1002/hbm.20971

PubMed Abstract | CrossRef Full Text | Google Scholar

14. Behler A, Kassubek J, Müller HP. Age-related alterations in DTI metrics in the human brain – consequences for age correction. Front Aging Neurosci. (2021) 13:682109. doi: 10.3389/fnagi.2021.682109

PubMed Abstract | CrossRef Full Text | Google Scholar

15. Rosskopf J, Müller HP, Dreyhaupt J, Gorges M, Ludolph AC, Kassubek J. Ex post facto assessment of diffusion tensor imaging metrics from different MRI protocols: preparing for multicentre studies in ALS. Amyotroph Lateral Scler Frontotemporal Degener. (2015) 16:92–101. doi: 10.3109/21678421.2014.977297

PubMed Abstract | CrossRef Full Text | Google Scholar

16. Müller HP, Turner MR, Grosskreutz J, Abrahams S, Agosta F, Benatar M, et al. A large-scale multicentre cerebral diffusion tensor imaging study in amyotrophic lateral sclerosis. J Neurol Neurosurg Psychiatry. (2016) 87:570–9. doi: 10.1136/jnnp-2015-311952

PubMed Abstract | CrossRef Full Text | Google Scholar

17. Kalra S, Müller HP, Ishaque A, Zinman L, Korngut L, Genge A, et al. A prospective harmonized multicenter DTI study of cerebral white matter degeneration in ALS. Neurology. (2020) 95:e943–52. doi: 10.1212/WNL.0000000000010235

PubMed Abstract | CrossRef Full Text | Google Scholar

18. Müller HP, Unrath A, Sperfeld AD, Ludolph AC, Riecker A, Kassubek J. Diffusion tensor imaging and tractwise fractional anisotropy statistics: quantitative analysis in white matter pathology. Biomed Eng Online. (2007) 6:42. doi: 10.1186/1475-925X-6-42

PubMed Abstract | CrossRef Full Text | Google Scholar

19. Kassubek J, Müller HP, Del Tredici K, Brettschneider J, Pinkhardt EH, Lulé D, et al. Diffusion tensor imaging analysis of sequential spreading of disease in amyotrophic lateral sclerosis confirms patterns of TDP-43 pathology. Brain. (2014) 137:1733–40. doi: 10.1093/brain/awu090

PubMed Abstract | CrossRef Full Text | Google Scholar

20. Müller HP, Lulé D, Roselli F, Behler A, Ludolph AC, Kassubek J. Segmental involvement of the corpus callosum in C9orf72-associated ALS: a tract of interest-based DTI study. Ther Adv Chronic Dis. (2021) 12:20406223211002969. doi: 10.1177/20406223211002969

PubMed Abstract | CrossRef Full Text | Google Scholar

21. Tu S, Wang C, Menke RAL, Talbot K, Barnett M, Kiernan MC, et al. Regional callosal integrity and bilaterality of limb weakness in amyotrophic lateral sclerosis. Amyotroph Lateral Scler Frontotemporal Degener. (2020) 21:396–402. doi: 10.1080/21678421.2020.1733020

PubMed Abstract | CrossRef Full Text | Google Scholar

22. Kunimatsu A, Aoki S, Masutani Y, Abe O, Hayashi N, Mori H, et al. The optimal trackability threshold of fractional anisotropy for diffusion tensor tractography of the corticospinal tract. Magn Reson Med Sci. (2004) 3:11–7. doi: 10.2463/mrms.3.11

PubMed Abstract | CrossRef Full Text | Google Scholar

23. Burges CA. tutorial on support vector machines for pattern recognition. Data Min Knowl Discov. (1998) 2:121–67. doi: 10.1023/A:1009715923555

CrossRef Full Text | Google Scholar

24. van der Burgh HK, Schmidt R, Westeneng HJ, de Reus MA, van den Berg LH, van den Heuvel MP. Deep learning predictions of survival based on MRI in amyotrophic lateral sclerosis. Neuroimage Clin. (2016) 13:361–9. doi: 10.1016/j.nicl.2016.10.008

PubMed Abstract | CrossRef Full Text | Google Scholar

25. Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, et al. Scikit-learn: machine learning in python. J Mach Learn Res. (2011) 12:2825–30.

Google Scholar

26. Mandrekar JN. Receiver operating characteristic curve in diagnostic test assessment. J Thorac Oncol. (2010) 5:1315–6. doi: 10.1097/JTO.0b013e3181ec173d

PubMed Abstract | CrossRef Full Text | Google Scholar

27. Safari S, Baratloo A, Elfil M, Negida A. Evidence based emergency medicine; part 5 receiver operating curve and area under the curve. Emergency. (2016) 4:111–3. doi: 10.22037/AAEM.V4I2.232.G232

PubMed Abstract | CrossRef Full Text | Google Scholar

28. Sarica A, Cerasa A, Valentino P, Yeatman J, Trotta M, Barone S, et al. The corticospinal tract profile in amyotrophic lateral sclerosis. Hum Brain Mapp. (2017) 38:727–39. doi: 10.1002/hbm.23412

PubMed Abstract | CrossRef Full Text | Google Scholar

29. Foerster BR, Dwamena BA, Petrou M, Carlos RC, Callaghan BC, Churchill CL, et al. Diagnostic accuracy of diffusion tensor imaging in amyotrophic lateral sclerosis: a systematic review and individual patient data meta-analysis. Acad Radiol. (2013) 20:1099–106. doi: 10.1016/j.acra.2013.03.017

PubMed Abstract | CrossRef Full Text | Google Scholar

30. Fekete T, Zach N, Mujica-Parodi LR, Turner MR. Multiple kernel learning captures a systems-level functional connectivity biomarker signature in amyotrophic lateral sclerosis. PLoS ONE. (2013) 8:e85190. doi: 10.1371/journal.pone.0085190

PubMed Abstract | CrossRef Full Text | Google Scholar

31. Gorges M, Del Tredici K, Dreyhaupt J, Braak H, Ludolph AC, Müller HP, et al. Corticoefferent pathology distribution in amyotrophic lateral sclerosis: in vivo evidence from a meta-analysis of diffusion tensor imaging data. Sci Rep. (2018) 8:15389. doi: 10.1038/s41598-018-33830-z

PubMed Abstract | CrossRef Full Text | Google Scholar

32. Meier JM, van der Burgh HK, Nitert AD, Bede P, de Lange SC, Hardiman O, et al. Connectome-based propagation model in amyotrophic lateral sclerosis. Ann Neurol. (2020) 87:725–38. doi: 10.1002/ana.25706

PubMed Abstract | CrossRef Full Text | Google Scholar