Machine learning methods for predicting major types of rheumatic heart diseases in children of Southern Punjab, Pakistan

Objective: Rheumatic heart disease (RHD) is a major health problem in the world, particularly in developing countries. This study aimed to predict mitral regurgitation (MR) and mitral stenosis (MS) RHD among children with RHD.

Methodology: Data was collected from the Pediatric Cardiology Department at Chaudhry Pervaiz Elahi Institute of Cardiology Multan, Pakistan from March to October 2019. A sample of 561 children aged 4–14 years, who were diagnosed with RHD of either MR or MS, were recruited from the hospital’s outpatient department. The presence of multivariate outliers was detected, and different machine learning methods, including subset logistic regression, subset logistic regression after deletion, stepwise winsorized logistic regression, robust logistic regression, subset deep neural network, and random forest models were compared using the area under receiver operating characteristics (ROC) curve, sensitivity, and specificity. Parsimony was also considered in model selection.

Results: Out of 561 patients in this study, 75.94% had RHD MR and 24.06% had RHD MS. The average age of study participants was 9.19 ± 2.45 years and of them 55.43% were male. Among the male participants, 58.6 and 45.2% had MR and MS, respectively; and among female participants, those were 70.4 and 29.6%, respectively. Subset logistic regression after deletion appeared as competitive with a discrimination power of 90.1% [95% CI 0.818–0.983]. The sensitivity and specificity of this model were 85.1 and 70.6%.

Conclusion: The best predictive model was subset logistic regression after deletion. The predicted method will be used in the decision-making process, which helps early diagnosis of the disease and leads to prevention. The study findings provide the proper guideline for earlier diagnosis of the RHD MR and MS cases among children with RHD in Pakistan.

Introduction

Rheumatic heart disease (RHD) is a primary heart problem globally, particularly in developing countries (1–4). RHD is caused by rheumatic fever, an inflammatory disease that can affect many connective tissues, especially in the heart, skin, and brain. It also causes extreme pains in joints, particularly in the knees and ankles. The valves of the heart become sore and can be damaged over time and this can result in leaking and narrowing of the valve of the heart making it harder for the heart to work normally (5). According to WHO, each year an estimated 16 million people suffer from RHD worldwide (6). In developing countries, RHD was a major health burden, causing mortality and morbidity among both children and adults. Among the RHD (lesions), mitral regurgitation (MR) was the most common lesion followed by mitral stenosis (MS) (7, 8). In Latin America, the prevalence of RHD was 1.3/1,000 and in Peru, it ranged from 3.9 to 19.7/1,000 (9). In India, the prevalence of RHD was 1.3–4.5/1,000 (10) and in Bangladesh, it was 1.2/1,000 (11).

In Pakistan, RHD mostly occurred among young adults (12), and the most common lesion was the MR, of which 10.5% had mild, 31.4% had moderate, and 31.4% had severe. MS was the second most common lesion of which 7.0% had mild, 8.1% had moderate, and 55.8% had severe (13). The reported prevalence of RHD in Pakistan was found to be 22/1,000 in school-going children (14). Studies in Bangladesh, India, and Pakistan showed that the common non-clinical predictors of RHD were age, gender, living place, family size, number of people sharing a living room, wall material used in the home, number of siblings, and maternal occupation (4, 15, 16). Further studies conducted in Nigeria showed that the major risk factors of RHD are poverty, malnutrition, overcrowding, poor housing condition, low-socioeconomic status, lack of healthcare facilities, and environmental and genetic factors (2, 17). Several studies identified the factors related to RHD using the simple logistic regression model (3, 8, 15).

None of the previous research identifies the risk factor for the further types of RHD. These factors consist of optimal combinations of clinical and non-clinical symptoms used for the earlier prediction of the disease. These predictions are used in the decision-making process, which helps early diagnosis of the disease leading to prevention.

Furthermore, the presence of multivariate outliers is common in medical data. However, detection of these outliers before the modeling is rare in the literature, especially in generalized linear modeling. Different machine learning models were used in literature to predict heart disease. Prediction of heart disease (18, 19) by machine learning techniques using random forest and artificial neural network. A study (20) also used a random forest model to predict the risk factors of heart disease.

None of the studies focused on the prediction of further types of heart disease in children.

This study aimed to identify the multivariate outliers in data and compare present machine learning algorithms to efficiently predict the RHD MR and RHD MS in children with RHD.

Materials and methods

Study design, population, and sample

A cross-sectional and retrospective study design was used to collect data from the outpatient department (OPD) of the Pediatric Department at the Chaudhry Pervaiz Elahi Institute of Cardiology Multan, Pakistan. The data was collected from March to October 2019 and the data was collected from all the accessible children who were present during the given period in the hospital. A sample of 561 children diagnosed with RHD was recruited. This sample achieves more than 80% power.

Patient’s consent and ethics approval

The study was approved by the Departmental Ethics Committee and boards of Bahauddin Zakariya University, Multan, Pakistan. Written permission was obtained from the hospital. The guardians of each child were informed about the study objectives and data confidentiality and were invited to participate in the study. Upon agreement, they signed a consent form.

Operational definition of the variables

The data was collected by the principal author. The variables related to demographics, socio-economic status, and general health were collected from children and their parents. While the variables related to patients’ symptoms, clinical measurements, and medical diagnostics were collected from the hospital records.

A brief description of the variables is as follows. The variable gender was classified as male and female, and body mass index (BMI) kg/m² was categorized into underweight (<5th percentile), normal to a healthy weight (5th percentile to <85th percentile), overweight (85th percentile to <95th percentile), and obese (95th percentile or greater) (21). A child is physically active if he/she was playing for at least 60 min a day. A child was considered to have fast food, low-caloric food, and staple food if eating fast food more than once a week, a calorie intake of fewer than 1,200 calories per day, and using cereal grain or tubers as a staple food, respectively.

Access to health care facilities was defined if there was any government hospital or medical unit and government doctor available in their area. Similarly, access to education facilities was defined if there was any government school available in their area. The education level of parents was categorized as uneducated, primary to the middle, middle to higher secondary, higher secondary to graduate, and masters to higher. The occupation of the father was categorized into unemployed or diseased, labor or farmer, private job, business, or civil servant. The dwelling area was categorized into rural and urban. The nutrition was categorized as Good nutrition (protein of more than 5 ounces, fruits up to two cups, vegetables up to three cups, grains up to 6 ounces, and dairy up to 3 cups), normal nutrition (protein up to 5 ounces, fruits up to 1.5 cups, vegetables up to 2.5 cups, grains up to 5 ounces, and dairy up to 2.5 cups), less than normal nutrition is considered poor nutrition. Home environment and parental interaction were categorized into poor, normal, and good. The health condition of other people living in the home was categorized as follows. Good health: no respiratory infections or asthma, no lead poisoning, no injuries, and good mental health. The good quality of health care facilities was defined as well-trained and motivated staff, accurate medical records, water, energy, sanitation, hand hygiene, and waste disposal facilities that are functional, reliable, and safe, and adequate stocks of medicines, supplies, and equipment. The house may consider in good condition if there was good ventilation, sanitation, heating, lighting, facilities for cooking, suitable storage for food, and good access to shops and facilities. The housing tenure was categorized as rented or own house.

The included clinical factors were given here. The heart size was categorized as large and normal; it was written on the children’s file by the doctor after examining the chest X-ray of the child. Whether the child had anemia or not was written on the hospital file by the doctor after examining the complete blood count. The family history of diabetes, smoking status in the family, family history of cardiac disease, sleeping apnea, and wheezing were also written on the children’s file. The echocardiography measurement [which are left ventricular internal-systolic dimension (LVISD) (mm), left ventricular posterior wall dimensions (LVPWD) (mm), left ventricular internal dimension diastole (LVIDD) (mm), left ventricular internal dimension systole (LVIDS) (mm)], and ejection fraction (EF) (%) were retrospectively collected from hospital records. The following blood measures were also collected from hospital records: white blood cell count (WBC) (109/L), red blood cell count (RBC) (1012/L), hemoglobin (HB) (g/dL), hematocrit (HCT) (w), mean corpuscular volume (MCV) (fl), mean corpuscular hemoglobin (MCH) (pg), mean corpuscular hemoglobin concentration (MCHC) (g/dl), platelet counts (109/L), neutrophils (%), lymphocytes (%), monocytes (%), and eosinophils (%).

The outcome variable was RHD type, i.e., RHD MS and RHD MR.

Data management and analysis

Data were summarized and presented as mean (±standard deviation) for numerical data and relative percentages for categorical data. For statistical modeling and validation, the data were randomly divided into two parts, where the first part (85% of data) was used for model training and the second part (15% of data) was used for testing. The following measurements were used to detect multivariate outliers in the generalized linear model (GLM): Leverage (22, 23), Cook’s Distance (24, 25), Modified Cook Distance (26), Andrew’s Pregibon (27), Covariance Ratio (28), and Welsch’s Distance (29). The cases that were jointly identified by all these methods were considered outliers. For prediction, the following methods were used and compared: stepwise logistic regression (SLR) (30, 31), stepwise winsorized logistic regression (SWLR) where winsorization was detail discussed in literature (32, 33), stepwise logistic regression after deletion (SLRAD), robust logistic regression (RLR) (34, 35) subset deep neural network (sDNN) based on deep neural network (DNN) (36) and random forest (RF) (37). The mathematical and procedural details of the diagnostic measures and models are described in Appendix A in Supplementary material. Finally, using the test data the area under the receiver operating characteristic (ROC) curve, its 95% confidence interval, sensitivity, and specificity were calculated for each model.

R language was used for data analysis. For The sDNN model, we used the neuralnet package, and for the random forest model and the randomForest package in the R language. All the presented results are based on 10-fold cross-validation.

Results

The univariate analysis presented in Tables 1, 2 showed that for the RHD MS group, the average age was 8.85 (±2.51) years, 19.6% male and 29.6% female. A total of 71.8% of mothers were uneducated, 25.9% were primary or middle, and 2.2% were secondary or higher educated. A total of 37.0% of fathers were uneducated, 60.0% were primary or middle, and 2.9% were secondary and higher educated. A total of 1.5% of fathers were unemployed, 47.4% were laborers or farmers, 46.7% had small businesses, and 4.44% had private jobs. For the RHD MR group, the average age was 9.30 (±2.44) years, 80.4% male and 70.4% female. A total of 66.4% of mothers were uneducated, 29.1% were primary or middle, and 4.5% were secondary or higher educated. A total of 52.3% of fathers were uneducated, 41.5% were primary or middle, and 6.1% were secondary and higher educated. A total of 62.2% of fathers were laborers or farmers, 36.4% had a small business, 0.47% had a private job, and 0.93% were civil servants. Diabetes in the family and smoking in the family was found higher in the RHD MR group. Meanwhile, family history, wheezing, and the use of staple food were found higher in children having RHD MS. Mostly the fathers of children having RHD MR were labor or former, while the fathers of children having RHD MS had a small business. The children having RHD MR belonged to urban areas, while the children having RHD MS belonged to rural areas.

TABLE 1

Table 1. Descriptive analysis of the categorical variables.

TABLE 2

Table 2. Descriptive analysis of the quantitative variables.

The result of outlier detection using various outlier detection methods was presented in Figure 1, and according to these methods, there were 13 (2.7%) cases detected as outliers.

FIGURE 1

Figure 1. Outliers detected using multivariate outliers detection approach in GLM. (A) Outliers detection using the cook distance. (B) Outliers detection using leverages. (C) Outliers detection using the covariance ratio. (D) Outliers detection using the modified cook distance. (E) Outliers detection using Andrew’s Pregibon method. (F) Outliers detection using Welsch’s Distance. The red marked straight line parallel to the x-axis represents the cut-off point. Any observation that falls above this line is considered an outlier.

The hidden structure and layers of the sDNN model are presented in Figure 2. The sDNN model consists of the first layer, hidden layers, and the last layer. The first layer is the input layer, while the last layer is the output layer, where the explanatory variables are the input variable.

FIGURE 2

Figure 2. sDNN modeling structure.

Figure 3 shows the weights assigned by the sDNN model to each variable, where a higher weight shows the higher importance of the respective variable. The variables anemia had the highest positive weight followed by the physically inactive child, education of the child’s mother, WBC, etc. LVIDS, use of fast food, LVIDD, LVISD, MCHC, number of children in a family, dwelling area, and use of low-calorie food have the highest weights. In the sDNN model, there are 33 input variables, 8 hidden variables, and 1 output variable.

FIGURE 3

Figure 3. Weights assigned to each variable by the sDNN model.

Table 3 shows the results of logistic regression models. The variable that was found significant in all models were LVISD, WBC, male children, wheezing in children, no access to basic health care facilities, and poor and normal housing condition.

TABLE 3

Table 3. Multivariate logistic models using stepwise selection approach.

Table 4 and Figure 4 show the comparison of all models. The results showed that the SLRAD model has the highest ROC 0.901 [95% CI 0.818–0.983] with sensitivity and specificity of 85.1 and 70.6%, respectively. The second highest ROC 89.5% [95% CI 0.815–0.974] was for the SWLR model with sensitivity and specificity of 83.6 and 64.7%, respectively. The sDNN model had the third highest ROC 89.2% [95% CI 0.811–0.979] with a sensitivity of 86.5% and specificity of 82.4%. Though there is no significant difference between the ROC of the models (95% CIs are overlapping for all models); however, a tradeoff between ROC and sensitivity/specificity and the number of variables in the model showed that SLRAD is the better model.

TABLE 4

Table 4. Performance comparison of the models on validation data.

FIGURE 4

Figure 4. ROC curves for comparison of model performance.

Discussion

In this study, the results show that male children are more affected by both RHD MS and RHD MR than female children the previous study also supports our finding (16). The model comparison results show that the best prediction model was SLRAD.

The results of the best-selected model SLRAD show that wheezing contributes 3.56 times, LVISD contributes 3.12 times, RBC contributes 1.81 times, eosinophils contribute 1.19 times, WBC contributes 1.15 times, and HCT contributes 1.07 times in RHD MR as compared to RHD MS. While the risk of RHD MS increases approximately 47% with male children, 30% with diabetes in the family, 27% with staple food, 20% with not a good housing condition [a similar result is also established by the study (1, 3)], and 1.9% with low family income, as compared to RHD MR. The result shows that the poor facilities of health care were a barrier to detecting and diagnosing at an early stage of disease and medication or treatment of RHD, the previous study also supports our findings (12).

Similarly, for the sDNN, the weights of these variables are also significant from our suggested cutoff points. The best selected SLRAD models show some additional variables significant as compared to other models, which are as follows: the use of fast food contributes approximate 2.9 times more is RHD MR as compared to RHD MS. Similarly, the risk of RHD MS increases by approximately 30% when fathers are uneducated and 75% with the risk of LVPWD as compared to RHD MR.

Conclusion

Improvement in socioeconomic status leads to a significant reduction in RHD MS cases. Therefore, the policymakers/facilitators need to be properly monitored and facilitate those children and families having a low family income, uneducated fathers, no access to basic health care facilities, and poor housing conditions. To control the cases of RHD MR, children should be discouraged for to use fast food. And child suffering from wheezing and non-normal WBC, RBC, HCT, and MCV counts need to be monitored as it has a significant contribution to RHD MR cases. Moreover, the best predictive model is SLRAD. The selected model can help medical practitioners and cardiologists to understand and diagnose the risk factors, their contribution, and risk estimation of RHD MS and RHD MR in children. Thus, the study findings provide the proper guideline to control the RHD MR and MS cases among children with RHD in Pakistan.

Data availability statement

The raw data supporting the conclusions of this article will be made available by the first and corresponding author, without undue reservation.

Ethics statement

The studies involving human participants were reviewed and approved by Board of Advanced Studies and Research, Bahauddin Zakariya University, Multan, Pakistan. Written informed consent to participate in this study was provided by the participants or their legal guardian/next of kin.

Author contributions

SS, HK, BB, and AA contributed to the conception and design of the study. SS and HK organized the dataset, performed the statistical analysis, and wrote the first draft of the manuscript with the help of BB, AA, MAS, and MFS. MAS and MFS reviewed the final draft from a statistical and medical perspective, respectively. All authors contributed to the manuscript revision, read, and approved the final version.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fcvm.2022.996225/full#supplementary-material

References

1. Thakur JS, Negi PC, Ahluwalia SK, Vaidya NK. Epidemiological survey of rheumatic heart disease among school children in the Shimla Hills of northern India: prevalence and risk factors. J Epidemiol Commun Health. (1996) 50:62–7. doi: 10.1136/jech.50.1.62

PubMed Abstract | CrossRef Full Text | Google Scholar

2. Sani MU, Karaye KM, Borodo MM. Prevalence and pattern of rheumatic heart disease in the Nigerian savannah: an echocardiographic study. Cardiovasc J Afr. (2007) 18:295.

Google Scholar

3. Dobson J, Steer AC, Colquhoun S, Kado J. Environmental factors and rheumatic heart disease in Fiji. Pediatr Cardiol. (2012) 33:332–6. doi: 10.1007/s00246-011-0139-x

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Joseph N, Madi D, Kumar GS, Nelliyanil M, Saralaya V, Rai S. Clinical spectrum of rheumatic fever and rheumatic heart disease: a 10-year experience in an urban area of South India. North Am J Med Sci. (2013) 5:647. doi: 10.4103/1947-2714.122307

PubMed Abstract | CrossRef Full Text | Google Scholar

5. Harris C, Croce B, Cao C. Rheumatic heart disease. Ann Cardiothor Surg. (2015) 4:492.

Google Scholar

6. Mulatu HA, Woldemichael M, Aberra M. Prevalence of rheumatic heart disease among primary school students in Mid-Eastern Ethiopia. Biol Syst Open Access. (2015) 5:149. doi: 10.4172/2329-6577.1000149

CrossRef Full Text | Google Scholar

7. Negi PC, Mahajan K, Rana V, Sondhi S, Mahajan N, Rathour S, et al. Clinical characteristics, complications, and treatment practices in patients with RHD: 6-year results from HP-RHD registry. Global Heart. (2018) 13:267–74. doi: 10.1016/j.gheart.2018.06.001

PubMed Abstract | CrossRef Full Text | Google Scholar

8. Lubega S, Aliku T, Lwabi P. Echocardiographic pattern and severity of valve dysfunction in children with rheumatic heart disease seen at Uganda Heart Institute, Mulago hospital. Afr Health Sci. (2014) 14:617–25. doi: 10.4314/ahs.v14i3.17

PubMed Abstract | CrossRef Full Text | Google Scholar

9. Spitzer E, Mercado J, Islas F, Rothenbühler M, Kurmann R, Zürcher F, et al. Screening for rheumatic heart disease among Peruvian children: a two-stage sampling observational study. PLoS One. (2015) 10:e0133004. doi: 10.1371/journal.pone.0133004

PubMed Abstract | CrossRef Full Text | Google Scholar

10. Periwal KL, Gupta BK, Panwar RB, Khatri PC, Raja S, Gupta R. Prevalence of rheumatic heart disease in school children in Bikaner: an echocardiographic study. JAPI. (2006) 54:279–82.

Google Scholar

11. Ahmed J, Mostafa Zaman M, Monzur Hassan MM. Prevalence of rheumatic fever and rheumatic heart disease in rural Bangladesh. Trop Doct. (2005) 35:160–1. doi: 10.1258/0049475054620879

PubMed Abstract | CrossRef Full Text | Google Scholar

12. Abrar A. Frequency of rheumatic heart disease in patients undergoing echocardiography in district Dera Ismail Khan. Gomal J Med Sci. (2014) 12: 147–50.

Google Scholar

13. Anam A, Faisal AWK, Afzal N, Shahzad F. Immune system activation in rheumatic heart disease. J Coll Physicians Surg Pak. (2018) 28:518–22. doi: 10.29271/jcpsp.2018.07.518

PubMed Abstract | CrossRef Full Text | Google Scholar

14. Ali F, Hasan B, Ahmad H, Hoodbhoy Z, Bhuriwala Z, Hanif M, et al. Detection of subclinical rheumatic heart disease in children using a deep learning algorithm on digital stethoscope: a study protocol. BMJ Open. (2021) 11:e044070. doi: 10.1136/bmjopen-2020-044070

PubMed Abstract | CrossRef Full Text | Google Scholar

15. Riaz BK, Selim S, Karim MN, Chowdhury KN, Chowdhury SH, Rahman MR. Risk factors of rheumatic heart disease in Bangladesh: a case-control study. J Health Popul Nutr. (2013) 31:70. doi: 10.3329/jhpn.v31i1.14751

PubMed Abstract | CrossRef Full Text | Google Scholar

16. Beg AB, Younas M, Asma T. Rheumatic Heart Disease (RHD). Profess Med J. (2016) 23:324–7. doi: 10.29309/TPMJ/2016.23.03.1482

CrossRef Full Text | Google Scholar

17. Cannon J, Roberts K, Milne C, Carapetis JR. Rheumatic heart disease severity, progression and outcomes: a multi-state model. J Am Heart Assoc. (2017) 6:e003498. doi: 10.1161/JAHA.116.003498

PubMed Abstract | CrossRef Full Text | Google Scholar

18. Katarya R, Meena SK. Machine learning techniques for heart disease prediction: a comparative study and analysis. Health Technol. (2021) 11:87–97. doi: 10.1007/s12553-020-00505-7

CrossRef Full Text | Google Scholar

19. Krittanawong C, Virk HUH, Bangalore S, Wang Z, Johnson KW, Pinotti R, et al. Machine learning prediction in cardiovascular diseases: a meta-analysis. Sci Rep. (2020) 10:16057. doi: 10.1038/s41598-020-72685-1

PubMed Abstract | CrossRef Full Text | Google Scholar

20. Molla S, Shamrat FJM, Rafi RI, Umaima U, Arif MAI, Hossain S, et al. A predictive analysis framework of heart disease using machine learning approaches. Bull Electr Eng Inform. (2022) 11:2705–16. doi: 10.11591/eei.v11i5.3942

CrossRef Full Text | Google Scholar

21. Centers for Disease Control and Prevention. Defining Childhood Obesity: BMI for Children and Teens. Atlanta: Centers for Disease Control and Prevention (2018).

Google Scholar

22. Belsley DA, Kuh E, Welsch R. Regression Diagnostics: Identifying Influential Data and Sources of Collinearity. New York, NY: Wiley (1980). doi: 10.1002/0471725153

CrossRef Full Text | Google Scholar

23. Bagheri A, Midi H, Imon AHMR. The effect of collinearity-influential observations on collinear data set: a monte carlo simulation study. J Appl Sci. (2010) 10:2086–93. doi: 10.3923/jas.2010.2086.2093

CrossRef Full Text | Google Scholar

24. Cook RD. Detection of influential observations in linear regression. Technometrics. (1977) 19:15–8. doi: 10.1080/00401706.1977.10489493

CrossRef Full Text | Google Scholar

25. Hardin JW, Hilbe JM. Generalized Linear Models and Extensions. 3rd ed. Houston, TX: Stata Press Publication (2012).

Google Scholar

26. Amin M. Influence Diagnostic Methods in Generalized Linear Models with Biased Estimators. Ph.D. thesis. Multan: Bahauddin Zakariya University (2016).

Google Scholar

27. Andrews DF, Pregibon D. Finding the outliers that matter. J R Stat Soc Ser B. (1978) 40:85–93. doi: 10.1111/j.2517-6161.1978.tb01652.x

CrossRef Full Text | Google Scholar

28. Ullah MA, Pasha GR. The origin and developments of influence measures in regression. Pak J Stat. (2009) 25:295–307.

Google Scholar

29. Welsch RE. Influence functions and regression diagnostics. In: R Launer, A Siegel editors. Modern Data Analysis. New York, NY: Academic Press (1982). p. 149–69. doi: 10.1016/B978-0-12-438180-3.50014-1

CrossRef Full Text | Google Scholar

30. Hosmer DW, Lemeshow S. Applied Logistic Regression. New York, NY: John Wiley & Sons (2000). doi: 10.1002/0471722146

CrossRef Full Text | Google Scholar

31. Peng CYJ, So TSH, Stage FK, John EPS. The use and interpretation of logistic regression in higher education journals: 1988–1999. Res High Educ. (2002) 43:259–93. doi: 10.1023/A:1014858517172

CrossRef Full Text | Google Scholar

32. Reifman A, Keyton K. Winsorize. In: NJ Salkind editor. Encyclopedia of Research Design. Thousand Oaks, CA: Sage (2010).

Google Scholar

33. Wicklin R. Winsorization: the Good, the Bad, and the Ugly. (2017). Available online at: https://blogs.sas.com/content/iml/2017/02/08/winsorization-good-bad-and-ugly.html (accessed February 08, 2017).

Google Scholar

34. Frank R, Hampel. Robust Statistics: The Approach Based on Influence Functions. (Vol. 196). Hoboken, NJ: Wiley-Interscience (1986).

Google Scholar

35. Huber PJ. Robust statistics. In: M Lovric editor. International Encyclopedia of Statistical Science. Berlin: Springer (2011). doi: 10.1007/978-3-642-04898-2_594

CrossRef Full Text | Google Scholar

36. Montavon G, Samek W, Müller KR. Methods for interpreting and understanding deep neural networks. Digit Sign Process. (2018) 73:1–15. doi: 10.1016/j.dsp.2017.10.011

CrossRef Full Text | Google Scholar

37. Breiman L. Random forests. Mach Learn. (2001) 45:5–32. doi: 10.1023/A:1010933404324

CrossRef Full Text | Google Scholar