Michael A. Pienaar
Joseph B. Sempa
Nicolaas Luwes
Elizabeth C. George4- 1Department of Paediatrics and Child Health, Paediatric Critical Care Unit, University of the Free State, Bloemfontein, South Africa
- 2Department of Biostatistics, Faculty of Health Sciences, University of the Free State, Bloemfontein, South Africa
- 3Department of Electrical, Electronic and Computer Engineering, Faculty of Engineering, Built Environment and Information Technology, Central University of Technology, Bloemfontein, South Africa
- 4Medical Research Council Clinical Trials Unit, University College London, London, United Kingdom
- 5Paediatric Cardiology Unit, Department of Paediatrics and Child Health, University of the Free State, Bloemfontein, South Africa
Objectives: Delays in identification, resuscitation and referral have been identified as a preventable cause of avoidable severity of illness and mortality in South African children. To address this problem, a machine learning model to predict a compound outcome of death prior to discharge from hospital and/or admission to the PICU was developed. A key aspect of developing machine learning models is the integration of human knowledge in their development. The objective of this study is to describe how this domain knowledge was elicited, including the use of a documented literature search and Delphi procedure.
Design: A prospective mixed methodology development study was conducted that included qualitative aspects in the elicitation of domain knowledge, together with descriptive and analytical quantitative and machine learning methodologies.
Setting: A single centre tertiary hospital providing acute paediatric services.
Participants: Three paediatric intensivists, six specialist paediatricians and three specialist anaesthesiologists.
Interventions: None.
Measurements and main results: The literature search identified 154 full-text articles reporting risk factors for mortality in hospitalised children. These factors were most commonly features of specific organ dysfunction. 89 of these publications studied children in lower- and middle-income countries. The Delphi procedure included 12 expert participants and was conducted over 3 rounds. Respondents identified a need to achieve a compromise between model performance, comprehensiveness and veracity and practicality of use. Participants achieved consensus on a range of clinical features associated with severe illness in children. No special investigations were considered for inclusion in the model except point-of-care capillary blood glucose testing. The results were integrated by the researcher and a final list of features was compiled.
Conclusion: The elicitation of domain knowledge is important in effective machine learning applications. The documentation of this process enhances rigour in such models and should be reported in publications. A documented literature search, Delphi procedure and the integration of the domain knowledge of the researchers contributed to problem specification and selection of features prior to feature engineering, pre-processing and model development.
Introduction
Failures in acute care and critical care systems such as triage, identification and resuscitation present a significant challenge in the treatment of life-threatening illness and injury in low to middle income countries (1). One of the key weaknesses is the failure of triage and identification systems to detect children in need of life-saving care (2). In South Africa, it has been found that failures in identification, accurate assessment of severity of illness, early resuscitation, and timely referral to higher levels of care were responsible for significant avoidable severity of illness and mortality (3).
Machine learning has drawn considerable interest in recent medical literature. Machine learning offers a wide range of possible use-cases in the clinical setting, including diagnosis, prognosis, workflow and improving patient access to services (4). The exploration of these applications has extended to paediatric research (5). Lonsdale et al. and Rajkomar et al., however, both point out that this area is under-explored to date (4, 5). This new field demands a broad interdisciplinary collaboration. In the South African and developing context, it is also vitally important that research be conducted in this field to ensure that capacity is developed to investigate and implement such technology in a manner suited to the unique needs and contexts of our setting.
The integration of human knowledge is a crucial aspect of the development, architecture, interpretation, and use of machine learning models. This includes the integration of human knowledge in a field (domain knowledge) as well as knowledge of learning, the human brain, computer science and statistics (general knowledge) (6). Kerrigan et al. have conducted a survey of the elicitation of domain knowledge in applied machine learning and pointed to the need for the documentation of the elicitation process (7). In doing so, they have developed a taxonomy for the elicitation process that includes the elicitation goal, the elicitation target, the elicitation process and the use of elicited knowledge. This has aimed to address the specific challenge of this elicitation process being undertaken in an ad-hoc manner (8).
The use of machine learning in triage and mortality prediction has been described in high-income countries. Goto et al. have described the use of machine learning to predict paediatric intensive care unit (PICU) admission and hospitalisation (9). Aczon et al. and Kim et al. have described models for the prediction of mortality in PICU. These models represent significant progress on existing models in their ability to make dynamic or continuous assessments of mortality risk over time (10, 11). There is a paucity of literature describing such applications in (12) Africa. This paucity together with the contribution of failures in triage and identification of critically ill children provide a strong rationale for conducting research in this area and developing clinically implementable tools for the identification of critically ill children that are appropriate to the South African setting. With this rationale, this study undertook to elicit expert clinical domain knowledge to clearly define the goals of and select viable features for predictive models for paediatric critical illness. The application of this work is described in a recent publication by our group in which we describe the prospective development of machine learning models for the identification of critically ill children (12).
Methods and study design
In this study, the domain knowledge elicitation was documented within the taxonomy proposed by Kerrigan et al. and its role in the various aspects of the development of the machine learning application was described. A literature search was combined with a Delphi procedure to provide a documented account of the elicitation process. By searching the literature, a framework can be developed by which knowledge of a domain can be integrated into further analysis (13). Delphi procedure is a methodology employing both quantitative and qualitative elements to gain consensus of expert opinion using iterative questionnaires (rounds) and controlled feedback (14, 15). The aim of producing a rigorous, documented, reproducible approach to domain knowledge elicitation was pursued by documenting these two processes, their integration and their application at different stages of the model development process. In this study, the data elicitation process employed in problem specification and selection of features is described, but human domain knowledge was also employed in feature engineering and pre-processing steps employed in model development. The overall methodological process employed is summarised in Figure 1.
Figure 1. Data elicitation methodology. Data elicitation goals are depicted in blue, data elicitation targets are depicted in orange, data elicitation processes are depicted in green and data elicitation uses are depicted in red.
Ethical clearance
Internal review board approval was obtained from the Health Sciences Research Ethics Committee (UFS-HSD2020/2204/2505-0003) and the Free State Department of Health (FS_202102_019). Informed consent was obtained in writing from the participants in the Delphi procedure. All data was collected anonymously directly into a REDCap® database and stored on a secure server prior to exporting as a CSV file for analysis.
Literature search
The literature search was performed using the method described by vom Brocke et al. (13). A Medline database search was conducted, and search terms and results were documented. Articles that reported risk factors for mortality in hospitalised children after 2000 were included. No laboratory values were included. Articles which were not available as full text or in English were excluded. Titles, abstracts, and full text articles were reviewed respectively. Articles were separated into articles from lower- and middle-income countries (LMIC) and high-income countries. A concept matrix was constructed, and the results were summarised.
Delphi procedure
The Delphi procedure included twelve experts from South Africa. This group was made up of three paediatric intensivists (from the University of the Free State, the University of the Witwatersrand and the private sector respectively), six specialist paediatricians (five from the University of the Free State and one from the private sector) and three specialist anaesthesiologists from the University of the Free State. All participants were anonymous from one another.
The purpose of the Delphi procedure was to set operational priorities for the model relative to the problem specification and to identify candidate features (independent variables) that are likely to be associated with the study outcome. Prior to the start of the Delphi, participants were provided with a summary of the proposed research as well as a table depicting eligible variables included in existing models.
The number of rounds was set a priori at three, based on the recommendation of Trevelyan and Robinson (14), feedback from Rounds 2 and 3 was provided as interquartile ranges and medians and consensus (used as an assessment, not an endpoint) was determined by percentage agreement. Consensus was defined as 75% agreement. For Rounds 2 and 3 a five-point Likert scale was employed (strongly disagree, disagree, no comment, agree, strongly agree). All three rounds of the Delphi were conducted using individual online REDCap® surveys.
In Round 1, the participants were asked the following open-ended questions:
1. What are the important characteristics of a variable for inclusion in a study designed to predict severe illness in children in a resource-limited setting?
2. Should completeness or practicality of data collection be given priority in data collection for this study?
3. In your clinical experience, what findings are predictive of severe illness in children?
The responses from Round 1 were summarised and feedback given to the panellists. In Rounds 2 and 3, participants were asked how strongly they agree with the inclusion of variables, considering the feedback from Round 1.
Feature selection
Candidate features for data collection were identified by the researcher (implied elicitation) by combining the findings of the literature search and Delphi procedure. It was determined that features should be applicable to a wide range of clinical settings, particularly where specialised expertise and advanced investigations are not available. To that end the following eligibility criteria were set for variables:
1. No laboratory data were included (except point of care glucose testing).
2. Variables were such that a nurse or general practitioner could be expected to collect them in clinical consultations.
3. Variables were required to be relevant to the clinical services offered in the research site.
Feature selection for data collection
Integrating the above, a list of features was selected for the data collection phase. Considering the results of Round 1 of the Delphi procedure, an approach that compromised between comprehensiveness, veracity, practicality and expected correlation with clinical outcome was chosen. Features of equivalent meaning were combined (accessory muscle use and grunting were combined as respiratory distress for example) or where a quantifiable candidate was available (such as in the case of cyanosis or peripheral oxygen saturation), the quantifiable metric was selected. For level of consciousness, both the AVPU scale (an ordinal score for consciousness—alert, response to verbal stimuli, response to pain, unresponsive) and a broader variable of altered level of consciousness were available (16). These features represent clinically detectable markers of specific organ dysfunctions. In view of the high prevalence of human immunodeficiency virus (HIV) and malnutrition in South Africa (17–19), the current status of HIV diagnosis and treatment together with anthropometry values were also included for collection as these were thought to be potentially informative features in this clinical setting.
Results
Literature search
The search procedure and results are summarised in Table 1.
Table 1. Literature search procedure.
Features that have been found to be predictive of mortality in hospitalised children were recorded together with the number of participants in each study and pathology included in the study (for example, hospitalised children in general, pneumonia, malnutrition, burns etc.). Where predictive scoring systems or models were reported [such as the Paediatric Index of Mortality (20)], the eligible features included in the model were recorded as prognostic features. These results are summarised in Figure 2 and Supplementary Figure S1. in terms of the number of publications in which specific factors are reported to be predictive of mortality. A wide range of candidate features was identified by this process. In summary, the majority of features included were some measure of neurological, cardiovascular, respiratory or other organ dysfunction (jaundice, hypoglycaemia, hypothermia).
Figure 2. Alluvial diagram of literature search results—frequency with which clinical features are reported as risk factors for mortality in hospitalised children. Where features were present in existing predictive models, each feature was recorded as one instance. No laboratory data or special investigations except capillary blood glucose were included in analysis. Features present in more than 15% of publications are included in this visualization.
The pathological categories in each group are presented in Figure 3. General populations of hospitalised children made up the majority of the literature, with malnutrition and pneumonia also being significant contributors in the LMIC group and trauma in the high-income group.
Figure 3. The frequency of pathological categories evaluated in studies reporting risk factors for mortality in hospitalised children.
Delphi
The response rate for Round 1 of the Delphi procedure was 91.67%. Free text responses were analysed thematically and collated. The responses are summarised in Table 2.
Table 2. Expert responses from round 1 of delphi procedure.
These findings were provided to participants in writing at the start of Round 2. The response rate for both Rounds 2 and 3 was 91.67%. The results of Rounds 2 and 3 are presented in Supplementary Table S1 and Figure 4.
Figure 4. Stacked bar plot of agreement on features associated with severe illness in children.
The list of features included 21 features which were collected is presented in Table 3.
Table 3. Selected features for data collection.
Discussion
In this study we have described the design of a dataset for a machine learning model designed to detect critically ill children in a hospital setting. This model aims to address the contribution of delays in identification, resuscitation and referral to avoidable severity of illness and mortality in South African Children (3). To achieve this, a documented process of literature search and expert consensus was undertaken to identify, and engineer candidate features and set operational priorities for the intended application. The taxonomy provided by Kerrigan et al. (7) provided a useful framework to represent the elicitation process. In this study, domain knowledge from experts, a literature search and the researcher were integrated to achieve the goals of problem specification and feature selection prior to data collection, feature engineering and pre-processing.
A documented and transparent Delphi procedure and literature search provide a more careful and rigorous approach to the elicitation of domain knowledge over unstructured or ad-hoc approaches to decision making (13, 15). This makes a useful contribution to the description of methods for domain knowledge elicitation in applied medical machine learning.
The elicitation of human domain knowledge is not directly addressed in the publications on machine learning models for PICU mortality prediction published by Aczon et al. and Kim et al. (10, 11). Goto et al. report the use of a priori knowledge in their model for prediction of clinical outcomes for children undergoing emergency department triage, but do not elaborate further (9).
The Transparent Reporting of multivariable prediction model of Individual Prognosis or Diagnosis (TRIPOD) statement provides a rigorous guideline for the publication of predictive models (21). The inclusion of robust descriptions of the elicitation of domain knowledge would further strengthen the ability of such guidelines to promote rigour and transparency in the reporting of predictive models and other applied machine learning models in medical research.
Conclusion
The integration of domain knowledge is an important aspect of applied machine learning in medicine. The documentation of this process for eliciting such knowledge promotes the transparent and rigorous reporting of machine learning models and should be seen as an important methodological aspect of such research. In this study we have described and documented the use of a Delphi procedure and literature search as part of a method for problem specification, data set development and feature engineering processes in model development. These models are intended for the identification of critically ill children in South Africa. This process aimed to ensure that suitable features are made available during the development, training and testing phases in order to develop a usable, context-appropriate predictive model that addresses the clinical problem of delays in the recognition, resuscitation and referral of children with severe illness.
Data availability statement
The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.
Ethics statement
The studies involving human participants were reviewed and approved by Health Sciences Research Ethics Committee University of the Free State. Written informed consent to participate in this study was provided by the participants' legal guardian/next of kin.
Author contributions
MP was the primary investigator, conducted and oversaw the execution of the study and compiled the manuscript. EG, NL and SB advised on study design, data analysis and compilation of the manuscript. JS assisted with statistical analysis. All authors contributed to the article and approved the submitted version.
Acknowledgments
The authors would like to acknowledge the National Research Foundation of South Africa, who provided funding on the Thuthuka Facility. We would also like to acknowledge the University of the Free State for assistance with open access publication fees. We acknowledge the expert participants who participated in the Delphi procedure. Finally, we acknowledge the Free State Department of Health and Pelonomi Tertiary Hospital who accommodated the conduct of this clinical research in their facility.
Conflict of interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Publisher's note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
Supplementary material
The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fped.2023.1005579/full#supplementary-material.
References
1. Turner EL, Nielsen KR, Jamal SM, von Saint André-von Arnim A, Musa NL. A review of pediatric critical care in resource-limited settings: a look at past, present, and future directions. Front Pediatr. (2016) 4:1–15. doi: 10.3389/fped.2016.00005
2. Nolan T, Angos P, Cunha AJ, Muhe L, Qazi S, Simoes EA, et al. Quality of hospital care for seriously ill children in less-developed countries. Lancet. (2001) 357(9250):106–10. doi: 10.1016/S0140-6736(00)03542-X
3. Hodkinson P, Argent A, Wallis L, Reid S, Perera R, Harrison S, et al. Pathways to care for critically ill or injured children: a cohort study from first presentation to healthcare services through to admission to intensive care or death. PLoS One. (2016) 11(1):1–17. doi: 10.1371/journal.pone.0145473
4. Rajkomar A, Dean J, Kohane I. Machine learning in medicine. N Engl J Med. (2019) 380(14):1347–58. doi: 10.1056/NEJMra1814259
5. Lonsdale H, Jalali A, Ahumada L, Matava C. Machine learning and artificial intelligence in pediatric research: current state, future prospects, and examples in perioperative and critical care. J Pediatr. (2020) 221:S3–S10. doi: 10.1016/j.jpeds.2020.02.039
6. Deng C, Ji X, Rainey C, Zhang J, Lu W. Integrating machine learning with human knowledge. iScience. (2020) 23(11):101656. doi: 10.1016/j.isci.2020.101656
7. Kerrigan D, Hullman J, Bertini E. A survey of domain knowledge elicitation in applied machine learning. Multimodal Technol Interact. (2021) 5(12):1–19. doi: 10.3390/mti5120073
8. O’hagan A. Expert knowledge elicitation: subjective but scientific. Am Stat. (2019) 2019(S1):69–81. doi: 10.1080/00031305.2018.1518265
9. Goto T, Camargo CA Jr, Faridi MK, Freishtat RJ, Hasegawa K. Machine learning-based prediction of clinical outcomes for children during emergency department triage. JAMA Netw Open. (2019) 2(1):e186937. doi: 10.1001/jamanetworkopen.2018.6937
10. Aczon MD, Ledbetter DR, Laksana E, Ho LV, Wetzel RC. Continuous prediction of mortality in the PICU: a recurrent neural network model in a single-center dataset. Pediatr Crit Care Med. (2021) 22(6):519–29. doi: 10.1097/PCC.0000000000002682
11. Kim SY, Kim S, Cho J, Kim YS, Sol IS, Sung Y, et al. A deep learning model for real-time mortality prediction in critically ill children. Crit Care. (2019) 23(1):279. doi: 10.1186/s13054-019-2561-z
12. Pienaar MA, Sempa JB, Luwes N, George EC, Brown SC. Development of artificial neural network models for paediatric critical illness in South Africa. Front Pediatr. (2022) 10:1–13. doi: 10.3389/fped.2022.1008840
13. vom Brocke JM, Simons A, Niehaves B, Riemer K, Plattfaut R, Cleven A. Reconstructing the giant: on the importance of rigour in documenting. 17th European conference on information systems; (2013). p. 1–13
14. Trevelyan EG, Robinson N. Delphi methodology in health research: how to do it? Eur J Integr Med. (2015) 7(4):423–8. doi: 10.1016/j.eujim.2015.07.002
15. Rowe G, Wright G. The delphi technique as a forecasting tool. Int J Forecast. (1999) 2070(99):353–75. doi: 10.1016/S0169-2070(99)00018-7
16. George EC, Walker AS, Kiguli S, Olupot-Olupot P, Opoka RO, Engoru C, et al. Predicting mortality in sick African children: the FEAST paediatric emergency triage (PET) score. BMC Med. (2015) 13:174. doi: 10.1186/s12916-015-0407-3
17. Rollins NC, Ndirangu J, Bland RM, Coutsoudis A, Coovadia HM, Newell ML. Exclusive breastfeeding, diarrhoeal morbidity and all-cause mortality in infants of HIV-infected and HIV uninfected mothers: an intervention cohort study in KwaZulu natal, South Africa. PLoS One. (2013) 8(12):e81307. doi: 10.1371/journal.pone.0081307
18. Slogrove A, Reikie B, Naidoo S, de Beer C, Ho K, Cotton M, et al. HIV-exposed uninfected infants are at increased risk for severe infections in the first year of life. J Trop Pediatr. (2012) 58(6):505–8. doi: 10.1093/tropej/fms019
19. Msemburi W, Pillay-van Wyk V, Dorrington R, Neethling I, Nannan N, Groenewald P, et al. Second national burden of disease study for South Africa: Cause-of-death profile for South Africa, 1997–2012 (2016). p. 1–23. Available at: http://www.mrc.ac.za/sites/default/files/files/2016-12-08/SouthAfrica2012.pdf.
20. Straney L, Clements A, Parslow RC, Pearson G, Shann F, Alexander J, et al. Paediatric index of mortality 3: an updated model for predicting mortality in pediatric intensive care. Pediatr Crit Care Med. (2013) 14(7):673–81. doi: 10.1097/PCC.0b013e31829760cf
Keywords: machine learning, critical care, children, domain knowledge, triage, severity of illness
Citation: Pienaar MA, Sempa JB, Luwes N, George EC and Brown SC (2023) Elicitation of domain knowledge for a machine learning model for paediatric critical illness in South Africa. Front. Pediatr. 11:1005579. doi: 10.3389/fped.2023.1005579
Received: 28 July 2022; Accepted: 25 January 2023;
Published: 21 February 2023.
Edited by:
Stephen Aronoff, Temple University, United StatesReviewed by:
Monty Mazer, Rainbow Babies & Children's Hospital, United StatesNicole Rübsamen, University of Münster, Germany
© 2023 Pienaar, Sempa, Luwes, George and Brown. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Michael A. Pienaar cGllbmFhcm1hMUB1ZnMuYWMuemE=
Specialty Section: This article was submitted to Pediatric Critical Care, a section of the journal Frontiers in Pediatrics