A systematic literature review of AI-based digital decision support systems for post-traumatic stress disorder

Objective: Over the last decade, an increase in research on medical decision support systems has been observed. However, compared to other disciplines, decision support systems in mental health are still in the minority, especially for rare diseases like post-traumatic stress disorder (PTSD). We aim to provide a comprehensive analysis of state-of-the-art digital decision support systems (DDSSs) for PTSD.

Methods: Based on our systematic literature review of DDSSs for PTSD, we created an analytical framework using thematic analysis for feature extraction and quantitative analysis for the literature. Based on this framework, we extracted information around the medical domain of DDSSs, the data used, the technology used for data collection, user interaction, decision-making, user groups, validation, decision type and maturity level. Extracting data for all of these framework dimensions ensures consistency in our analysis and gives a holistic overview of DDSSs.

Results: Research on DDSSs for PTSD is rare and primarily deals with the algorithmic part of DDSSs (n = 17). Only one DDSS was found to be a usable product. From a data perspective, mostly checklists or questionnaires were used (n = 9). While the median sample size of 151 was rather low, the average accuracy was 82%. Validation, excluding algorithmic accuracy (like user acceptance), was mostly neglected, as was an analysis concerning possible user groups.

Conclusion: Based on a systematic literature review, we developed a framework covering all parts (medical domain, data used, technology used for data collection, user interaction, decision-making, user groups, validation, decision type and maturity level) of DDSSs. Our framework was then used to analyze DDSSs for post-traumatic stress disorder. We found that DDSSs are not ready-to-use products but are mostly algorithms based on secondary datasets. This shows that there is still a gap between technical possibilities and real-world clinical work.

Introduction

According to Sauter, Digital Decision Support Systems (DDSSs) are computer-based systems that bring together information from various sources, assist in the organization and analysis of information and facilitate the evaluation of assumptions underlying the use of specific models (1). The concept of decision support systems originated in the 1960s (2) when researchers began to study computerized methods to assist in decision-making (3–5). Since then, the idea has extended throughout a broad spectrum of domains, one of which is healthcare. This work focuses on decision support systems in mental health, more precisely on decision support systems for PTSD. The American Psychiatric Association defines PTSD as “a psychiatric disorder that can occur in people who have experienced or witnessed a traumatic event such as a natural disaster, a serious accident, a terrorist act, war/combat, rape or other violent personal assault” (6). People with PTSD experience recurrent thoughts about their traumatic experience that influence their daily life. The lifetime prevalence of PTSD is around 12.5% (7). However, people suffering from PTSD are often undiagnosed or misdiagnosed, resulting in incorrect, incomplete or missing treatment (8). To investigate whether DDSSs could be a solution to this problem, we aim to review available decision support systems for PTSD and map their technological approaches in order to understand possible research gaps and obstacles in introducing decision support systems to clinical processes. Since no available reference architecture for decision support systems is applicable to our research, we contribute by introducing a novel framework for decision support systems that can be used to analyze existing systems. Ultimately, this also accelerates the development of new systems by highlighting essential dimensions.

Designers of earlier DDSSs have applied multiple alternative approaches for converting real-world data into something that stimulates better decisions. Information-management-based DDSSs try to organize data into usable presentations; modeling-(or data-analytics)-based DDSSs attempt to apply statistical (learning) methods for finding patterns or calculating indicators; and knowledge-management-based systems apply externally prepared algorithms (expert rules) to find matching data or derive new facts (9). While AI has been an essential element of DDSSs throughout its history, only recently has a new generation of decision support been facilitated by the availability of powerful computing tools to properly manage big data and to analyze and generate new knowledge. The evaluation of AI’s earlier implementations was limited to the design and development phase; machine learning-based algorithms often do not generalize beyond the training data set (10). However, studies have still shown the benefits of machine learning algorithms in DDSSs (11–13). Current studies that test the application of healthcare AI algorithms often omit details of DDSS tools that apply AI models. A well-designed DDSS is likely to enable the real-world application of AI technology (14).

This review aims to contribute by introducing a framework for the features of DDSS implementation in mental health. We aim to identify the prevalent features of the current state of research on DDSS. Often, the development of information systems involves the continuous introduction of new features and quality improvements. We hypothesized that each available article presents only a selection of features, a selection which is dependent on the maturity of the DDSS. Maturity models are increasingly used as a means of benchmarking or self-assessment of development (15). In healthcare informatics, many maturity models are available [e.g., Hospital Information System Maturity Model (16)], but none of these models strictly provides an informed approach for the assessment of research on decision support systems (17). The available maturity models instead tend to look at the level of organizational adoption of specific technologies (e.g., how much an organization values data analytics technology) and provide little support for deciding on the readiness of DDSS tools in their early phases of development. As AI is often an essential element of a DDSS, we also explored AI maturity models. AI maturity models mostly look into the level of AI adoption in an organization rather than the maturity of the AI technology itself (18–20).

A DDSS is not a single technology but rather a set of integrated technologies (21–25). Sauser et al. (26) suggested a measure of System Readiness Level (SRL), which expresses the level of maturity of a system consisting of a set of integrated technologies (26). Exploring AI technology readiness or maturity, we encountered suggestions to look separately into the AI system’s capacities of integrating existing data sources (machine-machine intelligence), interacting with human users (human-computer intelligence) and applying intelligent reasoning (core cognitive intelligence) (27).

Methods

To have a transparent and objective approach for this literature review, we decided to apply the five stages suggested by Kitchenham’s “Guidelines for performing Systematic Literature Reviews in Software Engineering” (28):

(1) Search Strategy

(2) Study Selection

(3) Study Quality Assessment

(4) Data Extraction

(5) Data Synthesis

Research questions

Since our aim is to understand current research on decision support systems for PTSD, this paper is based on two research questions. First, we look for state-of-the-art decision support systems for post-traumatic stress disorder (RQ1). Second, we investigate the component elements of current decision support systems for PTSD (RQ2).

Search strategy

We built a search string based on the research questions identified and applied it to the Scopus abstract and citation database. Scopus was chosen as the primary source because it is the largest abstract and citation database of research literature with 100% MEDLINE coverage (29). The initial search string consisted of the disease to investigate – post-traumatic stress disorder – its abbreviation PTSD as well as the term “decision support.” To find papers that covered the prediction and classification of PTSD, we also added Artificial Intelligence. In Scopus, we applied the search string to the title, abstract and tags of the research papers. We restricted our search to only include journal articles or conference proceedings in English. We also conducted a manual search using Google Scholar and the web to find additional research; however, this did not bring up any new articles not already covered by our database search and our reference screening process. We formed our search criteria as (“decision support” OR “Artificial Intelligence”) AND [PTSD OR (post AND traumatic AND stress AND disorder)].

We conducted the search in Scopus on 3 March 2021. It resulted in 75 papers; reference screening of the included literature brought up an additional 13 papers. Our search process is visualized in Figure 1.

FIGURE 1

Figure 1. Search strategy.

Study selection

The titles and abstracts of the queried articles were analyzed to identify relevant articles from the results of the search string queries. Articles fitting the research questions and meeting the inclusion criteria (see section “inclusion criteria”) as well as the quality criteria (see section “study quality assessment”) were included. Since the goal of this research is to give an overview of the state of the art, we did not put any constraints on study types and designs. To reduce bias in the study selection process, the task was done by two researchers independently. The two result sets were then merged and deviations were discussed among the authors. This resulted in a total set of 17 research papers.

We then repeated this process step to extract relevant studies from the reference lists of the selected articles. This resulted in 13 new research papers.

Inclusion criteria

Table 1 presents the inclusion criteria applied to the articles in our review (Inclusion criteria).

TABLE 1

Table 1. Inclusion criteria.

Study quality assessment

Table 2 presents the inclusion criteria applied to the articles in our review (Quality criteria).

TABLE 2

Table 2. Quality criteria.

Data extraction and synthesis

Data extraction and synthesis were based on an inductive approach. We applied thematic analysis (30) to answer our research questions. First, clear, scoped questions for data extraction were formed. Two researchers read through all the articles and iteratively clustered all of the information available on decision support systems into the extraction parameters. These extraction parameters describe how decision support systems work. This process is shown in Figure 2.

FIGURE 2

Figure 2. Extraction process.

The answers extracted from the EQs (see Table 3) were then combined upon the agreement of the authors to create a feature matrix. The extracted features were then further clustered to create a common terminology that allows further analysis and the possibility to compare results. In the end, we combined the developed extraction questions and the clustered scales of each question into a novel framework for decision support systems in mental health.

TABLE 3

Table 3. Extraction questions (EQ).

Results

The selected 30 research articles (31–60) were published between 2001 and 2019. Three articles were published in journals about medical informatics, 10 in computer science journals or proceedings and 17 in medical journals. The following table shows how often each extraction parameter was present and indicates the terminology used in the selected studies. The terminology shown in Table 4 was developed by manual, iterative clustering of the extracted features until the authors were satisfied with the granularity.

TABLE 4

Table 4. Terminology extraction.

A framework for digital decision support systems

Based on our aim to find all relevant features of decision support systems in the PTSD area and our systematic literature review results, we propose a multidimensional framework that covers the different areas of DDSS. Each dimension represents one of our extraction parameters. Figure 3 illustrates our framework with the different dimensions of DDSSs. Based on the extracted data, we clustered the terminology to develop scales for dimensions in order to make results better analyzable.

FIGURE 3

Figure 3. Framework for DDSS.

Input Data: The input data dimension defines the information needed by a decision support system in order to function. Possible data could be structured like socio-demographic information or coded data [for example, with the International Statistical Classification of Diseases and Related Health Problems (ICD) (61) or the Diagnostic and Statistical Manual of Mental Disorders (DSM) (62)] as well as semi-structured information like patient records or unstructured information like free text or medical images. A combination of different structured, semi-structured and/or unstructured data is also possible.

Technology: The technology dimension describes how the decision support system is implemented. This involves three sub-dimensions:

Decision technology: The decision technology explains the intelligence of the cognition of the system. This is the algorithm that powers the decision-making. Examples are different machine learning algorithms such as support vector machines or other statistical methods as well as rule-based approaches.

Interaction technology: This sub-dimension describes the technology needed to interact with other systems or user groups in the clinical process. Interaction technology can be API-based interfaces to systems, graphical user interfaces (websites, mobile apps) or sensory input like conversational interfaces (chatbots).

Data collection technology: The data collection technology sub-dimension defines how the data described in the input data dimension are collected. Examples are instance sensors, questionnaires or chatbots.

Validation: Validation describes how the success of decision support systems is measured.

Accuracy: The decision support system is evaluated by how many right or wrong decisions it makes. Examples are accuracy, recall (sensitivity), precision, specificity, area under the curve (AUC) values and F1 scores (harmonic mean of recall and precision).

User acceptance: End-users are involved in the evaluation of the DDSS.

Efficacy: The impact of the decision support system is evaluated based on potential benefits.

Security: The DDSS is evaluated against security regulations.

Legal: The legal compliance of the DDSS is evaluated.

User group: This dimension captures the different user groups interacting with the decision support system in the clinical process.

Medical domain: The medical domain dimension describes the disease for which the decision support system can be applied.

Decision: The following scale defines the decisions a digital decision support system can support:

Prediction: The system outputs a risk score based on the likelihood that someone gets a disease.

Assessment: The patient is already sick (knowingly or unknowingly).

Diagnosis: Testing individuals with symptoms and/or suspicion of illness

Screening: Testing for individuals without specific symptoms

Monitoring: Decision support that evaluates symptom severity or treatment progress

Treatment: Recommendation or intervention concerning care or therapy

Maturity: As none of the existing maturity models fits our research, we designed a DDSS maturity model based on the SLR scale (26), but with adaptions specific to healthcare. It introduces additional gradation for noticing the moment where human interaction is added to the core AI algorithm. Our maturity levels describe on a scale from one to seven how advanced the DDSS is. Not all of the abovementioned dimensions are necessarily present in each of the maturity levels. As the maturity level gets higher, more dimensions are described.

1. Idea without implementation

2. Implementation without real-world interaction (algorithm development)

3. Implementation with real-world interaction but without patient intervention

4. Fully functioning prototype, system triggers real-world action, e.g., clinical trial

5. Operational product (at least one adopter, certified if required)

6. Locally adopted product

7. World-wide adopted product (transformational).

Data synthesis input data (EQ1)

The data used by digital decision support systems in the context of PTSD is diverse. Voice data (35, 45, 46, 55), text data (38, 48, 50), checklists and questionnaires (32, 33, 37, 41–43, 52, 53, 59), bio signals (32, 33, 36, 44, 45, 51, 57) and electronic medical records (34, 47, 56) as well as secondary data from other clinical studies (31, 40, 49, 54) are used. One article used the choices made by a virtual avatar in a role-playing game as input data (39). Of the 30 publications included in this review, 28 mentioned the sample size of the data they used to develop and test their decision support system. The minimum sample size was 10, and the maximum was 89,840 with a median (IQR) m = 151.5 (54.25 to 656.25). The violin plots (Figures 4, 5) below show the distribution of the sample size. The top three outliers (89,840; 89,840; 5,972) were neglected in Figure 5 for better visibility.

FIGURE 4

Figure 4. Sample size distribution.

FIGURE 5

Figure 5. Sample size distribution excluding outliers.

Figure 6 shows the data dimension of the studies in our review and indicates how the data used correlate with the average maturity levels of the DDSS. It visualizes the frequency and maturity of DDSSs based on the different data sources.

FIGURE 6

Figure 6. Data dimension concepts.

Data synthesis implementation (EQ2)

The majority (n = 15) of the investigated research uses a neural network approach (including support vector machines) in their systems. In 11 cases, support vector machines (SVM) were used. Other algorithms used were regressions, decision trees, random forest and rule-based approaches. We observed that 20 research papers did not have or mention any user interaction but worked solely on secondary data. The others used questionnaires or surveys, virtual humans or virtual reality. McWorther et al. proposed using temperature control, aromatherapy and auditory therapy capabilities for user interaction (36). Concerning maturity levels, AI algorithms are still mostly on maturity level two. Most advanced in terms of maturity were statistical methods and text mining methods, as indicated in Figure 7. The categories “statistics” and “machine learning” (ML) arose because some studies mentioned only these broad categories without further specifics.

FIGURE 7

Figure 7. Decision technology concepts.

Data synthesis validation (EQ3)

The majority (n = 23) of articles validated the accuracy of the DDSS studied. Three articles validated user acceptance, two validated efficacy and three did not mention validation. Comparing algorithmic validation among research papers was difficult since a variety of scores, such as F1 scores, area under the receiver operating curve (63) or overall accuracy, were used and they cannot be converted. To be able to provide an estimation of how well current DDSSs perform, we extracted all accuracy measurements present in each paper and aggregated each scale individually. The mean accuracy (n = 11) of the DDSSs is μ = 82.2% with a median of η = 82% and a standard deviation of σ = 0.095. The mean area under the curve value (n = 8) is μ = 0.845 with a median of η = 0.84 and a standard deviation of σ = 0.064.

Data synthesis user groups (EQ4)

The user groups mentioned were patients, clinicians and supporters of patients; however, the majority of papers did not explicitly mention specific user groups for their systems. Research covering decision support systems with higher maturity levels (four and above) included this information. Research dealing with decision support systems with lower maturity often lacked a clear user group since the process of using the proposed systems was not defined at that stage.

Data synthesis medical domain (EQ5)

In addition to PTSD, which was tackled by all 30 research papers, four investigated depression (46–48, 55), two anxiety (34, 48) and one paranoia (58).

Data synthesis decisions supported (EQ6)

Research focusing on predicting PTSD or its symptoms was most common (n = 11). Six papers focused on screening (35, 38, 45, 46, 50, 55) and six on treatment (32, 36, 43, 51, 53, 56). Four papers investigated the diagnosis of PTSD (37, 41, 52, 60) and five focused on monitoring PTSD (33, 35, 56, 58, 59).

Data synthesis maturity level (EQ7)

The decision support systems were ranked according to the maturity scale described in see section “a framework for digital decision support systems.” As stated by answering research question two, the majority of papers work with secondary data. This is supported by the high volume of research with a maturity level of two. Figure 8 shows the number of articles grouped by maturity level.

FIGURE 8

Figure 8. Bar chart – maturity levels.

Discussion

This research highlights the state of the art in digital decision support systems for PTSD based on our proposed framework. We developed the framework to ensure a holistic overview of all features of a DDSS. The dimensions of the framework represent the topics of interest and the choice of features is based on the conceptualization of the terminology extracted from the included articles dimension by dimension.

Concerning the data dimension, we noticed that questionnaires and checklists are still the most common and most mature (see Figure 6) input for decision support systems. When examining clinical guidelines like NICE (64) for diagnosing PTSD, questionnaires and checklists are still the only approach mentioned for diagnostics. Even though some new technologies, such as virtual or augmented reality, were investigated in the research found in this review, we noticed an absence of input parameters based on smartphones or wearables like GPS sensors or accelerometers. We hypothesize that this is due to the short life cycle of modern technologies, making it difficult to offer clinical evidence of their benefits. Questionnaires and checklists, however, have been around for many years and the methodology for administering them has not changed, therefore there is more scientific evidence of their use. Researchers and medical professionals are more likely to research, invest and adopt technology with strong evidence. This could be another reason why DDSSs using new technology are not widely included in clinical processes.

The data dimension also showed that the sample size is on average small and the statistical significance of the results was not proven by the majority of the research articles. Several reasons contribute to this. In general, medical data are hard to obtain for research because secondary use is still not easy with many digital healthcare records and/or applications. Even if data can be obtained, they need to include the right parameters and have a structure that is usable for AI algorithms. Unstructured and text-based information is especially challenging to use for an AI. Further, most available datasets like the Jerusalem Trauma Outreach and Prevention Study do not include data on modern sensors (65).

The most common AI algorithm found during this literature review was support vector machines. Over the last few years, they have been developed to a de facto standard because they are easy to use, have good library support for programming and have low assumptions on the training data. We also observed that the number of research items resulting in usable products (maturity level ≥ 4) was low in three articles. Clinical studies with patient intervention (maturity level ≥ 3) were relatively low in nine papers out of 30. One reason for this could be that the small sample size of the research items does not provide sufficient evidence for clinical use.

All articles with a maturity level of 4 or more had, as one focus, validation of user acceptance and clearly defined user groups. Most articles with lower maturity levels did not have defined user groups. This could indicate a lack of strategic development and difficulties in bringing the research to a clinical setting. Our hypothesis is that interaction with users or integration into clinical processes is often much more challenging to solve than intelligence of cognition. Still, most papers focus on cognition, not user interaction; our framework’s validation dimension is evidence of this. We found 23 papers evaluating accuracy, which is an evaluation of AI technology, and five papers evaluating user acceptance or efficacy, meaning that they attempted to improve the current clinical process. Since most papers in our review are of maturity levels 1, 2 or 3 (meaning algorithm research), they do not include the clinical component necessary for user acceptance and efficacy evaluation. This shows a research gap when it comes to the enrichment of clinical processes with IT. The same goes for evaluating legal and IT-security constraints, which were not mentioned by any paper in our review. Since eHealth systems are getting increasingly focused by cyber attacks (66), IT and data security need to be a vital part of the evaluation to allow a safe DDSS adoption.

Further research has to be conducted on how the clinical process needs to be adapted for DDSSs to work, also in the context of the supported decisions. Most DDSS designers do not really understand the medical decision process but provide decisions in an “IT way.” One limitation of this general hypothesis is that our research focuses solely on DDSS for PTSD. However, the narrow approach to include only PTSD shows that even in a very well-scoped area, a DDSS is hard to implement.

Since we used an inductive research approach to design our framework based on currently available literature, some important framework dimensions might be missing. One example is that the framework includes many technical aspects of the implementations and fewer organizational and financial perspectives. We encourage further research to include dimensions that describe the adoption of DDSSs in clinical processes.

Introducing our novel framework for DDSS, we provide a guide for decision support system evaluation. The framework is complementary to other healthcare technology evaluation methods (clinical, organizational, financial) and thus supports the design of comprehensive evaluation systems for DDSSs. Applying the maturity dimension helped us to examine what features of a DDSS are present, thereby indicating the steps to take in order to move up in maturity when developing decision support systems. Since the framework was developed out of general considerations, it can be applied to decision support systems outside of PTSD or mental health. However, it should be further evaluated to examine whether the terminology suits other domains. Higher maturity scales in particular need additional verification, since only two papers in our review had a maturity level above 4.

Conclusion

Our research aimed to analyze existing decision support systems for PTSD. Based on this goal, we developed a generic framework covering all dimensions of digital decision support systems. Our framework not only accelerates the development and benchmarking of DDSSs, but also acts as the foundation for our systematic literature review. Extracting data for all framework dimensions ensures consistency in our analysis and gives a holistic overview of DDSSs. During our review, we found working DDSS prototypes for PTSD and described their components. However, most of the systems are not evaluated in production use; they are only algorithmic models based on secondary datasets. This shows that there is still a gap between technical possibilities and actual clinical work. We proposed some possible explanations: small sample size, missing domain expertise, lack of focus to bring research to production. However, this gap should be analyzed further by testing our hypothesis and examining it with data from research on DDSSs for other mental diseases. For now, we conclude that only a few rare DDSSs for PTSD are ready for large-scale adoption in healthcare. The long-promised revolution of AI and ML for diagnosis in psychiatry, at least for PTSD, is yet to come.

Data availability statement

The original contributions presented in this study are included in the article/supplementary material, further inquiries can be directed to the corresponding author.

Author contributions

MB: conceptualization, methodology, investigation, resources, data curation, and writing – original draft. JM: investigation, data curation, and writing – original draft. PR: writing, review, editing, and supervision. All authors contributed to the article and approved the submitted version.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

1. Sauter VL. Decision support systems for business intelligence. Hoboken, NJ: John Wiley & Sons (1997). p. 618.

Google Scholar

2. Power DJ. Decision support systems: A historical overview. In: FW Burstein, C Holsapple editors. Handbook on decision support systems 1: Basic themes. (Berlin: Springer) (2008). p. 121–40. doi: 10.1007/978-3-540-48713-5_7

CrossRef Full Text | Google Scholar

3. Raymond RC. Use of the time-sharing computer in business planning and budgeting. Manag Sci. (1966) 12:B–363. doi: 10.1287/mnsc.12.8.B363

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Turban E. The use of mathematical models in plant maintenance decision making. Manag Sci. (1967) 13:B–342. doi: 10.1287/mnsc.13.6.B342

PubMed Abstract | CrossRef Full Text | Google Scholar

5. Holt CC, Huber GP. A computer aided approach to employment service placement and counseling. Manag Sci. (1969) 15:573–94. doi: 10.1287/mnsc.15.11.573

PubMed Abstract | CrossRef Full Text | Google Scholar

6. American Psychiatric Association. What is PTSD? Virginia, VA: American Psychiatric Association (2020).

Google Scholar

7. Spottswood M, Davydow DS, Huang H. The prevalence of posttraumatic stress disorder in primary care: A systematic review. Harv Rev Psychiatry. (2017) 25:159–69. doi: 10.1097/HRP.0000000000000136

PubMed Abstract | CrossRef Full Text | Google Scholar

8. Meltzer EC, Averbuch T, Samet JH, Saitz R, Jabbar K, Lloyd-Travaglini C, et al. Discrepancy in diagnosis and treatment of post-traumatic stress disorder (PTSD): Treatment for the wrong reason. J Behav Health Serv Res. (2012) 39:190–201. doi: 10.1007/s11414-011-9263-x

PubMed Abstract | CrossRef Full Text | Google Scholar

9. Sen A, Banerjee A, Sinha AP, Bansal M. Clinical decision support: Converging toward an integrated architecture. J Biomed Inform. (2012) 45:1009–17. doi: 10.1016/j.jbi.2012.07.001

PubMed Abstract | CrossRef Full Text | Google Scholar

10. Magrabi F, Ammenwerth E, McNair JB, Keizer NFD, Hyppönen H, Nykänen P, et al. Artificial intelligence in clinical decision support: Challenges for evaluating AI and practical implications. Yearb Med Inform. (2019) 28:128–34. doi: 10.1055/s-0039-1677903

PubMed Abstract | CrossRef Full Text | Google Scholar

11. Manogaran G, Lopez D. A survey of big data architectures and machine learning algorithms in healthcare. Int J Biomed Eng Technol. (2017) 25:182–211. doi: 10.1504/IJBET.2017.087722

PubMed Abstract | CrossRef Full Text | Google Scholar

12. Liang Z, Zhang G, Huang JX, Hu QV. Deep learning for healthcare decision making with EMRs. In: Proceedings of the 2014 IEEE International Conference on Bioinformatics and Biomedicine (BIBM). Belfast: (2014). p. 556–9. doi: 10.1109/BIBM.2014.6999219

CrossRef Full Text | Google Scholar

13. Miotto R, Wang F, Wang S, Jiang X, Dudley JT. Deep learning for healthcare: Review, opportunities and challenges. Brief Bioinform. (2018) 19:1236–46. doi: 10.1093/bib/bbx044

PubMed Abstract | CrossRef Full Text | Google Scholar

14. Buchlak QD, Esmaili N, Leveque JC, Farrokhi F, Bennett C, Piccardi M, et al. Machine learning applications to clinical decision support in neurosurgery: An artificial intelligence augmented systematic review. Neurosurg Rev. (2019) 43:1235–53. doi: 10.1007/s10143-019-01163-8

PubMed Abstract | CrossRef Full Text | Google Scholar

15. Mettler T, Rohner P, Winter R. Towards a classification of maturity models in information systems. In: A D’Atri, M De Marco, AM Braccini, F Cabiddu editors. Management of the interconnected world. (Heidelberg: Physica-Verlag) (2010). p. 333–40. doi: 10.1007/978-3-7908-2404-9_39

CrossRef Full Text | Google Scholar

16. Carvalho JV, Rocha Á, Abreu A. HISMM – Hospital information system maturity model: A synthesis. In: J Mejia, M Muñoz, Á Rocha, T San Feliu, A Peña editors. Trends and applications in software engineering. (Cham: Springer International Publishing) (2017). p. 189–200. doi: 10.1007/978-3-319-48523-2_18

CrossRef Full Text | Google Scholar

17. Gomes J, Romão M. Information system maturity models in healthcare. J Med Syst. (2018) 42:235. doi: 10.1007/s10916-018-1097-0

PubMed Abstract | CrossRef Full Text | Google Scholar

18. Lichtenthaler U. Five maturity levels of managing AI: From isolated ignorance to integrated intelligence. J Innov Manag. (2020) 8:39–50. doi: 10.24840/2183-0606_008.001_0005

CrossRef Full Text | Google Scholar

19. Ellefsen APT, Oleśków-Szłapka J, Pawłowski G, Toboła A. Striving for excellence in AI implementation: AI maturity model framework and preliminary research results. LogForum. (2019) 15:363–76. doi: 10.17270/J.LOG.2019.354

CrossRef Full Text | Google Scholar

20. Saari L, Kuusisto O, Pirttikangas S. AI maturity web tool helps organisations proceed with AI. (2019). Available online at: https://cris.vtt.fi/en/publications/ai-maturity-web-tool-helps-organisations-proceed-with-ai (accesed June 19, 2020).

Google Scholar

21. Loya SR, Kawamoto K, Chatwin C, Huser V. Service oriented architecture for clinical decision support: A systematic review and future directions. J Med Syst. (2014) 38:140. doi: 10.1007/s10916-014-0140-z

PubMed Abstract | CrossRef Full Text | Google Scholar

22. El-Sappagh SH, El-Masri S. A distributed clinical decision support system architecture. J King Saud Univers Comput Inform Sci. (2014) 26:69–78. doi: 10.1016/j.jksuci.2013.03.005

CrossRef Full Text | Google Scholar

23. Basilakis J, Lovell NH, Celler BG. A decision support architecture for telecare patient management of chronic and complex disease. In: Proceedings of the 2007 29th Annual International Conference of the IEEE Engineering in Medicine and Biology Society. (Manhattan, NY: IEEE) (2007). doi: 10.1109/IEMBS.2007.4353296

PubMed Abstract | CrossRef Full Text | Google Scholar

24. Welch BM, Loya SR, Eilbeck K, Kawamoto KA. Proposed clinical decision support architecture capable of supporting whole genome sequence information. J Pers Med. (2014) 4:176–99. doi: 10.3390/jpm4020176

PubMed Abstract | CrossRef Full Text | Google Scholar

25. Lin HC, Wu HC, Chang CH, Li TC, Liang WM, Wang JYW. Development of a real-time clinical decision support system upon the web mvc-based architecture for prostate cancer treatment. BMC Med Inform Decis Mak. (2011) 11:16. doi: 10.1186/1472-6947-11-16

PubMed Abstract | CrossRef Full Text | Google Scholar

26. Sauser B, Verma D, Ramirez-Marquez J, Gove R. From TRL to SRL: The concept of systems readiness levels. In: Proceedings of the Conference on Systems Engineering Research. Los Angeles, CA: (2006).

Google Scholar

27. Dataversity. A pragmatic AI maturity model. (2020). Available online at: https://www.slideshare.net/Dataversity/a-pragmatic-ai-maturity-model (accessed May 21, 2020).

Google Scholar

28. Kitchenham B, Charters S. Guidelines for performing systematic literature reviews in software engineering. (2007). Available online at: https://www.elsevier.com/__data/promis_misc/525444systematicreviewsguide.pdf (accessed June 30, 2022).

Google Scholar

29. Falagas ME, Pitsouni EI, Malietzis GA, Pappas G. Comparison of pubmed, scopus, web of science, and google scholar: Strengths and weaknesses. FASEB J. (2008) 22:338–42. doi: 10.1096/fj.07-9492LSF

PubMed Abstract | CrossRef Full Text | Google Scholar

30. Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol. (2006) 3:77–101. doi: 10.1191/1478088706qp063oa

PubMed Abstract | CrossRef Full Text | Google Scholar

31. Ma S, Galatzer-Levy IR, Wang X, Fenyö D, Shalev AY. A first step towards a clinical decision support system for post-traumatic stress disorders. AMIA Annu Symp Proc. (2016) 2016:837–43.

Google Scholar

32. Barish G, Aralis H, Elbogen E, Lester P. A mobile app for patients and those who care about them: A case study for veterans with PTSD + anger. In: Proceedings of the 13th EAI International Conference on Pervasive Computing Technologies for Healthcare. (Trento: Association for Computing Machinery) (2019). p. 1–10. doi: 10.1145/3329189.3329248

CrossRef Full Text | Google Scholar

33. Mallol-Ragolta A, Dhamija S, Boult TE. A multimodal approach for predicting changes in PTSD symptom severity. In: ICMI – Proceedings of the International Conference Multimodal Interact. (New York, NY: Association for Computing Machinery, Inc) (2018). p. 324–33. doi: 10.1145/3242969.3242981

CrossRef Full Text | Google Scholar

34. Dabek F, Caban JJ. A neural network based model for predicting psychological conditions. (2015). Available online at: https://www.scopus.com/inward/record.uri?eid=2-s2.0-84945954195&doi=10.1007%2f978-3-319-23344-4_25&partnerID=40&md5=46be7a873337815b5db9d904941d4f1c (accessed June 30, 2022).

Google Scholar

35. Xu R, Mei G, Zhang G, Gao P, Judkins T, Cannizzaro M, et al. A voice-based automated system for PTSD screening and monitoring. Stud Health Technol Inform. (2012) 173:552–8.

Google Scholar

36. McWhorter J, Brown L, Khansa L. A wearable health monitoring system for posttraumatic stress disorder. Biol Inspired Cogn Archit. (2017) 22:44–50. doi: 10.1016/j.bica.2017.09.004

CrossRef Full Text | Google Scholar

37. Omurca SI, Ekinci E. An alternative evaluation of post traumatic stress disorder with machine learning methods. In: Proceedings of the INISTA – International Symposium on Innovations in Intelligent SysTems and Applications. (Piscataway, NJ: Institute of Electrical and Electronics Engineers Inc) (2015). doi: 10.1109/INISTA.2015.7276754

CrossRef Full Text | Google Scholar

38. He Q, Veldkamp BP, Glas CAW, de Vries T. Automated assessment of patients’ self-narratives for posttraumatic stress disorder screening using natural language processing and text mining. Assessment. (2017) 24:157–72. doi: 10.1177/1073191115602551

PubMed Abstract | CrossRef Full Text | Google Scholar

39. Myers CE, Radell ML, Shind C, Ebanks-Williams Y, Beck KD, Gilbertson MW. Beyond symptom self-report: Use of a computer “avatar” to assess post-traumatic stress disorder (PTSD) symptoms. Stress. (2016) 19:593–8. doi: 10.1080/10253890.2016.1232385

PubMed Abstract | CrossRef Full Text | Google Scholar

40. Karstoft KI, Galatzer-Levy IR, Statnikov A, Li Z, Shalev AY, Ankri Y, et al. Bridging a translational gap: Using machine learning to improve the prediction of PTSD. BMC Psychiatry. (2015) 15:30. doi: 10.1186/s12888-015-0399-8

PubMed Abstract | CrossRef Full Text | Google Scholar

41. Finkelman MD, Lowe SR, Kim W, Gruebner O, Smits N, Galea S. Customized computer-based administration of the PCL-5 for the efficient assessment of PTSD: A proof-of-principle study. Psychol Trauma. (2017) 9:379–89. doi: 10.1037/tra0000226

PubMed Abstract | CrossRef Full Text | Google Scholar

42. Karstoft KI, Statnikov A, Andersen SB, Madsen T, Galatzer-Levy IR. Early identification of posttraumatic stress following military deployment: Application of machine learning methods to a prospective study of Danish soldiers. J Affect Disord. (2015) 184:170–5. doi: 10.1016/j.jad.2015.05.057

PubMed Abstract | CrossRef Full Text | Google Scholar

43. Miner A, Kuhn E, Hoffman JE, Owen JE, Ruzek JI, Taylor CB. Feasibility, acceptability, and potential efficacy of the PTSD coach app: A pilot randomized controlled trial with community trauma survivors. Psychol Trauma. (2016) 8:384–92. doi: 10.1037/tra0000092

PubMed Abstract | CrossRef Full Text | Google Scholar

44. Pyne JM, Constans JI, Wiederhold MD, Gibson DP, Kimbrell T, Kramer TL, et al. Heart rate variability: Pre-deployment predictor of post-deployment PTSD symptoms. Biol Psychol. (2016) 121:91–8. doi: 10.1016/j.biopsycho.2016.10.008

PubMed Abstract | CrossRef Full Text | Google Scholar

45. Zhuang X, Rozgić V, Crystal M, Marx BP. Improving speech-based PTSD detection via multi-view learning. In: Proceedings of the 2014 IEEE Spoken Language Technology Workshop (SLT). (Manhattan, NY: IEEE) (2014). p. 260–5. doi: 10.1109/SLT.2014.7078584

CrossRef Full Text | Google Scholar

46. Scherer S, Stratou G, Gratch J, Morency LP. Investigating voice quality as a speaker-independent indicator of depression and PTSD. In: Proceedings of the Annual Conference International Speech Communication Association. Lyon: (2013). p. 847–51. doi: 10.21437/Interspeech.2013-240

CrossRef Full Text | Google Scholar

47. Dabek F, Caban JJ. Leveraging big data to model the likelihood of developing psychological conditions after a concussion. In: A Roy, K Venayagamoorthy, A Alimi, P Angelov, T Trafalis editors. Procedia computer science. (Amsterdam: Elsevier) (2015). p. 265–73. doi: 10.1016/j.procs.2015.07.303

CrossRef Full Text | Google Scholar

48. Alvarez-Conrad J, Zoellner LA, Foa EB. Linguistic predictors of trauma pathology and physical health. Appl Cognit Psychol. (2001) 15:S159–70. doi: 10.1002/acp.839

CrossRef Full Text | Google Scholar

49. Saxe GN, Ma S, Ren J, Aliferis C. Machine learning methods to predict child posttraumatic stress: A proof of concept study. BMC Psychiatry. (2017) 17:223. doi: 10.1186/s12888-017-1384-1

PubMed Abstract | CrossRef Full Text | Google Scholar

50. Coppersmith G, Harman C, Dredze M. Measuring post traumatic stress disorder in twitter. In: Proceedings of the International Conference Weblogs Social Media. (Palo Alto, CA: The AAAI Press) (2014). p. 579–82.

Google Scholar

51. Ćosić K, Popović S, Kukolja D, Horvat M, Dropuljić B. Physiology-driven adaptive virtual reality stimulation for prevention and treatment of stress related disorders. Cyberpsychol Behav Soc Netw. (2010) 13:73–8. doi: 10.1089/cyber.2009.0260

PubMed Abstract | CrossRef Full Text | Google Scholar

52. Marinić I, Supek F, Kovačić Z, Rukavina L, Jendričko T, Kozarić-Kovačić D. Posttraumatic stress disorder: Diagnostic data analysis by data mining methodology. Croat Med J. (2007) 48:185–97.

Google Scholar

53. Kuhn E, Greene C, Hoffman J, Nguyen T, Wald L, Schmidt J, et al. Preliminary evaluation of PTSD coach, a smartphone app for post-traumatic stress symptoms. Mil Med. (2014) 179:12–8. doi: 10.7205/MILMED-D-13-00271

PubMed Abstract | CrossRef Full Text | Google Scholar

54. Galatzer-Levy IR, Karstoft KI, Statnikov A, Shalev AY. Quantitative forecasting of PTSD from early trauma responses: A machine learning application. J Psychiatr Res. (2014) 59:68–76. doi: 10.1016/j.jpsychires.2014.08.017

PubMed Abstract | CrossRef Full Text | Google Scholar

55. Scherer S, Lucas GM, Gratch J, Rizzo A, Morency LP. Self-reported symptoms of depression and PTSD are associated with reduced vowel space in screening interviews. IEEE Trans Affect Comput. (2016) 7:59–73. doi: 10.1109/TAFFC.2015.2440264

CrossRef Full Text | Google Scholar

56. Zatzick D, O’Connor SS, Russo J, Wang J, Bush N, Love J, et al. Technology-enhanced stepped collaborative care targeting posttraumatic stress disorder and comorbidity after injury: A randomized controlled trial. J Trauma Stress. (2015) 28:391–400. doi: 10.1002/jts.22041

PubMed Abstract | CrossRef Full Text | Google Scholar

57. Shaikh al arab A, Guédon-Moreau L, Ducrocq F, Molenda S, Duhem S, Salleron J, et al. Temporal analysis of heart rate variability as a predictor of post traumatic stress disorder in road traffic accidents survivors. J Psychiatr Res. (2012) 46:790–6. doi: 10.1016/j.jpsychires.2012.02.006

PubMed Abstract | CrossRef Full Text | Google Scholar

58. Freeman D, Antley A, Ehlers A, Dunn G, Thompson C, Vorontsova N, et al. The use of immersive virtual reality (VR) to predict the occurrence 6 months later of paranoid thinking and posttraumatic stress symptoms assessed by self-report and interviewer methods: A study of individuals who have been physically assaulted. Psychol Assess. (2014) 26:841–7. doi: 10.1037/a0036240

PubMed Abstract | CrossRef Full Text | Google Scholar

59. Hossain MF, George O, Johnson N, Madiraju P, Flower M, Franco Z, et al. editors. Towards clinical decision support for veteran mental health crisis events using tree algorithm. In: V Getov, J-L Gaudiot, N Yamai, S Cimato, M Chang, Y Teranishi, et al. editors. In: Proceedings of the International Computer Software and Applications Conference. (Washington, DC: IEEE Computer Society) (2019). p. 386–90. doi: 10.1109/COMPSAC.2019.10237

CrossRef Full Text | Google Scholar

60. Gong Q, Li L, Tognin S, Wu Q, Pettersson-Yeo W, Lui S, et al. Using structural neuroanatomy to identify trauma survivors with and without post-traumatic stress disorder at the individual level. Psychol Med. (2014) 44:195–203. doi: 10.1017/S0033291713000561

PubMed Abstract | CrossRef Full Text | Google Scholar

61. World Health Organization. ICD-10 : International statistical classification of diseases and related health problems : Tenth revision. Geneva: World Health Organization (2004).

Google Scholar

62. American Psychiatric Association. Diagnostic and statistical manual of mental disorders. 5th ed. Virginia, VA: American Psychiatric Association (2013). doi: 10.1176/appi.books.9780890425596

CrossRef Full Text | Google Scholar

63. Bradley AP. The use of the area under the ROC curve in the evaluation of machine learning algorithms. Pattern Recognit. (1997) 30:1145–59. doi: 10.1016/S0031-3203(96)00142-2

CrossRef Full Text | Google Scholar

64. National Institute for Health and Care Excellence. Post-traumatic stress disorder – [D] evidence reviews for psychological, psychosocial and other non-pharmacological interventions for the treatment of PTSD in adults. London: National Institute for Health and Care Excellence (2020).

Google Scholar

65. Shalev AY, Ankri Y, Israeli-Shalev Y, Peleg T, Adessky R, Freedman S. Prevention of posttraumatic stress disorder by early treatment: Results from the Jerusalem trauma outreach and prevention study. Arch Gen Psychiatry. (2012) 69:166–76. doi: 10.1001/archgenpsychiatry.2011.127

PubMed Abstract | CrossRef Full Text | Google Scholar

66. Bertl M. News analysis for the detection of cyber security issues in digital healthcare. Young Inform Sci. (2019) 4:1–15.

Google Scholar