Classification of Suicide Attempts through a Machine Learning Algorithm Based on Multiple Systemic Psychiatric Scales

Classification and prediction of suicide attempts in high-risk groups is important for preventing suicide. The purpose of this study was to investigate whether the information from multiple clinical scales has classification power for identifying actual suicide attempts. Patients with depression and anxiety disorders (N = 573) were included, and each participant completed 31 self-report psychiatric scales and questionnaires about their history of suicide attempts. We then trained an artificial neural network classifier with 41 variables (31 psychiatric scales and 10 sociodemographic elements) and ranked the contribution of each variable for the classification of suicide attempts. To evaluate the clinical applicability of our model, we measured classification performance with top-ranked predictors. Our model had an overall accuracy of 93.7% in 1-month, 90.8% in 1-year, and 87.4% in lifetime suicide attempts detection. The area under the receiver operating characteristic curve (AUROC) was the highest for 1-month suicide attempts detection (0.93), followed by lifetime (0.89), and 1-year detection (0.87). Among all variables, the Emotion Regulation Questionnaire had the highest contribution, and the positive and negative characteristics of the scales similarly contributed to classification performance. Performance on suicide attempts classification was largely maintained when we only used the top five ranked variables for training (AUROC; 1-month, 0.75, 1-year, 0.85, lifetime suicide attempts detection, 0.87). Our findings indicate that information from self-report clinical scales can be useful for the classification of suicide attempts. Based on the reliable performance of the top five predictors alone, this machine learning approach could help clinicians identify high-risk patients in clinical settings.

Introduction

Predicting and estimating suicidal behavior in psychiatric patients is an important and vexing clinical issue. A number of methods have been developed and tested for their ability to assess and correlate suicide attempts with various clinical factors. History of previous suicide attempts has long been considered to be a crucial factor in predicting future suicide attempts (1), but attempts history lacks predictive value as only one-third of suicide attempters have a previous history of suicide attempts (2). Clinical scales that directly assess suicidal ideation and behavior are widely used (3, 4), but they generally have a low specificity for suicide attempts as they tend to incorrectly classify low-risk suicide attempters as high-risk attempters (5).

To evaluate and understand the risk factors for suicide, the association between sociodemographic characteristics and actual suicide attempts has also investigated. In a national cohort study of Swedish adults, the presence of psychiatric disorders and chronic diseases were independent risk factors for suicide among both men and women (6). Although depressive symptoms and a family history of suicide were associated with in-patient suicide, they were inadequate in discriminating actual suicide attempters with non-attempters because of relatively low predictive value for high suicide risk categorizations (<2%) (7).

Leveraging the development and growth of various artificial intelligence techniques, recent studies have shown that machine learning approaches can be useful in estimating suicide attempts. A machine learning algorithm trained with patients’ longitudinal electronic health records reliably predicted suicidal behavior (8). Another machine learning approach showed that sociodemographic information and psychopathological factors accurately predicted actual suicide in US Army soldiers (9). Linguistic-driven models that use the text of clinical notes have also been explored, but thus far have lacked sufficient accuracy (65% or more) (10).

Although machine learning techniques yielded a high performance that could be tested in clinical practice, classifying and predicting suicide attempts remains challenging. Central problems seem to be self-reporting bias of suicide attempts and a low accuracy rate of each variable that has been used to predict suicide attempts (8). In addition to these factors, inconsistency of predictors is a major barrier.

In estimating suicide attempts with clinical scales, it is common to use one or two scales such as hopelessness (11) and suicide intention (12). This approach can show reliable performance in certain circumstances, but can easily suffer from reporting biases of the respondents. Using multiple clinical scales at the same time in estimating suicide attempts may reduce biases and increase classification performance. Furthermore, negative aspects of psychological states (e.g., depression, anxiety, and hopelessness) have been frequently used as predictors of suicide attempts, but positive aspects [e.g., life satisfaction, purpose in life (PIL), and emotion regulation] have rarely been applied as parameters in estimating suicide attempts. As the absence of positive emotions and satisfaction can contribute to suicidality (13), consideration of the degree of patients’ positivity may be needed in estimating suicide attempts.

In this study, we report a model for classifying suicide attempts in patients with mental illnesses that was built from a customized artificial neural network classifier. To derive and test this novel model, we administered 31 self-report psychiatric and psychological scales that are widely used in clinical settings, and tested whether these variables have classification power for actual suicide attempts.

Materials and Methods

Participants and Case Definition

A total of 760 participants who visited the Mood and Anxiety Disorder Unit of Seoul St. Mary’s Hospital, The Catholic University of Korea, between 2011 and 2017 were enrolled in the study. 730 of the 760 participants agreed to participate in the survey. Of the 730 participants, 591 (81%) visited the following outpatient clinic and 18 out of 591 (3%) did not complete the questionnaire. Thus, survey data from 573 participants were used to train neural networks.

The characteristics of participants are presented in Table 1. Most of the participants (45.9%) had a depressive disorder as a primary diagnosis, with an anxiety disorder being the second most common illness. All diagnoses were based on the Diagnostic and Statistical Manual of Mental Disorders-IV (14).

TABLE 1

Table 1. Sociodemographic characteristics of participants, with average scores of psychiatric scales.

Participants could choose to complete either a booklet that contained all relevant questionnaires or an online website survey that was similar to the booklet. Participants were asked to complete the survey by the next outpatient visit, and the average return period for both the booklet and online surveys was about 1 week. If all survey contents were completed, the data of the participant were included in further analyses. To estimate the reliability and internal consistency of the questionnaires, we measured the Cronbach’s α for the 31-item scale, obtaining a value of 0.78 (SD = 0.20). All participants were informed of the emergency phone number when they were distressed during the response to the survey.

A suicide attempt was defined according to the following three questions: (1) “Have you ever attempted suicide in your lifetime?” (2) “Within the past year, have you attempted suicide?” (3) “Within the past month, have you attempted suicide?” If the participant answered “yes” to any of the above questions, they were asked to write down the number and the method of their attempts. Among the 573 participants, 163 (28.4%) had a history of suicide attempts in their lifetime, of which 68 (11.9%) were in the past year and 39 (6.8%) in the previous month (Table 2). All subjects who participated in this study provided written informed consent, and the study was approved by the Institutional Review Board of the Ethics Committee of Seoul St. Mary’s Hospital at The Catholic University of Korea (KC09FZZZ0211).

TABLE 2

Table 2. Confusion matrix and classification scores in each predictive model.

Measurement of Psychiatric Rating Scales

To measure the subjective symptoms and individual characteristics of the participants, we adopted 31 self-report rating scales that are widely used in psychiatry and clinical psychology (a list of all scales is presented in Figure 2). Scales included assessments that are known to be suitable for measuring depression and anxiety symptoms, such as the Beck Depression Inventory (BDI) and State-Trait Anxiety Inventory (15, 16). Scales for measuring cognitive emotion regulation strategies were also selected (e.g., Connor–Davidson Resilience Scale and Cognitive Emotion Regulation Questionnaire), as these traits have been significantly correlated with resilience in patients with depression and anxiety disorders (17, 18). As it has been suggested that the lack of positive expectancies are also related to degree of depression, we included scales that measure one’s positivity and attitude toward their life [e.g., Life Orientation Test-Revised (LOT-R) and PIL (19)].

Each participant completed the presented form, which consisted of 31 rating scales and 3 questionnaires about history of suicide attempts. The total scores of each scale were used as predictors, and suicide attempts (yes = 1, no = 0) was set as the primary outcome. Then, we divided the 31 scales into two categories based on characteristics of positivity and negativity. If the scale mainly measures positive aspects of emotion or a psychological state (e.g., optimism in LOT-R), it was classified as a positive scale. The Anxiety Sensitivity Index (ASI) and Childhood Trauma Questionnaire belonged to the negative scale category, as they measure the degree of anxiety and traumatic experiences of childhood, respectively. 15 scales were assigned to the positive category (green text in Figure 2), and 16 to the negative category (red text in Figure 2). Figure 2 details the classification of positive and negative scale; green text represents the positive category, and red text denotes the negative category.

To compute the relative contribution of each variable, we removed the variable and computed the classification performance in its absence (40 variables). Then, we ranked the variables from the highest cross entropy to the lowest. High cross entropy indicates that the classification performance was highly disrupted because of the removal of the variable, meaning that the variable highly contributed to the classification performance.

Model Development and Validation

Forty-one variables (31 psychiatric scales and 10 sociodemographic variables) were used as the inputs for an artificial neural network that consisted of 1 hidden layer with 41 neurons (Figure S1 in Supplementary Material). Two parameters (presence or absence of a suicide attempts) were set as the output. Data were randomly divided into three sets (70% for training, 15% for validation, and 15% for test), and the scaled conjugate gradient method was used for training (20, 21). Training automatically stopped when validation reached the minimum cross entropy, and the performance of each variable was measured by the value of cross entropy (Figure S2 in Supplementary Material).

Results

Classification of Suicide Attempts with All Variables

The artificial neural network pattern classifier trained with 31 psychiatric rating scales and 10 sociodemographic elements showed a reliable performance in classifying suicide attempts (Figure 1). The overall accuracy rate was the highest for 1-month suicide attempts detection (93.7%), followed by 1-year and lifetime detection (90.8 and 87.4%, respectively). The area under the receiver operating characteristic curve (AUROC) was also highest for 1-month suicide attempts detection (0.93), followed by lifetime (0.89) and 1-year detection (0.87). The confusion matrix and classification scores of each predictive model are presented in Table 2. Our model had high specificity for the detection of suicide attempts (99.6% in 1-month, 98.4% in 1-year, and 91.2% in lifetime suicide attempts), but the sensitivity was relatively low except in the case of lifetime suicide attempts detection (77.9%) (Table 2).

FIGURE 1

Figure 1. Receiver operating characteristics curves and area under the curve (AUC) for classifying suicide attempts with 41 predictors.

Contribution of Each Variable in Classifying Suicide Attempts

Performance analysis revealed the contribution of each variable in the classification of suicide attempts (Figure 2). Among 41 variables, the Emotion Regulation Questionnaire (ERQ) had the highest contribution, followed by the Anger Rumination Scale (ARS) and the Satisfaction with Life Scale (SWLS). Although the Scale for Suicide Ideation (SSI) directly measures the wish to die and frequency of suicide ideation (4), it ranked eighth out of all scales for predicting actual suicide attempts. Among the sociodemographic information, the status of employment occupied the largest contribution in the classification of suicide attempts, followed by marriage status and religious beliefs.

FIGURE 2

Figure 2. Contribution ranking for classifying suicide attempts with all predictors. The text colored in green represents scales that mainly measure positivity, whereas the text colored in red represent scales primarily measuring negativity. The gray text denotes the sociodemographic data.

Effects of Scale Characteristics on Classification Performance

To investigate whether the characteristics of psychiatric rating scales were related to the classification of suicidality, we independently trained the artificial neural network with positive or negative scale categories and 10 sociodemographic elements.

The classification performances of each model are presented in Figure 3A. The overall accuracy rate and AUROC were not largely different between the two models. AUROC for the lifetime suicide attempts detection was slightly higher in the model trained with positive scales than in the model trained with negative scales (positive = 0.87, negative = 0.86), but the accuracy of the two groups was similar (positive = 79.9%, negative = 79.9%). AUROC for 1-year and 1-month detection was lower in the model with positive scales than negative scales (positive vs. negative; 1-month, 0.82 vs. 0.86, 1-year, 0.81 vs. 0.86). However, the overall accuracy of the two categories was not largely different, either in the 1-month or 1-year suicide attempts detection (model with positive scales vs. model with negative scales; 1-month, 92.8 vs. 92.8%, 1-year, 87.6 vs. 87.8%).

FIGURE 3

Figure 3. Receiver operating characteristics curves and area under the curve (AUC) for classifying suicide attempts with different categories of clinical scales (A) and with the top five ranked variables (B).

Classification of Suicide Attempts with the Top Five Ranked Variables

To evaluate the applicability of our findings in the clinical setting, we examined whether the model could classify suicide attempts with a small number of predictors. The artificial neural network was newly trained with the top five ranked variables (ERQ, ARS, SWLS, SAI, and ASI), and none of the sociodemographic information was used. Our results showed that the classification performance was relatively preserved for detection of lifetime (AUROC and accuracy rate; 0.87 and 79.6%), 1-year (0.85 and 89.2%), and 1-month detection (0.75 and 92.7%) of suicide attempts. Adding sociodemographic information to the model trained with the top five ranked variables improved the classification performance of 1-month detection (AUROC from 0.75 to 0.81; Figure S3 in Supplementary Material). However, there was no AUROC change in 1-year and lifetime detection (1-year, 0.85; lifetime, 0.87).

Discussion

Using a customized artificial neural network classifier, we showed that synthetic information from multiple self-report clinical scales and sociodemographic data well classified actual suicide attempts. A neural network with 41 parameters could identify retrospective lifetime suicide attempts with an AUROC of 0.89. With only five parameters, the model estimated lifetime suicide attempts with an AUROC of 0.87. Thus, we showed that this trained neural network with multiple self-report clinical scales was effective in correlating clinical scales and patients’ history of suicide attempts. Recent machine learning algorithms predicting suicidal behavior with electronic health records (8) or with sociodemographic information (9) showed a similar performance (AUROC; 0.83–0.85). However, direct comparison of our study to these previous reports should be done with caution, as we estimated retrospective suicide attempts rather than future ones.

The proposed model in this study had 41 predictors, but the characteristics of these variables were different from previous models for suicide attempts detection. Although clinical features of affective disorders (e.g., loss of interest and hopelessness) are well-known predictors of suicide (11, 22), these parameters were not used in our analysis. Instead, we included factors relevant to suicide propensity by using the total score on a battery of psychiatric and cognitive clinical scales (e.g., Beck Hopelessness Scale and BDI). Information on the participants’ diagnoses or substance use were also not used in training, though the lifetime risk for suicide has been shown to significantly vary between psychiatric illnesses (12, 23), and alcohol abuse was shown to be an independent predictor of suicide (9). Participant-reported subjective symptoms and psychological states were the main variables in our analysis, and the total number of variables was much less than in previous studies; previously, one study used 73–122 predictors (8), and another used over 100 (9). With about half the number of predictors, we showed that classification performance of suicide attempts could achieve a similar or higher AUROC value (0.89–0.93 with 41 predictors), though our study retrospectively detected suicide attempts, not future attempts.

Our results indicating high accuracy of our model in suicide attempts classification might owe to the direct measurement of suicide ideation, because we used the SSI (4) as a predictor. The SSI consists of 19 items that directly measures the “wish to live,” “wish to die,” and “frequency of suicide ideation.” Although a previous study showed that SSI was a significant measurement in detecting eventual suicide (24), it was only the eighth highest contributor in our model (Figure 2). Rather than the direct assessment of suicide ideation, indirect measures such as the ERQ, ARS, and SWLS were better predictors of actual suicide attempts. Furthermore, even without information from SSI, our model successfully classified suicide and non-suicide attempters (Figure 3B). Thus, the high accuracy rate and high AUROC value might be related to the subjective information of multiple scales, rather than direct information on suicide ideation.

While a number of studies have found that negative psychological states, such as hopelessness (11), anxiety (25), and depression (26) are related to suicidality, few studies have identified the role of positivity on suicide attempts. Our results showed that there were minimal differences between positive clinical scales and negative ones in detecting actual suicidal attempts. In addition, the best predictor of suicide attempts was the ERQ (27), which does not seem related to suicidality. This 10-item scale was designed to measure participants’ tendencies to regulate their emotions with two positive strategies—cognitive reappraisal and expressive suppression. Individuals with dysfunction in the neural circuitry of emotion regulation are at risk for violence and aggression (28), and deficits in emotion regulation are known to be associated with self-harm behavior (29). As the main outcome of this study was “actual suicide attempts” rather than “eventual suicide,” the status of emotion regulation might have made an important contribution to our findings. These results suggest that both lack of positive aspects of psychological states and excess negative emotions might similarly contribute to the detection of suicide attempts.

Our model with 41 predictors had a reliable performance in classifying 1-month, 1-year, and lifetime suicide attempts. However, it is difficult to measure over 30 self-report scales in the outpatient setting. Thus, it is notable that our model with only five predictors made a relatively accurate classification of suicide attempts with the exception of 1-month suicide attempts detection (Figure 3B). This result is somewhat interesting, as we did not include any sociodemographic information and only used the total score from five self-report psychiatric scales. As the performance of the neural network generally increases with number of predictors and sample size (30), we would expect weaker performance with 5 variables than with all 41 variables.

Several limitations should be considered in interpreting our results. First, we note that the sensitivity of 1-month and 1-year suicide attempts detection was fairly low in our model (12.8% in 1-month detection, 33.8% in 1-year detection). These results might be due to the small number of actual suicide attempts (N; 1-month = 39, 1-year = 68), as the classification power can be limited by training sample size (31). The lifetime suicide attempts category (163 samples) of our model showed improved performance (sensitivity = 77.9%) vs. 1-month and 1-year suicide attempts. Second, there was heterogeneity in the diagnosis of psychiatric illnesses. As patients with depression and anxiety disorders were similarly distributed in our sample, it is difficult to apply our results to specific illness groups. Third, as the inner workings of machine learning algorithms act like a “black box,” it is more difficult to interpret the meaning of this model compared to classical approaches (e.g., Bayesian modeling) (8). Fourth, administering a large number of questionnaires to a psychologically sensitive population raises ethical concerns. Although we provided a choice of response type (offline or online survey) and did not severely restrict the reply period of questionnaires, responding to all of 31 self-report questionnaires can be quite burdensome, especially to in-patients.

In conclusion, our findings suggest that systemic information reported on multiple self-report psychiatric scales can accurately classify suicide attempts. Scales that measure positive psychological state yielded comparable and even greater classification performance than scales with negative psychological states. Our model for suicide attempts detection with only five self-report scales has practical implications for the screening and estimation of suicide risk in clinical settings. This machine learning approach can be used in precision medicine in that it can assess and estimate one’s suicide attempts based on individual psychiatric and psychological scales.

Ethics Statement

This study was carried out in accordance with the recommendations of Institutional Review Board of the Ethics Committee of Seoul St. Mary’s Hospital at The Catholic University of Korea with written informed consent from all subjects. All subjects gave written informed consent in accordance with the Declaration of Helsinki. The protocol was approved by the Institutional Review Board of the Ethics Committee of Seoul St. Mary’s Hospital at The Catholic University of Korea.

Author Contributions

JO and J-HC conceived the idea and designed the study. J-HH and J-HC acquired the data. JO, KY, and J-HC analyzed, interpreted the data, and drafted the manuscript. All authors commented and supervised on the manuscript and approved the final version of the manuscript.

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Acknowledgments

This research was supported by a grant of the Korea Health Technology R&D Project through the Korea Health Industry Development Institute (KHIDI), funded by the Ministry of Health & Welfare, Republic of Korea (grant number: HM15C1054).

Supplementary Material

The Supplementary Material for this article can be found online at http://journal.frontiersin.org/article/10.3389/fpsyt.2017.00192/full#supplementary-material.

Figure S1. Neural network structure with one hidden layer.

Figure S2. Cross entropy change of neural network classifier according to epoch.

Figure S3. Receiver operating characteristics curves and area under the curve (AUC) for classifying suicide attempts with top 5 predictors and 10 sociodemographic information.

References

1. Leon AC, Friedman RA, Sweeney JA, Brown RP, Mann JJ. Statistical issues in the identification of risk factors for suicidal behavior: the application of survival analysis. Psychiatry Res (1990) 31:99–108. doi:10.1016/0165-1781(90)90112-I

PubMed Abstract | CrossRef Full Text | Google Scholar

2. Mann JJ, Waternaux C, Haas GL, Malone KM. Toward a clinical model of suicidal behavior in psychiatric patients. Am J Psychiatry (1999) 156:181–9. doi:10.1176/ajp.156.2.181

PubMed Abstract | CrossRef Full Text | Google Scholar

3. Posner K, Brown GK, Stanley B, Brent DA, Yershova KV, Oquendo MA, et al. The Columbia-Suicide Severity Rating Scale: initial validity and internal consistency findings from three multisite studies with adolescents and adults. Am J Psychiatry (2011) 168:1266–77. doi:10.1176/appi.ajp.2011.10111704

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Beck AT, Kovacs M, Weissman A. Assessment of suicidal intention: the Scale for Suicide Ideation. J Consult Clin Psychol (1979) 47:343–52. doi:10.1037/0022-006X.47.2.343

CrossRef Full Text | Google Scholar

5. McMillan D, Gilbody S, Beresford E, Neilly L. Can we predict suicide and non-fatal self-harm with the Beck Hopelessness Scale? A meta-analysis. Psychol Med (2007) 37:769–78. doi:10.1017/S0033291706009664

PubMed Abstract | CrossRef Full Text | Google Scholar

6. Crump C, Sundquist K, Sundquist J, Winkleby MA. Sociodemographic, psychiatric and somatic risk factors for suicide: a Swedish national cohort study. Psychol Med (2014) 44:279–89. doi:10.1017/S0033291713000810

PubMed Abstract | CrossRef Full Text | Google Scholar

7. Large M, Smith G, Sharma S, Nielssen O, Singh SP. Systematic review and meta-analysis of the clinical factors associated with the suicide of psychiatric in-patients. Acta Psychiatr Scand (2011) 124:18–29. doi:10.1111/j.1600-0447.2010.01672.x

PubMed Abstract | CrossRef Full Text | Google Scholar

8. Barak-Corren Y, Castro VM, Javitt S, Hoffnagle AG, Dai Y, Perlis RH, et al. Predicting suicidal behavior from longitudinal electronic health records. Am J Psychiatry (2017) 174:154–62. doi:10.1176/appi.ajp.2016.16010077

PubMed Abstract | CrossRef Full Text | Google Scholar

9. Kessler RC, Warner CH, Ivany C, Petukhova MV, Rose S, Bromet EJ, et al. Predicting suicides after psychiatric hospitalization in US Army soldiers: the army study to assess risk and resilience in servicemembers (Army STARRS). JAMA Psychiatry (2015) 72:49–57. doi:10.1001/jamapsychiatry.2014.1754

PubMed Abstract | CrossRef Full Text | Google Scholar

10. Poulin C, Shiner B, Thompson P, Vepstas L, Young-Xu Y, Goertzel B, et al. Predicting the risk of suicide by analyzing the text of clinical notes. PLoS One (2014) 9:e85733. doi:10.1371/journal.pone.0085733

PubMed Abstract | CrossRef Full Text | Google Scholar

11. Beck AT, Steer RA, Kovacs M, Garrison B. Hopelessness and eventual suicide: a 10-year prospective study of patients hospitalized with suicidal ideation. Am J Psychiatry (1985) 142:559–63. doi:10.1176/ajp.142.5.559

PubMed Abstract | CrossRef Full Text | Google Scholar

12. Stefansson J, Nordström P, Jokinen J. Suicide Intent Scale in the prediction of suicide. J Affect Disord (2012) 136:167–71. doi:10.1016/j.jad.2010.11.016

CrossRef Full Text | Google Scholar

13. Heisel MJ, Flett GL. Purpose in life, satisfaction with life, and suicide ideation in a clinical sample. J Psychopathol Behav Assessement (2004) 26:127–35. doi:10.1023/B:JOBA.0000013660.22413.e0

CrossRef Full Text | Google Scholar

14. American Psychiatric Association AP. DSM-IV-TR, Diagnostic and Statistical Manual of Mental Disorders. 4th ed. Washington, DC: APA (2000).

Google Scholar

15. Beck AT, Ward CH, Mendelson M, Mock J. An inventory for measuring depression. Arch Gen Psychiatry (1961) 4:561–71. doi:10.1001/archpsyc.1961.01710120031004

CrossRef Full Text | Google Scholar

16. Spielberger CD, Gorsuch RL, Lushene R, Vagg PR, Jacobs GA. Manual for the State-Trait Anxiety Inventory. CA: Consulting Psychologists Press (1983).

Google Scholar

17. Min J-A, Yu JJ, Lee C-U, Chae J-H. Cognitive emotion regulation strategies contributing to resilience in patients with depression and/or anxiety disorders. Compr Psychiatry (2013) 54:1190–7. doi:10.1016/j.comppsych.2013.05.008

PubMed Abstract | CrossRef Full Text | Google Scholar

18. Huh HJ, Kim KH, Lee H-K, Chae J-H. The relationship between childhood trauma and the severity of adulthood depression and anxiety symptoms in a clinical sample: The mediating role of cognitive emotion regulation strategies. J Affect Disord (2017) 213:44–50. doi:10.1016/j.jad.2017.02.009

PubMed Abstract | CrossRef Full Text | Google Scholar

19. Beevers CG, Meyer B. Lack of positive experiences and positive expectancies mediate the relationship between BAS responsiveness and depression. Cogn Emot (2002) 16:549–64. doi:10.1080/02699930143000365

CrossRef Full Text | Google Scholar

20. Yun K, Stoica A. Improved target recognition response using collaborative brain-computer interfaces. In Systems, Man, and Cybernetics (SMC), 2016 IEEE International Conference on; Budapest: IEEE (2016). p. 2220–3.

Google Scholar

21. Møller MF. A scaled conjugate gradient algorithm for fast supervised learning. Neural Networks (1993) 6:525–33. doi:10.1016/S0893-6080(05)80056-5

CrossRef Full Text | Google Scholar

22. Inskip HM, Harris EC, Barraclough B. Lifetime risk of suicide for affective disorder, alcoholism and schizophrenia. Br J Psychiatry (1998) 172:35–7. doi:10.1192/bjp.172.1.35

PubMed Abstract | CrossRef Full Text | Google Scholar

23. Dumais A, Lesage AD, Alda M, Rouleau G, Dumont M, Chawky N, et al. Risk factors for suicide completion in major depression: a case-control study of impulsive and aggressive behaviors in men. Am J Psychiatry (2005) 162:2116–24. doi:10.1176/appi.ajp.162.11.2116

PubMed Abstract | CrossRef Full Text | Google Scholar

24. Brown GK, Beck AT, Steer RA, Grisham JR. Risk factors for suicide in psychiatric outpatients: a 20-year prospective study. J Consult Clin Psychol (2000) 68:371–7. doi:10.1037/0022-006X.68.3.371

PubMed Abstract | CrossRef Full Text | Google Scholar

25. Sareen J, Cox BJ, Afifi TO, de Graaf R, Asmundson GJG, Have ten M, et al. Anxiety disorders and risk for suicidal ideation and suicide attempts: a population-based longitudinal study of adults. Arch Gen Psychiatry (2005) 62:1249–57. doi:10.1001/archpsyc.62.11.1249

PubMed Abstract | CrossRef Full Text | Google Scholar

26. Oquendo MA, Galfalvy H, Russo S, Ellis SP, Grunebaum MF, Burke A, et al. Prospective study of clinical predictors of suicidal acts after a major depressive episode in patients with major depressive disorder or bipolar disorder. Am J Psychiatry (2004) 161:1433–41. doi:10.1176/appi.ajp.161.8.1433

PubMed Abstract | CrossRef Full Text | Google Scholar

27. Gross JJ, John OP. Individual differences in two emotion regulation processes: implications for affect, relationships, and well-being. J Pers Soc Psychol (2003) 85:348–62. doi:10.1037/0022-3514.85.2.348

CrossRef Full Text | Google Scholar

28. Davidson RJ, Putnam KM, Larson CL. Dysfunction in the neural circuitry of emotion regulation – a possible prelude to violence. Science (2000) 289:591–4. doi:10.1126/science.289.5479.591

CrossRef Full Text | Google Scholar

29. Chapman AL, Dixon-Gordon KL. Emotional antecedents and consequences of deliberate self-harm and suicide attempts. Suicide Life Threat Behav (2007) 37:543–52. doi:10.1521/suli.2007.37.5.543

PubMed Abstract | CrossRef Full Text | Google Scholar

30. Karayiannis N, Venetsanopoulos AN. Artificial Neural Networks: Learning Algorithms, Performance Evaluation, and Applications. Boston, MA: Kluwer Academic Publishers (1993).

Google Scholar

31. Markham IS, Rakes TR. The effect of sample size and variability of data on the comparative performance of artificial neural networks and regression. Comput Oper Res (1998) 25:251–63. doi:10.1016/S0305-0548(97)00074-9

CrossRef Full Text | Google Scholar