Development and validation study of the suicide screening questionnaire-observer rating (SSQ-OR)

Background: Observer rating scales are necessary to evaluate the risk of suicide because individuals at risk for suicide are often unwilling to seek help on their own. Reliability and validity were evaluated for the newly developed Suicide Screening Questionnaire-Observer Rating (SSQ-OR).

Methods: Preliminary items were assessed by 251 experts online and 25 questions were selected. 328 individuals at high-risk and 661 controls from 12 Crisis Response Centers and 5 university counseling centers were recruited to complete SSQ-OR, Beck Scale for Suicide Ideation (BSSI) and Patient Health Questionnaire-9 (PHQ-9). In a 6 months follow-up, we reached out to 176 participants to ask whether they had experienced a suicidal thought, plan, or attempt since the baseline assessment. Cronbach's α, Mann-Whitney U test, Spearman's correlation, factor analyses, Receiver operating characteristic (ROC) analysis and logistic regression analysis were used to verify the SSQ-OR.

Results: Structural validity was supported by a two-factor solution using exploratory and confirmatory factor analyses. Excellent model fit indices for the two-factor structure using exploratory factor analysis were confirmed (RMSEA = 0.033, TLI = 0.980, CFI = 0.983). The SSQ-OR demonstrated strong internal consistency. The concurrent validity based on the correlations with other self-reported indicators of suicidal potential–BSSI and PHQ-9– revealed substantial relationships. The high-risk group was effectively characterized by a cut-off point of 4, with a sensitivity of 0.73 and a specificity of 0.79. The SSQ-OR scores were significant predictors of suicidal thoughts and behaviors within 6 months.

Conclusions: The SSQ-OR exhibits sound psychometric properties, and could be used as a complement to a self-report or clinical-administered scale to screen suicide risk comprehensively.

Introduction

Suicide is a serious mental health concern, with approximately 800,000 suicide deaths reported annually worldwide (1). Similarly, suicide rates have substantially increased over the past years in South Korea. Suicide is the fifth leading cause of death, and a high mortality rate of 25.7 per 100,000 population in 2020 is reported (2). Despite extensive research over the past 50 years, predicting suicide risk is still limited and are not effectively implemented (3, 4). This underscores the need for developing an assessment tool that addresses limitations of current suicide risk assessment.

Most of the suicide risk assessment tools only focus on certain risk factors such as suicidal thoughts, suicidal behavior, or other risk factors (5, 6). However, previous literature have that suicidal thoughts and behaviors (STBs) are caused by a complex interaction of various proximal and distal factors (3, 7–9). It is necessary to include a broader context of risk factors based on relatively distal risk factors as well as STBs (10). Based on the results of previous suicide studies (3, 7, 9), when designing a tool to screen the risk of STBs, it is necessary to consider risk factors not sufficiently covered by current assessment tools such as the social factors, negative life events, and physical disorders.

In a recent systematic review of risk scales, the use of a single instrument are not recommended in clinical settings (11). Rather than relying on a single tool, there is a need to combine multiple risk assessment tools together to effectively measure suicide risk. Moreover, non-disclosure or denial of suicidal ideation is quite common among psychiatric patients and those who attempt and/or complete suicide (12–14). Since concealment is a risk factor for STBs (15, 16), it is likely that the evaluation using only self-reports or clinician interviews might have limitations. The observer rating complements the self-report or clinician-administered scale since it is based on direct observation of subjects' behavior in a daily environment. The observer rating method accounted for significant variance beyond self-report and revealed unique information relative to self or assessor evaluations in suicide behaviors (17, 18). In particular, suicide warning signs, revealed through psychological autopsy, allow observers to detect the imminent risk of STBs (19). Hence, it can be helpful to use an observer rating scale that complements the existing instruments when screening individuals with suicide risk.

To this end, the present study was undertaken to develop a more comprehensive, observer-rated screening tool for identifying suicide risk. The study aims to develop and validate a Suicide Screening Questionnaire-Observer Rating (SSQ-OR). We described the process of (1) investigating structural validity, (2) examining of the measure's concurrent validity, predictive validity, and internal consistency, and (3) setting the best cut-off score that discriminate between individuals with a high risk of suicide and those with at low risk.

Methods

Scale development

For scale development, preliminary questions were constructed through extensive research on existing suicidal risk scales, previous studies on suicide risk factors [e.g., Franklin et al., Rudd et al., O'Connor and Nock (3, 7, 8)], qualitative interviews with individuals who have a history of STBs and counselors, and the nationwide reports on suicide deaths [e.g., (20, 21)]. According to psychological autopsy results, warning signs were observed in more than 30% of suicide decedents, and these warning signs were reflected in the preliminary items (22). In order to explore the characteristics of high-risk groups for suicide that can be observed by the subject's families or acquaintances, counseling psychologists interviewed various groups at high risk of suicide and performed a qualitative study. Based on these findings, we generated a total of 64 preliminary items composed of 5 categories: suicidal behavior, mental health problems, temperament & personality, environmental factors, and warning signs.

Next, we evaluated the relevance and clarity of each question as well as the content validity. We conducted an online survey of 251 mental health professionals in the field of psychiatry, clinical psychology, counseling psychology, social welfare, education, or nursing, to evaluate the relevancy and clarity of each item. Based on the results of the first survey, a group of 17 professionals (i.e., psychiatrists, clinical psychologists, counseling psychologists) selected 39 questions and refined the wordings of each question. Then each professional rated the degree of adequacy and significance of the items on a five-point scale on (1) adequacy and (2) significance of the item (ranging from 1= ‘very inappropriate/not important' to 5= ‘very appropriate/important'). Finally, 25 items were selected after excluding those with low scores in adequacy or importance. Supplementary Appendix A presents the relevance and importance scores of the final 25 items rated by the 17 experts.

Procedure

We recruited participants from 12 Crisis Response Centers and 5 university counseling centers across South Korea between October 2020 to March 2021. These centers recruited community-dwelling adults as well as visitors to the center. The Crisis Response Center operated by university hospitals provide case management services for those who visit the emergency room due to a suicide crisis. Inclusion criteria were being at least 18 years old and having the reading level of a middle school graduate. Participants were excluded if they had neurodegenerative disease, intellectual disability, clinically significant personality disorder, brain injury, and any other conditions that might interfere with participation. After a full explanation of this study, those who voluntarily signed the consent form were included. The compensation was provided for both participants and clinicians. All participants were interviewed in a one-on-one setting by trained clinicians of the centers and asked to complete a set of self-report questionnaires. The SSQ-OR requested a rating from the subject's families or acquaintances including counselor/social worker, spouse, friend/colleague, lover, and teacher/professor. Clinicians interviewed the participants from Crisis Response Centers 6 months later to establish whether there had been suicide ideation, plan, or attempt since the baseline assessment. This study was approved by the Institutional Review Board of the Samsung Medical Center (IRB No. SMC 2020-04-184).

Participants

A total of 989 subjects participated in this study. We used the term ‘suicidal group' for those who had suicidal ideation and/or suicide attempt together as a high-risk group. We defined the suicidal group as those who thought about killing themselves but had not attempted suicide and/or those who had attempted suicide within 6 months and the control group as those who did not have suicidal ideation, suicidal plan, or suicidal attempt within 6 months. Among the total participants, 328 (33.16%) were recruited for the suicidal group and 661 (66.84%) for the control group. The information of observers who rated SSQ-OR is as follows: counselor/social worker 435 (43.98%), friend/colleague 160 (16.18%), family 142 (14.36%), teacher/professor 95 (9.61%), spouse 63 (6.37%), lover 51 (5.16%), and others 43 (4.35%). Out of all subjects, 176 (17.80%) were followed up through phone calls after 6 months to assess whether they have ever experienced STBs (i.e., suicidal ideation, suicide plan, and suicidal attempt). Among those, 157 participated in this follow-up interview and 19 did not answer.

Measures

Suicide screening questionnaire-observer rating (SSQ-OR)

Based on the aforementioned procedures, the final version of the SSQ-OR consisted of 25 items. The SSQ-OR asks the informants, such as families or acquaintances, to rate the presence of observable behaviors of an individual over the last month on a binary scale (yes/no). The total score of SSQ-OR ranges from 0 to 25. The ‘do not know' response to each item is not included in the total score, and the range of the response rate of ‘do not know' (see Supplementary Appendix B) was 0.93 to 6.41% in the present study. Except for item 8 (“Has attempted suicide more than once so far”) and item 13 (“Says he/she wanted to die following a deceased family member, friend, pet or celebrity”), the response rates of ‘do not know' were <5%.

Beck scale for suicide ideation (BSSI)

To verify the concurrent validity of the SSQ-OR, we used the BSSI, the most frequently used measure of suicidal risk. BSSI developed by Beck, Steer (23) consists of 21 questions that measure suicidality and the severity thereof. Based on the participants' experience of the past weeks, a three-point Likert scale (0–2 points) was used. The psychometric properties of the BSSI have been established for Koreans (24, 25). The value of Cronbach's alpha for this study sample was 0.92.

Patient health questionnaire (PHQ-9)

The PHQ-9 is a nine items self-report measure used to assess the severity of depression. The items are based on the DSM-IV's diagnostic criteria for major depressive disorder. On each of the nine items, participants are asked to self-rate how often they have experienced the indicated symptoms of depression over the previous 2 weeks on a four-point Likert scale (0–3 points). The Korean version of PHQ-9 standardized by Han, Jo (26) and Kim and Lee (27) demonstrated adequate internal consistency and convergent validity. The Cronbach's alpha coefficient of the study was 0.91.

Statistical analyses

The Statistical Package for Social Sciences 21 was used to examine descriptive characteristics, concurrent validity, internal consistency, corrected item-total correlation, Receiver Operating Characteristic curve (ROC) analysis, and predictive validity of the sample. Intergroup comparisons were conducted using the Mann–Whitney U-test. Correlation analyses were performed using Spearman's rank correlation analysis. The prevalence of the lowest scores, 0/25 (floor effect), and highest scores, 25/25 (ceiling effect), for the SSQ-OR were calculated. If more than 15% of the participants scored maximum or minimum scores, we considered these to be floor and ceiling effects, respectively. We adopted a series of analyses to evaluate its structural validity. Bartlett's test of sphericity and the Kaiser-Meyer-Olkin index (KMO) were performed to determine whether the data is suitable for factor analysis. In order to identify the number of factors to retain, Kaiser's criterion, Cattell's scree test, and parallel analysis based on Minimum Rank Factor Analysis (PA-MRFA) were applied using Factor 10.10.02. We randomly split the total sample into two separated samples using RANDBETWEEN function in Microsoft Office Excel 2016. Based on the two split samples, an exploratory factor analysis (EFA) and a confirmatory factor analysis (CFA) were conducted with geomin rotation using Mplus 7.0. Due to the ordinal nature of the SSQ-OR item responses, the SSQ-OR models were examined treating the items as categorical with the mean-and variance-adjusted weighted least squares estimator (WLSMV) (28, 29) which has been found to be robust to violations of normality Missing values were treated as pairwise missing by convention with this estimator. Assessment of the fit of each model was based on several indices. Since the χ2 statistic is highly sensitive to sample size (30), three fit indices were considered together: (1) the comparative fit index (CFI), (2) the Tucker Lewis index (TLI), and (3) the root mean square error of approximation (RMSEA). According to the criteria proposed in previous studies, RMSEA <0.05 and CFI and TLI > 0.95 was considered to be good (31). Also, binary logistic regression analyses were conducted to investigate whether the SSQ-SR total and sub-factor scores predicted STBs within 6 months. ROC analysis was done to evaluate the screening ability of SSQ-SQ to discriminate between suicidal and control groups. We investigated the best cut-off point with the maximal area under the curve (AUC) and sensitivity/specificity. AUC values of >0.9, 0.8–0.9, 0.7–0.8, 0.6–0.7, and 0.5–0.6 are regarded as excellent, very good, good, sufficient, and bad diagnostic accuracy, respectively (32).

Results

Demographic characteristics

Table 1 shows descriptive results of basic characteristics. All of the participants were between 18 and 68 years of age (Mean age = 26.94, SD = 9.34). The male-to-female ratio was 1:1.76. There was no significant difference in the total score of SSQ-OR across gender, U = 85,585.50, ns.

TABLE 1

Table 1. Demographic characteristics of the study sample and scores of suicide screening questionnaire-observer rating (SSQ-OR).

Among 989 adults, 328 (33.16%) were in the suicidal group, which showed suicidal ideation or/and suicidal attempt over the last 6 months. Age of this group ranged from 18–68 years old (Mean age = 26.92, SD = 9.79), and 71.34% of them were female. Out of all samples, 661 (66.83%) were assigned to the control group, which showed no suicidal ideation or/and suicidal attempt in the last 6 months. The participants in the control group were between 18 to 67 years of age (Mean age = 26.94, SD = 9.12), and 60.06% of them were female.

Structural validity

Exploratory factor analysis (EFA)

In determining whether the data is suitable for factor analysis, Bartlett's test of sphericity and the Kaiser-Meyer-Olkin index (KMO) were computed. The Bartlett's test was significant [χ2(300) = 3751.0, p < 0.001] and the KMO index was 0.91, which was judged to be appropriate data for factor analysis.

To identify the number of factors to retain, the eigenvalues of the polycorrelation matrix were reviewed (see Supplementary Appendix C). Initially, the three-factor solution with eigenvalues more than one was found using the Kaiser criteria. However, the Kaiser criterion might overestimate the number of factors (33). Further analysis of the Scree test supported the single-factor solution in that the eigenvalues drop significantly from the first to the next. Finally, the results of the parallel analysis (see Supplementary Appendix D) demonstrated that the 95th percentile eigenvalue for the random data distribution was larger than the eigenvalue of the raw score distribution for a single-factor solution. Referring to these results, the range of the number of factors of SSQ-OR that can be explored was determined to be 1–2.

We explored the underlying structure of SSQ-OR using EFA. A good model fit was obtained for a single-factor solution [χ2(275) = 483.780, p < 0.001; RMSEA = 0.042, TLI = 0.968, CFI = 0.970] and a two-factor solution [χ2(251) = 368.152, p < 0.001; RMSEA = 0.033, TLI = 0.980, CFI = 0.983]. Supplementary Appendix E presents the standardized factor loadings for the two-factor model. Based on Thurstone's criteria (34), the two-factor solution had: (1) zero items with salient loadings (≥0.40) on more than one factor, (2) zero items with no salient loading on any factor, and (3) well-defined salient loading per factor (i.e., factor 1 had 15 items, factor 2 had 10 items). With regards to the interpretability of the resulting factor structures and Thurstone's criteria (34), the two-factor model is found to be the most suitable for SSQ-OR. These factors were: (1) Suicide and mental health, and (2) Social and environmental stress. It was found that the factor loading of item 11 exceeded the absolute value of 1. However, the standardized regression coefficient might exceed the absolute value of one depending on the correlation between the independent variables and the correlation between each independent variable and the dependent variable (35).

Confirmatory factor analysis (CFA)

To confirm structural validity, CFA was performed on the other split sample. Based on the EFA results, our hypothetical factor model contained two factors: (1) Suicide and mental health, and (2) Social and environmental stress. A hypothesized two-factor solution showed a good model fit [χ2(274) = 438.003, p < 0.001; RMSEA = 0.037, TLI = 0.972, CFI = 0.974]. The standardized factor loadings of the CFA model are illustrated in Supplementary Appendix E. All factor loadings of the two-factor model demonstrated salient loadings (≥0.40). Therefore, the result of CFA cross-validated the two-factor structure suggested in the finding of EFA.

Concurrent validity

In order to examine the relationship between SSQ-OR and other suicide-related measures, we used self-reported scales (BSSI, PHQ-9). Spearman's correlations were assessed (Table 2). The SSQ-OR total and sub-factors were moderately correlated with BSSI and PHQ-9. Additionally, the correlation between the sub-factors of SSQ-OR was 0.778.

TABLE 2

Table 2. Means, standard deviations, and scale inter correlations.

Internal consistency

Coefficient alphas and corrected item-total correlations were computed for SSQ-OR. The Cronbach's alpha of the SSQ-OR total was 0.91. The Cronbach's alpha was 0.89 and 0.75 for Factor 1 and 2, respectively. As shown in Table 1, total and sub-factor scores revealed significant differences between suicidal and control groups. The Cronbach's alpha if-item-deleted coefficients ranged from 0.90 to 0.91. The corrected item-total correlations showed that all SSQ-OR items were significantly correlated with the SSQ-OR total scale score (Supplementary Appendix B). These values support the internal consistency of the SSQ-OR scores.

As for the floor and ceiling effects, the floor effect identified the percentage of individual results that corresponded to the theoretical minimum (0/25). 266 of 989 subjects (26.9%) scored zero so there was a floor effect. The absence of ceiling effect was observed, that is, none of the subjects scored the theoretical maximum (25/25).

ROC analysis

Results of the ROC analysis are presented in Figure 1. In comparing the suicidal group with the control group, the AUC area was 0.817 (95% Confidence Interval: 0.786, 0.848) and the cut-off point of four provided the optimal balance between the sensitivity and specificity when considering Youden's J statistic (Table 3). The point decided by Youden's index is the farthest point from the diagonal line, where the sum of sensitivity and specificity can be maximal (20). At the cut-off point of four, the values of sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) are 73.0, 79.2, 66.1, and 84.2%, respectively. If the researcher wanted to maximize sensitivity for screening purpose, it would be possible to minimize false negatives by choosing the cut-off score of 3.

FIGURE 1

Figure 1. Receiver operating characteristic curve (ROC) analysis for the suicide screening questionnaire-observer rating (SSQ-OR).

TABLE 3

Table 3. Sensitivity, specificity for each cut-off score.

Predictive validity

To investigate predictive validity, a total of nine logistic regression models were run with STBs (i.e., suicidal ideation, suicide plan, and suicidal attempt) as the outcome of interest for the subsample of those with 6 months follow-up data. Baseline SSQ-OR total and sub-factor scores were significant predictors of future STBs during the 6 months in adjusted analyses for gender and age (Table 4). The SSQ-OR total score was significantly associated with the outcome with the odds increasing by about 10% in all STBs. The sub-factor scores were also significantly associated with all future STBs: Factor 1 (Suicide and mental health) increased the odds ratio by about 10–20%, whereas Factor 2 (Social and environmental stress) increased the odds ratio by about 30–40%.

TABLE 4

Table 4. Logistic regression models of the relationship between SSQ-OR and Suicide Ideation, suicide plan, or suicide attempt during 6 months follow-up (n = 157).

Discussion

The present study evaluated the psychometric properties of the SSQ-OR (see Table 5 for the complete SSQ-OR questionnaire in English), the newly developed screening scale for suicide risk. It was found that the observer's 'don't know' ratio was generally <5%. That is, the SSQ-OR consisted of items that could be easily observed by acquaintances or family members of the subjects.

TABLE 5

Table 5. Suicide screening questionnaire-observer rating; SSQ-OR.

The structural validity was supported by a two-factor model using a series of EFA and CFA and showed good fit indices overall. Based on the loaded items of each factor, the two sub-factors indicated (1) Suicide and mental health and (2) Social and environmental stress, respectively. While the existing suicide risk measurements mainly focused on STBs or psychological factors related to STBs (5, 6, 36), the SSQ-OR also evaluated social and environmental stress corresponding to major risk factors (3). Suicide risk is a synergistic relationship between a number of intra- and inter-individual factors (37). Our findings were also consistent with several studies suggesting that suicide risk can be divided broadly into two categories: personal and social factors (38, 39). The total and sub-factor scores of the suicidal group were higher than those of the non-suicidal group, which demonstrated the content validity.

To investigate concurrent validity, we included the self-rated BSSI and PHQ-9 as measures of suicidal ideation and depression, respectively. It was found that moderate correlations between SSQ-OR total and suicidal ideation and depression scores. This is quite remarkable because the subject's acquaintances or families were not trained in the assessment and were simply instructed to score the SSQ-OR based on their observations in daily life. The present findings suggest a good convergence between self-reports and observer ratings. This was consistent with previous literature on the relationship between suicide risk and STBs. First, suicidal ideation is considered to be the starting point of STBs and is known to be a significant predictor of suicide attempts (40, 41). The severity of suicidal ideation is a major risk factor in predicting suicide leading to actual death. In particular, serious suicide attempters have previously reported suicidal thoughts lasting weeks to months, (42) and it was found that the probability of leading to a suicide attempt within 1 year after the accident occurred increased by 15–20% by countries (9). Second, depression is among the most commonly cited risk factors for STBs. Depression has been reported to be likely associated with the development of suicidal desire, and one of the prominent suicide theories, the Interpersonal Theory of Suicide, suggests that depression interacts with hopelessness and develops into STBs (43). Psychological autopsy studies have also repeatedly reported that depression is the most common psychiatric disorder among suicide deaths (22, 44). Consistent with theories and autopsy studies, a recent meta-analysis of longitudinal studies found that depression significantly confers future suicide ideation, attempt, and death (45). Therefore, concurrent validity was confirmed in that SSQ-OR showed a significant correlation with depression and suicidal ideation.

The SSQ-OR appeared to have strong internal consistency for total items and the sub-factors, and the item-total correlations were found to be favorable. Whereas no ceiling effects were found for the SSQ-OR, the floor effect was detected. This means that it could be hard to differentiate among the many individuals at a low level. It is possible that the binary scale (yes/no) to solicit observance of the subject's behavior might have led to a higher likelihood of floor effect. However, unlike other Likert scales, the developed tool was rated by the subject's informant and the binary scale seems appropriate. This is because it is difficult to respond on a Likert scale to behaviors observed by others, as well as the risk of distorting the responses of the actual subject.

A ROC analysis was performed to identify the screening performance of SSQ-OR. The AUC of 0.817 suggested a very good accuracy (32). It was slightly higher than the self-reported screening version of Columbia-Suicide Severity Rating Scale (C-SSRS) with an AUC value of 0.72 (46). A cut-off point of four effectively differentiated the suicidal group from the control group with adequate sensitivity and specificity. The SSQ-OR total score was prospectively associated with subsequent suicidal ideation, plan, and attempt during the 6 months follow up, as were the sub-factor scores after adjusting for gender and age. Among the sub-factors, the odds ratio of social and environmental stress factor was consistently higher than suicide and mental health factor. These results suggest that stress is an important variable in suicide risk as well as vulnerability, as seen in the traditional stress-vulnerability model of psychopathology. Indeed, suicide researchers have proposed and proved stress-vulnerability models in suicide (47–49). Taken together, the inclusion of social and environmental stress along with STBs seems to be an advantage in risk assessment.

With regards to gender difference on SSQ-OR total score, the result did not display any difference, which is similar to previous studies on risk assessment that total scores of clinician-rated C-SSRS and self-reported BSSI did not differ by gender (25, 50–53). Although various risk factors for suicide differed by gender (54), it is essential to point out that a complex interplay is usually found between risk factors (9, 47). For example, age was identified as a relevant moderator for gender differences in suicide (55). In most countries, suicide risk is highest in older males, and the risk for suicide attempts is highest in younger females (56, 57). Since suicide risk is affected by the interaction of various factors, the difference in total score might not be noticeable simply by gender.

This study should be seen in the light of its strengths and limitations. To our knowledge, the SSQ-OR is the first observer rating scale that supplements the suicide risk assessment tools consisting of self-reports or clinician interview scales. We developed a comprehensive screening scale that includes a broader context of risk factors as well as STBs. In particular, because warning signs are included in the scale, we intended to detect observable behaviors that signal the imminent risk of STBs. Limitations include the difficulties associated with the possibility of systematic distortion in the rater's observations (e.g., Rosenthal effect, Halo effect). Moreover, it was reported that the level of agreement between self-report and observer varied according to the type of disturbance or the degree of change over time (58). Also, the floor effect was observed, which means that it could be hard to differentiate among the many individuals at a low level for SSQ-OR. Taking these points together, multi methodological assessment in combination with self-report and observer rating scale offers the best in identifying suicide risk. It is important to use the observer rating with a self-report or clinician-administered scale rather than using the SSQ-OR alone. Clinicians could also refine risk formulation by exploring potential areas of discrepancy between the information reported in the SSQ-OR and during the clinical interview (59). Second, although nationwide multi-site sampling was performed, it was validated only for the East Asian population. Future studies in various cultures are necessary to generalize these findings. Finally, due to the limitation of the study design, a 6 months follow-up was conducted for only a subsample of the participants recruited from the Crisis Response Centers.

In conclusion, the results of this study indicate the SSQ-OR appears to be a promising screening measure of suicidal risk. The SSQ-OR exhibits sound psychometric properties with good reliability, validity, and screening ability. The SSQ-OR is a useful screening tool to manage high-risk suicidal individuals in schools, community centers, or clinical settings. The SSQ-OR could be used as a complement to a self-report or clinical-administered scale to screen suicide risk comprehensively. Clinicians and researchers should carefully consider the strengths and weaknesses of this tool before employing it.

Data availability statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics statement

The studies involving human participants were reviewed and approved by Institutional Review Board of Samsung Medical Center. The patients/participants provided their written informed consent to participate in this study.

Author contributions

Y-HC contributed to the search for background literature, to conduct the statistical analysis, to writing the original draft of the manuscript, to reviewing, and to editing the subsequent manuscript revisions. VY did the study design, coordinated the study, participated in statistical analysis, and provided discussion of results. KY and YC interpreted the data and discussed the study results. DeL coordinated the study, participated in statistical analysis, and interpreted the data. HJL coordinated the study and discussed the study results. DoL did the study design, coordinated the study, interpreted the data, and discussed the study results. HJJ contributed to conceptualization, project administration, and supervision. All authors contributed to writing and editing the manuscript.

Funding

This work was supported by the Technology Innovation Program (or Industrial Strategic Technology Development Program-Source Technology Development and Commercialization of Digital Therapeutics) (20014967, Development of Digital Therapeutics for Depression from COVID19) funded by the Ministry of Trade, Industry and Energy (MOTIE, Korea), and by a grant of the Korea Health Technology R&D Project through the Korea Health Industry Development Institute (KHIDI) funded by the Ministry of Health and Welfare, Republic of Korea (HR21C0885).

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpsyt.2022.945051/full#supplementary-material

References

1. World Health Organization. Preventing Suicide: A Global Imperative: Geneva: World Health Organization (2014).

Google Scholar

2. Statistics Korea. Annual Report on the Cause of Death Statistics. In: Finance MoSa, editor. Daejeon, Korea: Statistics Korea (2021).

3. Franklin JC, Ribeiro JD, Fox KR, Bentley KH, Kleiman EM, Huang X, et al. Risk factors for suicidal thoughts and behaviors: a meta-analysis of 50 years of research. Psychol Bull. (2017) 143:187. doi: 10.1037/bul0000084

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Roos L, Sareen J, Bolton JM. Suicide risk assessment tools, predictive validity findings and utility today: time for a revamp? Neuropsychiatry. (2013) 3:483–95. doi: 10.2217/npy.13.60

CrossRef Full Text | Google Scholar

5. Erford BT, Jackson J, Bardhoshi G, Duncan K, Atalay Z. Selecting suicide ideation assessment instruments: a meta-analytic review. Measur Eval Counsel Develop. (2018) 51:42–59. doi: 10.1080/07481756.2017.1358062

CrossRef Full Text | Google Scholar

6. Baek IC, Jo S, Kim EJ, Lee GR, Lee DH, Jeon HJ. A review of suicide risk assessment tools and their measured psychometric properties in Korea. Front Psych. (2021) 12:9779. doi: 10.3389/fpsyt.2021.679779

PubMed Abstract | CrossRef Full Text | Google Scholar

7. O'Connor RC, Nock MK. The psychology of suicidal behaviour. Lancet Psych. (2014) 1:73–85. doi: 10.1016/S2215-0366(14)70222-6

PubMed Abstract | CrossRef Full Text | Google Scholar

8. Rudd MD, Berman AL, Joiner TE Jr, Nock MK, Silverman MM, Mandrusiak M, et al. Warning signs for suicide: theory, research, and clinical applications. Sui Life-Threat Behav. (2006) 36:255–62. doi: 10.1521/suli.2006.36.3.255

PubMed Abstract | CrossRef Full Text | Google Scholar

9. Turecki G, Brent DA. Suicide and suicidal behaviour. Lancet. (2016) 387:1227–39. doi: 10.1016/S0140-6736(15)00234-2

PubMed Abstract | CrossRef Full Text | Google Scholar

10. The Assessment and Management of Risk for Suicide Working Group. In: Va/Dod Clinical Practice Guideline for Assessment and Management of Patients at Risk for Suicide. Washington, DC. Department of Veterans Affairs DoD (2013).

Google Scholar

11. Saab MM, Murphy M, Meehan E, Dillon CB, O'Connell S, Hegarty J, et al. Suicide and self-harm risk assessment: a systematic review of prospective research. Arch Sui Res. (2021) 3:1–21. doi: 10.1080/13811118.2021.1938321

PubMed Abstract | CrossRef Full Text | Google Scholar

12. Cukrowicz KC, Duberstein PR, Vannoy SD, Lin EH, Unützer J. What factors determine disclosure of suicide ideation in adults 60 and older to a treatment provider? Sui Life-Threat Behav. (2014) 44:331–7. doi: 10.1111/sltb.12075

PubMed Abstract | CrossRef Full Text | Google Scholar

13. Bernecker SL, Zuromski KL, Gutierrez PM, Joiner TE, King AJ, Liu H, et al. Predicting suicide attempts among soldiers who deny suicidal ideation in the army study to assess risk and resilience in servicemembers (army starrs). Behav Res Ther. (2019) 120:103350. doi: 10.1016/j.brat.2018.11.018

PubMed Abstract | CrossRef Full Text | Google Scholar

14. Høyen KS, Solem S, Cohen LJ, Prestmo A, Hjemdal O, Vaaler AE, et al. Non-disclosure of suicidal ideation in psychiatric inpatients: rates and correlates. Death Stud. (2021) 7:1–9. doi: 10.1080/07481187.2021.1879317

PubMed Abstract | CrossRef Full Text | Google Scholar

15. Friedlander A, Nazem S, Fiske A, Nadorff MR, Smith MD. Self-concealment and suicidal behaviors. Sui Life-Threat Behav. (2012) 42:332–40. doi: 10.1111/j.1943-278X.2012.00094.x

PubMed Abstract | CrossRef Full Text | Google Scholar

16. Beck AT, Kovacs M, Weissman A. Assessment of suicidal intention: the scale for suicide ideation. J Consult Clin Psychol. (1979) 47:343. doi: 10.1037/0022-006X.47.2.343

PubMed Abstract | CrossRef Full Text | Google Scholar

17. Lewis AJ, Bertino MD, Bailey CM, Skewes J, Lubman DI, Toumbourou JW. Depression and suicidal behavior in adolescents: a multi-informant and multi-methods approach to diagnostic classification. Front Psychol. (2014) 5:766. doi: 10.3389/fpsyg.2014.00766

PubMed Abstract | CrossRef Full Text | Google Scholar

18. Hawes M, Yaseen Z, Briggs J, Galynker I. The modular assessment of risk for imminent suicide (Maris): a proof of concept for a multi-informant tool for evaluation of short-term suicide risk. Compr Psych. (2017) 72:88–96. doi: 10.1016/j.comppsych.2016.10.002

PubMed Abstract | CrossRef Full Text | Google Scholar

19. Bagge CL, Littlefield AK, Glenn CR. Trajectories of affective response as warning signs for suicide attempts: an examination of the 48 h prior to a recent suicide attempt. Clin Psychol Sci. (2017) 5:259–71. doi: 10.1177/2167702616681628

CrossRef Full Text | Google Scholar

20. Kumar R, Indrayan A. Receiver operating characteristic (Roc) curve for medical researchers. Indian Pediatr. (2011) 48:277–87. doi: 10.1007/s13312-011-0055-4=

PubMed Abstract | CrossRef Full Text | Google Scholar

21. Korea Foundation for Suicide Prevention. Suicide Characteristic Analysis Report, 2013–2017. Seoul, Korea: Korea Foundation for Suicide Prevention (2019).

22. Korea Foundation for Suicide Prevention. Psychological Autopsy Interview Result Report, 2019. Seoul, Korea: Korea Foundation for Suicide Prevention (2020).

23. Beck AT, Steer RA, Ranieri WF. Scale for suicide ideation: psychometric properties of a self-report version. J Clin Psychol. (1988) 44:499–505. doi: 10.1002/1097-4679(198807)44:4<499::aid-jclp2270440404>3.0.co;2-6

PubMed Abstract | CrossRef Full Text | Google Scholar

24. Lee HS, Kwon JH. Validation for the beck scale for suicide ideation with Korean university students. Korean J Clin Psychol. (2009) 28:1155–72. doi: 10.15842/kjcp.2009.28.4.011

PubMed Abstract | CrossRef Full Text

25. Choi YH, Lee EH, Hwang ST, Hong SH, Kim JH. Reliability and validity of the beck scale for suicide ideation (BSS) in Korean adult participants. Korean J Clin Psychol. (2020) 39:111–23. doi: 10.15842/kjcp.2020.39.2.003

CrossRef Full Text | Google Scholar

26. Han C, Jo SA, Kwak JH, Pae CU, Steffens D, Jo I, et al. Validation of the patient health questionnaire-9 Korean version in the elderly population: the Ansan geriatric study. Compr Psychiatry. (2008) 49:218–23. doi: 10.1016/j.comppsych.2007.08.006

PubMed Abstract | CrossRef Full Text | Google Scholar

27. Kim YE, Lee B. The psychometric properties of the patient health questionnaire-9 in a sample of Korean University students. Psychiatry Investig. (2019) 16:904–10. doi: 10.30773/pi.2019.0226

PubMed Abstract | CrossRef Full Text | Google Scholar

28. Li CH. Confirmatory factor analysis with ordinal data: comparing robust maximum likelihood and diagonally weighted least squares. Behav Res Methods. (2016) 48:936–49. doi: 10.3758/s13428-015-0619-7

PubMed Abstract | CrossRef Full Text | Google Scholar

29. Beauducel A, Herzberg PY. On the performance of maximum likelihood vs. means and variance adjusted weighted least squares estimation in Cfa. Struct Equ Model. (2006) 13:186–203. doi: 10.1207/s15328007sem1302_2

CrossRef Full Text | Google Scholar

30. Cudeck R, Henly SJ. Model selection in covariance structures analysis and the “problem” of sample size: a clarification. Psychol Bull. (1991) 109:512–9. doi: 10.1037/0033-2909.109.3.512

PubMed Abstract | CrossRef Full Text | Google Scholar

31. Hu Lt, Bentler PM. Cutoff criteria for fit indexes in covariance structure analysis: conventional criteria vs. new alternatives. Struct Eq Model. (1999) 6:1–55. doi: 10.1080/10705519909540118

CrossRef Full Text | Google Scholar

32. Um O, Cn O. Evaluating measures of indicators of diagnostic test performance:fundamental meanings and formulars. J Biomet Biostat. (2012) 2012:132. doi: 10.4172/2155-6180.1000132

CrossRef Full Text | Google Scholar

33. Costello AB, Osborne J. Best practices in exploratory factor analysis: four recommendations for getting the most from your analysis. Pract Assess Res Eval. (2005) 10:7. doi: 10.7275/jyj1-4868

CrossRef Full Text | Google Scholar

34. Thurstone LL. Multiple-Factor Analysis; a Development and Expansion of the Vectors of Mind. Chicago, IL, US: University of Chicago Press (1947). xix, 535-xix, p.

Google Scholar

35. Deegan J. On the occurrence of standardized regression coefficients greater than one. Educ Psychol Meas. (1978) 38:873–88. doi: 10.1177/001316447803800404

CrossRef Full Text | Google Scholar

36. Goldston DB. Measuring Suicidal Behavior and Risk in Children and Adolescents: Washington, DC: American Psychological Association (2003). doi: 10.1037/10609-000

PubMed Abstract | CrossRef Full Text | Google Scholar

37. Fazel S, Runeson B. Suicide. N Eng J Med. (2020) 382:266–74. doi: 10.1056/NEJMra1902944

PubMed Abstract | CrossRef Full Text | Google Scholar

38. Yoshimasu K, Kiyohara C, Miyashita K. The Stress Research Group of the Japanese Society for H. Suicidal risk factors and completed suicide: meta-analyses based on psychological autopsy studies. Environ Health Prevent Med. (2008) 13:243–56. doi: 10.1007/s12199-008-0037-x

PubMed Abstract | CrossRef Full Text | Google Scholar

39. Schneider B, Grebner K, Schnabel A, Hampel H, Georgi K, Seidler A. Impact of employment status and work-related factors on risk of completed suicide: a case–control psychological autopsy study. Psychiatry Res. (2011) 190:265–70. doi: 10.1016/j.psychres.2011.07.037

PubMed Abstract | CrossRef Full Text | Google Scholar

40. Klonsky ED, May AM, Saffer BY. Suicide, suicide attempts, and suicidal ideation. Annu Rev Clin Psychol. (2016) 12:307–30. doi: 10.1146/annurev-clinpsy-021815-093204

PubMed Abstract | CrossRef Full Text | Google Scholar

41. Rihmer Z. Strategies of suicide prevention: focus on health care. J Affect Disord. (1996) 39:83–91. doi: 10.1016/0165-0327(96)00007-9

PubMed Abstract | CrossRef Full Text | Google Scholar

42. Anestis MD, Soberay KA, Gutierrez PM, Hernández TD, Joiner TE. Reconsidering the link between impulsivity and suicidal behavior. Pers Soc Psychol Rev. (2014) 18:366–86. doi: 10.1177/1088868314535988

PubMed Abstract | CrossRef Full Text | Google Scholar

43. Van Orden KA, Witte TK, Cukrowicz KC, Braithwaite SR, Selby EA, Joiner TE Jr. The interpersonal theory of suicide. Psychol Rev. (2010) 117:575–600. doi: 10.1037/a0018697

PubMed Abstract | CrossRef Full Text | Google Scholar

44. Conwell Y, Duberstein PR, Cox C, Herrmann JH, Forbes NT, Caine ED. Relationships of age and axis i diagnoses in victims of completed suicide: a psychological autopsy study. Am J Psychiatry. (1996) 153:1001–8. doi: 10.1176/ajp.153.8.1001

PubMed Abstract | CrossRef Full Text | Google Scholar

45. Ribeiro JD, Huang X, Fox KR, Franklin JC. Depression and hopelessness as risk factors for suicide ideation, attempts and death: meta-analysis of longitudinal studies. British Journal of Psychiatry. (2018) 212:279–86. doi: 10.1192/bjp.2018.27

PubMed Abstract | CrossRef Full Text | Google Scholar

46. Bjureberg J, Dahlin M, Carlborg A, Edberg H, Haglund A, Runeson B. Columbia-suicide severity rating scale screen version: initial screening for suicide risk in a psychiatric emergency department. Psychol Med. (2021) 5:1–9. doi: 10.1017/S0033291721000751

PubMed Abstract | CrossRef Full Text | Google Scholar

47. Cohen LJ, Mokhtar R, Richards J, Hernandez M, Bloch-Elkouby S, Galynker I. The narrative-crisis model of suicide and its prediction of near-term suicide risk. Sui Life-Threat Behav. (2021) 65:1–13. doi: 10.1111/sltb.12816

PubMed Abstract | CrossRef Full Text | Google Scholar

48. van Heeringen K. Stress-diathesis model of suicidal behavior. In: The Neurobiological basis of Suicide. Boca Raton, FL: CRC Press; Taylor and Francis (2012). p. 51–113.

Google Scholar

49. Rich AR, Bonner RL. Concurrent validity of a stress–vulnerability model of suicidal ideation and behavior: a follow-up study. Sui Life-Threat Behav. (1987) 17:265–70. doi: 10.1111/j.1943-278X.1987.tb00067.x

PubMed Abstract | CrossRef Full Text | Google Scholar

50. Serrani Azcurra D. Psychometric validation of the columbia-suicide severity rating scale in Spanish-speaking adolescents. Colomb Med. (2017) 48:174–82. doi: 10.25100/cm.v48i4.2294

PubMed Abstract | CrossRef Full Text | Google Scholar

51. Healy DJ, Barry K, Blow F, Welsh D, Milner KK. Routine use of the beck scale for suicide ideation in a psychiatric emergency department. Gen Hosp Psychiatry. (2006) 28:323–9. doi: 10.1016/j.genhosppsych.2006.04.003

PubMed Abstract | CrossRef Full Text | Google Scholar

52. Beck AT, Steer RA. Beck Scale for Suicidal Ideation. San Antonio, TX: The Psychological Corporation (1991).

Google Scholar

53. Mirkovic B, Belloncle V, Pellerin H, Guilé J-M, Gérardin P. Gender differences related to spirituality, coping skills and risk factors of suicide attempt: a cross-sectional study of french adolescent inpatients. Front Psych. (2021) 12:7383. doi: 10.3389/fpsyt.2021.537383

PubMed Abstract | CrossRef Full Text | Google Scholar

54. Qin P, Agerbo E, Westergård-Nielsen N, Eriksson T, Mortensen PB. Gender differences in risk factors for suicide in Denmark. Br J Psych. (2000) 177:546–50. doi: 10.1192/bjp.177.6.546

PubMed Abstract | CrossRef Full Text

55. Barrigon ML, Cegla-Schvartzman F. Sex, gender, and suicidal behavior. In: Baca-Garcia E, editor. Behav Neurobiol Sui Self Harm. Cham: Springer International Publishing (2020). p. 89-115. doi: 10.1007/7854_2020_165

PubMed Abstract | CrossRef Full Text | Google Scholar

56. Saxena S, Krug E, Chestnov O. Department of Mental Health and Substance Abuse. Preventing Suicide: A global imperative. Geneva: World Health Organization (2014).

PubMed Abstract

57. Naghavi M. Global, regional, and national burden of suicide mortality 1990 to 2016: systematic analysis for the global burden of disease study 2016. BMJ. (2019) 364:l94. doi: 10.1136/bmj.l94

PubMed Abstract | CrossRef Full Text | Google Scholar

58. Möller HJ. Standardised rating scales in psychiatry: methodological basis, their possibilities and limitations and descriptions of important rating scales. World J Biol Psych. (2009) 10:6–26. doi: 10.1080/15622970802264606

PubMed Abstract | CrossRef Full Text | Google Scholar

59. Rudd MD, Bryan CJ. Finding effective and efficient ways to integrate research advances into the clinical suicide risk assessment interview. Front Psych. (2022) 13:6244. doi: 10.3389/fpsyt.2022.846244

PubMed Abstract | CrossRef Full Text | Google Scholar