Impact of Systematic Tailored Assessment for Responding to Suicidality (STARS) Protocol Training on Mental Health Professionals' Attitudes, Perceived Capabilities, Knowledge, and Reluctance to Intervene

Background and Aims: Systematic Tailored Assessment for Responding to Suicidality (STARS) protocol and associated training were developed with the key objectives of supporting clinicians to conduct a suicide enquiry, obtaining a comprehensive account of psycho-social factors contributing to suicidality, and collaboratively developing a safety plan with clients. STARS training aims to address knowledge, attitudes and capabilities that influence intervention behavior/skills. This study aimed to examine associations between clinician characteristics and pre-training competencies in suicide risk assessment (SRA), as well as the impact of STARS training workshop on clinician competencies; and to determine the predictors of SRA training outcomes.

Method: Australian mental health professionals working with suicidal persons who undertook the STARS 2-day face-to-face workshop between 2018 and 2020 completed an online survey at pre- and post-training. Of the 222 participants who completed the pre-training questionnaire, 144 (64.9%) also completed the post-training questionnaire. Participants were mostly female (75.7%), had completed a university degree (86.4%), had <10 years of experience in suicide prevention (71.7%), and were allied and mental health professionals (78.1%). We used linear mixed-effects regression for statistical analyses.

Results: STARS participants who reported higher perceived capability at baseline had significantly greater formal and informal training, more years of experience in suicide prevention, and were more likely to have experienced client suicide and/or suicide attempt and to report fewer SRA related fears. We found overall significant positive impacts of STARS training on clinician competencies (attitudes, perceived capability, declarative knowledge) from pre- to post-training. The most distinct changes following STARS training were for perceived capability and declarative knowledge. Participants who had more positive attitudes after training were significantly more likely to have had less prior supervision/mentoring. Reluctance to intervene was not found to significantly change after training.

Conclusions: We found evidence that attitudes, perceived capability and declarative knowledge changed positively from pre- to post-STARS training among mental health professionals. Underpinned by the minimum standardized SRA competencies, STARS training may be critical for informing evidence-based knowledge and skills in SRA and safety planning.

Introduction

Suicide is one of the leading causes of death worldwide with over 700,000 suicides recorded in 2019 (1). Comprehensive assessment and monitoring of suicidality are key to preventing suicide (2). Uncovering suicide intent and understanding the psycho-social needs of vulnerable persons require capabilities largely neglected in the training and education of mental health professionals (3, 4). The current recommendations for comprehensive suicide risk assessment (SRA) practice extend beyond administration of clinician- and client-rated risk stratification assessment tools (i.e., low-medium-high risk), which are limited by their lack of predictive reliability (5, 6). Systematic psycho-social, person-centered, needs-based assessment has been advocated as one alternative approach to SRA (7), with a focus on understanding the individual within their ecological context or social environment (6, 8).

Cramer and Kapusta (8) posited that “structured professional judgment (SPJ) of key multi-level risk factors” is needed for suicide risk assessment (p. 8). SPJ offers an opportunity to explore multiple factors guided by empirical data, to inform care decisions (9) as opposed to unstructured discretionary clinician judgement (10), which is likely to have low reliability due to its subjectivity (11). The Systematic Tailored Assessment for Responding to Suicidality (STARS) protocol (12) (see Supplementary Material 1 for protocol description) and associated training is one such SPJ approach. The STARS protocol and training were developed with the key objectives of supporting clinicians to conduct a suicide enquiry, obtain a comprehensive account of psycho-social factors contributing to suicidality, and collaboratively develop a safety plan and management response with clients (13). Clinicians' feedback on the STARS protocol highlighted the need for training to improve ease of administration and clinicians' competency and confidence with SRA (13, 14).

To date, the suicide prevention training literature is dominated by gatekeeper training (GKT), which aims to identify suicide warning signs, engage clients, and make referrals. Less emphasis has been placed on comprehensive suicide-specific training for the workforce undertaking SRA and safety planning (15, 16), and in postgraduate curriculae (17, 18). STARS training (12) is designed to enhance SRA and safety planning capabilities of the mental health workforce and addresses competencies aligned with those of Cramer et al. (15) (see Table 1 below). A 2-day training program including six modules based on these competencies was introduced in 2018, following from the original 1-day, non-mandatory STARS training [based on the original STARS protocol (2015)]. Developed by lead author JH, STARS 2-day training also includes important design input and co-facilitation by those with a lived experience of suicide. To date, has been delivered to approximately 600 mental health workers in Australia. Despite the strong uptake of training, evaluation of training impacts has lagged behind roll-out of STARS training.

TABLE 1

Table 1. Suicide prevention competencies by Cramer et al. (15) and alignment with the STARS protocol training competencies.

STARS training aims to address knowledge, attitudes and capabilities that influence intervention behavior/skills. Based on Bandura's (21) social cognitive theory, Burnette et al. (22) conceptual model for suicide prevention training contends that training has its effect upon clinicians' intervention behaviors by altering their competencies, as reflected by knowledge of suicide, beliefs and attitudes toward suicide prevention, reluctance or stigma, and self-efficacy to intervene. Suicide knowledge refers to both subjective and declarative understanding of suicide and its causality. Attitudes (or beliefs) toward suicide and its prevention refer to whether suicide is perceived as preventable and whether intervention with an individual who is suicidal is appropriate. Reluctance to intervene is a perception of not wanting to intervene or not being responsible for intervening to prevent suicidality. Self-efficacy refers to perceptions of capability or feeling comfortable with intervening with someone who is suicidal (22). The model also recognizes the role of individual factors in influencing the development of competencies and associated intervention behaviors including demographic (e.g., age, gender, or ethnicity) and professional characteristics (e.g., education, prior training in suicide prevention, years of experience, and discipline), as well as the role of the social context within which the practitioner intervenes (e.g., resources and organizational support for training) (22).

While originally applied to understanding impacts of GKT, this conceptualization also has utility for investigating the impact of SRA training given the overlap in areas of practice capabilities (23). Although SRA training has been shown to be associated with increased clinician competencies such as knowledge, confidence in intervention skills and attitudes toward suicide prevention (16, 24, 25) the factors influencing clinicians' response to training are important to understand so as to inform the individualization of training.

Clinicians attending SRA training vary considerably in pre-training characteristics such as job role/discipline, prior training (formal and informal), and experience in suicide prevention (26). Formal training refers to workshops or structured education in SRA or intervention, whereas informal training refers to supervision and/or mentoring for SRA practice (27). LoParo et al. (28) found that multiple sessions of formal training in SRA, as opposed to a single session, were associated with greater increases in confidence and following best-practice SRA guidelines. However, the impact of informal training or supervision on clinician competencies is under-researched, despite the recognized importance of supervision in SRA (29, 30).

In addition to the type of prior training, clinician factors such as experience of client loss to suicide or suicide attempt and fear associated with SRA and suicide related outcomes potentially influence clinician practices following training, as may their perceived capability, knowledge, and attitudes to suicide prevention. Studies have shown that 23–65% of health professionals across mental health disciplines report losing a client to suicide (26, 31). Prior experience of clients' suicide attempt or client suicide enhance fidelity to structured assessment/intervention approaches (27) and competency in SRA (31) and reduce fears of engaging in SRA and intervention (32). Conversely, lack of knowledge and confidence in SRA (33, 34), reduced self-efficacy (35), SRA related and intervention fears (36), and negative attitudes (37, 38) and beliefs (39) about suicide are associated with reduced likelihood of intervening.

The STARS protocol was developed as a comprehensive psycho-social needs-based assessment of suicidality. However, there is limited evidence concerning the outcomes of SRA training, including for STARS, as well as the influence of demographic/clinician characteristics on training outcomes. Given the potential influence of previous training and work experiences relevant to suicide prevention on SRA training outcomes, it was also of interest to examine associations between clinician characteristics (e.g., formal and informal training and loss of client to suicide) and competencies prior to training. Such findings may inform the tailoring of training to clinicians' professional development needs.

Accordingly, the current study's specific objectives are to:

• Examine associations between clinician characteristics and pre-training clinician competencies (i.e., attitudes toward suicide prevention, perceived capability in SRA, knowledge about SRA, and reluctance to intervene). It was hypothesized that greater previous formal and informal training would be associated with greater knowledge and perceived capability, more positive attitudes and reduced reluctance to intervene at pre-training.

• Determine the impact of STARS 2-day training workshop on clinician competencies. It was hypothesized that STARS training would be associated with significant improvements (from pre- to post-training) in clinician competencies such that negative attitudes to suicide prevention would decline, perceived capability would increase, knowledge about SRA would increase, and reluctance to intervene would decline.

• Determine whether clinician characteristics predict STARS training outcomes (i.e., pre-to post-training gains on clinician competencies). Following Burnette et al. (22) theory, it was hypothesized that clinician characteristics would predict greater improvements in clinician competencies.

The influence of other pre-training variables (e.g., demographics, SRA related fears, prior client suicide or attempt) on SRA training outcomes represented an exploratory component, for which there were no hypotheses.

Method

Procedure

Australian mental health professionals working with suicidal persons who undertook the STARS 2-day face-to-face workshop between 2018 and 2020 were invited to participate in this study. All participants were sent a link to an online survey before and after the STARS training.

All procedures were approved by the Griffith University Human Research Ethics Committee (Ref number: 2015/813/HREC). Surveys were set up in Research Electronic Data Capture (REDCap) a secure online instrument developed by Vanderbilt University to distribute surveys for research purposes (40). All participants indicated consent by proceeding with the online survey. Pre-training surveys were distributed to all registered participants up to 1 week prior to their training attendance, with three email reminders sent within this week. Post-training surveys were disseminated within 48 h post training, with three reminders up to 2 weeks post workshop.

Survey Content and Measures

Predictor Variables

Demographic factors and experiences of client suicidality. Demographic and background information regarding age, gender, professional role, education, years in suicide prevention, SRA related fears, SRA training and supervision—more specifically, the amount of formal training (e.g., workshops) and informal training (supervision/mentoring)—and experience of client suicide and/or suicide attempt was collected. A “yes” response was allocated to participants who indicated having lost a client to suicide and/or suicide attempt, whereas a “no” was allocated to having no experiences of client suicide or suicide attempt.

Fears about conducting SRA included nine potential fears clinicians may face in conducting SRA. Items were informed by literature around clinician fears and anxieties concerning SRA or working with clients who are suicidal (41, 42). Specific items around fear of pushing a client toward suicide, or a client attempting or dying by suicide were based on Roush et al.'s (30) items developed specifically for their study. Participants were asked “In the past, what have been some of your reasons for not conducting a suicide risk assessment?” and were provided with a categorical list of nine common fears reported in the literature (e.g., “Fear that I might push the client toward suicide” and, “Fear of a positive answer requiring more clinical time”). Responses were rated as present or absent and the number of fears present was totalled (0–9). This scale showed good internal consistency (Cronbach's α = 0.79) for the current sample.

Training Outcome Variables

The STARS pre- and post-training survey includes a combination of standardized and researcher developed measures assessing clinician competency outcomes, namely, attitudes, perceived capability, declarative knowledge and reluctance to intervene.

Attitudes to Suicide Prevention Scale (ASP) is a 14-item scale measuring attitudes to suicide and suicide prevention and includes items associated with responsibility for suicide and its prevention (e.g., “suicide prevention is not my responsibility”), and views of suicide (e.g., “if a person survives a suicide attempt, this was a ploy for attention”) (43). A 5-point Likert scale was used, with 5 reflecting a very negative attitude and 1 reflecting a very positive attitude. An overall score (range: 14–70) was calculated by summing the scores from each item, with higher scores indicating more negative attitudes. Internal consistency for the current sample (Cronbach's α = 0.62) was lower than that reported by authors of the scale (α = 0.77) (43).

Perceived Capability Scale is a 5-item scale that measures a participant's perceptions of competence in SRA capabilities (12). Participants are asked to rate current level of perceived capability (e.g., “How much do you feel you know about the suicidal state?,” “…about engaging suicidal persons?,” “…about suicide risk and protective factors?”) on a 5-point Likert scale ranging from “Very little” (1) to “Everything there is to know” (5). Total scores range from 5 to 25, with higher scores reflecting greater perceived capability. This scale showed good internal consistency (Cronbach's α = 0.87) for the current sample.

Declarative Knowledge Scale includes 19 items directly aligned with the competencies of all training modules within the STARS training, as developed by the authors. Example items include, “Which one of the following is not a problem in current approaches to suicide risk assessment? (choose correct item)”; “Safety planning is primarily about (choose correct item)”; STARS has been founded strictly on CBT (Cognitive Behavior Therapy) models of understanding suicidality “(True or False).” Correct answers were totalled, with scores ranging from 0 to 22 (note: one item is scored out of 4; 18 items are scored 0 or 1). This scale showed good internal consistency (Cronbach's α = 0.81) for the sample.

Reluctance to Intervene Scale is a 9-item scale that measures reluctance to intervene with a suicidal person (44). Each item is rated on a 5-point Likert scale ranging from Strongly Disagree (1) to Strongly Agree (5). Two items are reverse-scored, and each item value is summed for a total score ranging from 9 to 45, whereby higher values mean less reluctance. This scale showed poor internal consistency (Cronbach's α = 0.58) for the current sample, although was comparable to the original testing results by the authors of the scale (α = 0.68) (44).

Control Variable

Social Desirability Response Set, a five-item survey, was included as a control variable, to assess the tendency to present oneself in an overly favorable light in terms of attitudes and behaviors (45). Given the sensitivity of the topics of inquiry and potential for demand characteristics to influence clinician's responses (46), social desirability was measured at pre-survey to inspect whether data collected on the attitudinal measures were associated with social desirability. An example item includes: “I am always courteous even to people who are disagreeable” (Item 1). Items were rated on a 5-point scale from 1 (definitely true) to 5 (definitely false), with two items reversed scored. Responses indicative of “social desirability responding” are scored as 1, while all other responses are scored 0. Total scores above 5 indicate a tendency toward socially desirable responses in answering the questionnaire. This scale showed modest internal consistency (Cronbach's α = 0.66).

Statistical Analysis

Preliminary analyses were conducted using simple means with standard deviations, frequencies and Pearson's correlations. Further analyses used linear mixed-effect modeling to predict the main outcome measures. This method is particularly useful for repeated measures as it accounts for both within- and between-subjects variance, including the correlation between the repeated measures of participants (47). As the linear-mixed effect models expects normal distribution, we examined Q-Q plots and skewness and kurtosis; the majority of the outcome measures were within the normal distribution range [skewness or kurtosis between + 1.5 and −1.5 as by (48)]; for some, removal of extreme outliers was required.

To examine associations between baseline (pre-training) scores on the training outcome, the predictor variables of demographic and clinician characteristics including age, gender, years in suicide prevention role, experience of client suicide or suicide attempt, the amount of formal (workshops etc.) and informal training (supervision/case study), and number of SRA related fears were entered simultaneously as fixed estimates in linear mixed-effects models, in the prediction of changes in each outcome measure (attitudes, perceived capability, declarative knowledge, and reluctance to intervene). Social desirability was entered as a control variable in all analyses. To reduce multicollinearity all variables included as fixed effects were centered (49).

To measure change between pre- and post-STARS training (outcome), time was used as a categorical variable (pre vs. post) under fixed effects. In addition, the same variables as for the baseline analyses were included as fixed effects to assess their potential confounding effect in the adjusted model. Time (pre vs. post) was included as a repeated effect. First-Order Autoregressive (AR1), First-Order Autoregressive Heterogenous (ARH1) and Unstructured (UN) covariance structures were examined using −2 Res Log Likelihood and Akaike's Information Criterion (AIC). All structures were applied to the levels of group (location)^*person (as STARS is delivered in groups, which means that participants are nested within groups). Random intercepts for participants were included to model for the correlation of within person factors at baseline. The ARH1 structure was identified as the model with the best fit with all outcome variables. Effect sizes were not computed because there is no broad agreement on which should be used with designs of this type.

To identify predictors of change in outcome measure, we measured change for all outcome measures by subtracting the post-test from the pre-test score [i.e., post minus pre; (50)]. The same variables as for the baseline analyses were included as fixed effects, and location of the workshop (coded 1–13) was entered as random effects.

A comparison was made between those who completed the survey both before and after the training with those who completed the survey only prior to the training. This showed that time working in suicide prevention was significantly different [Chi2 = 4.03 (df = 1) p = 0.045; further details in SM 3], with individuals working less years in suicide prevention being less likely to complete the post-training survey. Nevertheless, linear mixed-effects regression accommodates unbalanced data with the assumption that missing data are missing at random in the outcome measures (time: before, after). Cases with missing values were not dropped from the analyses, however, missing data for the predictor variables was addressed by listwise deletion of cases. We first used Little's test of missing completely at random (MCAR) and identified that there was no systematic variation between missing data points in different sets used for different outcome measures and separately also including post data. Therefore, we concluded that data was MCAR. In general, it has been shown that multiple imputation does not change the results with repeated measures (51). Over 10% of missing cases were identified due to missingness in predictor variables. Therefore, we conducted further sensitivity analyses with multiple imputation (n = 10) for the STARS impact. A probability level of 0.05 was employed for all statistical tests. IBM SPSS ver 27.0 was used.

Results

Participants

The demographic and clinical characteristics of participants are presented in Table 2. A total of 222 participants completed the pre-training survey. Participants were mostly female (75.7%), had completed a university undergraduate or post-graduate degree (86.4%), were non-Indigenous (88.7%), had <10 years of experience in suicide prevention (71.7%), and were allied and mental health professionals (78.1%). Of the 222 participants who completed the pre-training questionnaire, 144 (64.9%) also completed the post-training questionnaire. After the first STARS training workshop, the researchers considered more closely the theoretical model of training evaluation and influences on intervention behavior by Burnette et al. (22), which suggests that reluctance be measured as a variable potentially amendable to training influences. Therefore, the measure of reluctance was only incorporated into the survey administration for 12 out of 13 workshops in this study. Data were available for 189 participants on this measure at pre-training which was used for the mixed effects model analysis.

TABLE 2

Table 2. Description of study participants.

Associations Between Clinician Characteristics and Pre-training Competencies

Correlation analyses conducted between pre-training competencies and clinician characteristics (see Table 3) identified that negative attitudes to suicide prevention had significant but weak correlations with number of fears about conducting SRA (r = 0.16, p = 0.023). Perceived capability had a strong positive correlation with amount of formal training (r = 0.48, p < 0.001) and informal training (r = 0.47, p < 0.001), a moderate positive correlation with years of experience in suicide prevention (r = 0.34, p < 0.001). Further, greater perceived capability was related to having prior experience of client suicide and/or suicide attempt (r = 0.31, p < 0.001), and was associated with fewer SRA related fears (r = −0.30, p < 0.001). Likewise, more declarative knowledge was significantly (but weakly) correlated with having prior experience of client suicide or suicide attempt (r = 0.14, p = 0.002) and was moderately correlated with more informal training (r = 0.24, p < 0.001). Reluctance to intervene had significant, but weak correlation by gender (r = −0.17, p = 0.024) with females presenting lower levels of reluctance.

TABLE 3

Table 3. Correlations of variables included at baseline.

The main fixed effects for the linear mixed-effects regression for examining the effects of predictor variables upon outcome are presented in Table 4. None of the expected predictors remained significant in models predicting the baseline scores for attitudes, declarative knowledge and reluctance. However, baseline or pre-training perceived capability was predicted by amount of formal (F = 8.51, p < 0.001) and informal (F = 6.39, p = 0.002) training, whereby an increased amount of training was associated with greater perceived capability. In addition, higher perceived capability at baseline was associated with the experience of a client suicide and/or suicide attempt (F = 5.16, p = 0.024) and lower number of SRA related fears (F = 10.03, p = 0.002) predicted capability.

TABLE 4

Table 4. Fixed effect estimates for the predictors of the main outcome measure at baseline (pre-training) from the linear mixed-effects models.

Change in Competencies From Pre- to Post-training

Table 5 presents the fixed estimate in the change in time (pre vs. post-training competencies) including unadjusted and adjusted models. Figure 1 shows changes for all outcome measures using estimated marginal means with their confidence interval. The results show significant changes in all main outcome measures except for reluctance. Significant increases were evident for perceived capability (F = 232.48, p < 0.001), declarative knowledge (F = 176.56, p < 0.001) and attitudes (F = 6.24, p = 0.014). Further sensitivity analyses using multiple imputation showed the same results.

TABLE 5

Table 5. Fixed effect estimates of time (change pre to post) predicting main outcome measures in unadjusted and adjusted linear mixed-effects models.

FIGURE 1

Figure 1. Estimated marginal means of main outcome measure pre and post intervention.

Predictors of Change

The fixed-effect estimates in the models examining predictors of training outcomes (post minus pre-training scores on each outcome measure) are presented in Table 6. When all predictor variables were entered simultaneously, controlling for social desirability, change in attitudes (ASP: desired change = decline) was predicted only by amount of previous informal training (F = 4.42, p = 0.014); more specifically, those with more previous informal training (moderate to a lot) reported less attitudinal change. Change in capability (desired change = increase) was predicted only by amount of previous formal training (F = 5.49, p = 0.005); those with more previous formal training showed less improvement in capability. Change in declarative knowledge (desired change = increase) was predicted only by gender (F = 3.43, p = 0.036; those reporting their gender as “other” (N = 10) showed greater change in declarative knowledge compared to females (reference group), while there was no difference between males and females.

TABLE 6

Table 6. Fixed effect estimates predicting change (post-pre) of main outcome measures in linear mixed-effects models.

Discussion

STARS is a client-centered, psycho-social, needs-based, semi-structured interview protocol (12). STARS training is designed to support acquisition of clinician competencies required for comprehensive SRA and safety planning. Understanding the impacts of STARS training on clinician competency is critical for ongoing quality assurance and evaluative purposes.

Clinical Characteristics and Pre-training Competencies

Participants' demographic and clinical characteristics were generally not associated with the baseline clinician competency measures in the cross-sectional correlations at baseline. However, based on the final model participants who reported higher perceived capability at baseline tended to have had greater formal and informal training, more years of experience in suicide prevention, and to be more likely to have experienced client suicide and/or suicide attempt and to report fewer SRA related fears. The association between perceived capability at baseline and the amount of formal and informal training is consistent with the results of both LoParo et al. (28, 52). Also, the finding that perceived capability in SRA was associated with having prior experience of a client suicide/attempt and lower number of fears around SRA is consistent with previous studies (53, 54), and in accordance with a systematic review (55). Lund et al. (54) found that more frequent experience working with suicidal clients was related to higher perceived competency with suicide prevention. Silva et al. (53) found that health care staff who had more clinical contact, and those who experienced a client suicide, had higher confidence scores in SRA training. The current findings extend upon previous research by demonstrating that, in mental health professionals undergoing SRA training, perceptions of capability were higher at baseline for those with more training, more years of experience, greater probability of having experienced client suicide/attempt and having fewer fears about SRA practice. A possible explanation for this finding is that professionals who have higher perceived capabilities may, by virtue of their role, have more frequent contact with suicidal clients, thus increasing the likelihood of having experience of client suicide/attempt.

Due to the cross-sectional nature of the data, it is not possible to determine whether fears of engaging in SRA practice contribute to or are influenced by perceived capability. Nonetheless, the negative association between fears and perceived capability in SRA has important implications for training and suicide intervention more generally (30, 32). If training providers can identify participants who have greater SRA related fears prior to training, there may be scope to address the nature of their concerns during training. Notably, perceptions of having had sufficient training has been found to be related to higher levels of comfort in working with suicidal clients and having less fear in this work (30, 32). Therefore, identifying SRA related fears prior to training and seeking to address these may have positive effects on both fears and perceived capability. However, as such fears were only assessed at baseline in the current study, it is not possible to determine whether SRA related fears were reduced after STARS training.

Changes in Competency After STARS Training and Predictors of Change

Consistent with Burnette et al.'s (22) model, we found overall positive impacts of STARS training on clinician competencies (attitudes, perceived capability, declarative knowledge) from pre- to post-training. The most pronounced changes following STARS training were evident for perceived capability and declarative knowledge, which is consistent with other SRA training findings (35, 56). We note, however, that due to the lack of control group such outcomes cannot conclusively be attributed to training.

In terms of predictors of change, we found that a lower amount of prior formal training and fewer years' experience predicted a greater increase in perceived capability at post-training, potentially due to their greater scope for improvement (i.e., those with less experience and training reported lower perceived capability at baseline). Consistent with the literature on attitudinal changes following SRA training (56, 57), we also found that participants had more positive attitudes toward suicide prevention after training. Further analysis showed that participants with less prior supervision/mentoring experienced the most change in attitudes. It is perhaps not surprising that those who engage in more supervision/mentoring around SRA tend to have positive attitudes prior to training, and therefore are less likely to demonstrate improvement following training.

Reluctance to intervene was not found to significantly change after training in either unadjusted or adjusted models. This contrasts with previous findings on levels of reluctance post GKT (44, 58). However, our finding is not surprising given the high scores (low reluctance) of participants at baseline, which may reflect a potential ceiling effect. Further, the results should be considered with caution considering the low internal reliability of this scale (Cronbach's α = 0.58) for our sample, and the original low to moderate alpha reported by the authors (α = 0.68) (44). Likewise, Ayer et al. (59) reported low internal consistency for this scale (e.g., α = 0.67 and α = 0.64 for Army and Marine Corps participants, respectively). Notably, reluctance to intervene has not previously been evaluated in SRA training (as opposed to GKT); thus, more research on the measurement of this construct and SRA training outcomes is recommended.

Implications

This study provided evidence that three key theoretically based SRA competencies changed between pre- and post-STARS training, namely: more positive attitudes, greater perceived capability and increased declarative knowledge. Few training workshops with SRA competency-based frameworks supporting SPJ are currently available to Australian mental health workers. STARS training reflects such an approach and is well-aligned with Cramer et al.'s (15) SRA competencies, which we believe are critical for informing comprehensive evidence-informed knowledge and skills in SRA. There is a need for ongoing systematic evaluation of STARS, including a controlled trial of outcomes relative to other training (e.g., GKT).

Of note, our study highlighted the potential value of identifying professionals' fears around SRA practice at the outset of training due to the association with perceived capability. Facilitators may provide more emphasis on the impacts of multiple SRA related fears during training to increase participants' insight into impacts on their capability and suggest ways to address these (including seeking supervision). Further, it is possible that increased supervision around SRA practice increases positive attitudes toward suicide prevention, comfort and confidence in conducting SRA, which may subsequently enhance SRA interventions (60). Many benefits of supervision and mentoring have been identified for patient care and professionals' well-being in the mental health field generally (61, 62). However, scant attention has been paid to its application in regard to suicide-specific effective client outcomes, despite its proposed importance (60), and so this form of learning and support is strongly advocated.

Strengths and Limitations

Our results should be interpreted in the context of some limitations. We used a convenience sample, which may not be representative of all mental health workers in Australia who work with suicidal clients. While we used a fairly homogenous sample of mental health workers in our study, in the sense that all had completed STARS training, as recommended by Jager et al. (63), our findings may not be generalized to other settings (e.g., emergency departments or acute inpatient units) where workers are repeatedly exposed to presentations of suicidality. Further, the majority of workshops were conducted in Queensland (9 out of 13), and therefore findings may not be generalizable to the entire Australian mental health workforce. Our self-selected sample of participants seeking SRA training could be particularly motivated to learn and potentially to have positive perceptions about their capability and attitudes toward working in this field. The low baseline levels of reluctance to intervene seem to support this view. We also did not include a control group for evaluating the efficacy of STARS training, and therefore cannot claim that changes in clinician competencies were due to the training per se.

The attrition analysis (35.1% drop-out) indicated that clinicians with more years of experience in suicide prevention (>10) were less likely to complete the post-training surveys. However, the impact of missing data on the analysis was managed by mixed linear modeling analyses. Due to the high turnover of the community mental health workforce in Australia (64), it was not possible to conduct a follow-up of the longer-term effects of training with the broader sample. We attempted to do follow-up contact with all participants on our STARS training database but received approximately 35% returned emails (returned email messages indicated “no longer at this address” or “email invalid”). Finally, the reluctance and attitude measures used in our study showed low internal consistency and therefore results on these measures should be interpreted with caution.

Our study has made a unique contribution to the suicide prevention training literature in that this is the first SRA training evaluated in Australia, which involves a semi-structured interview based on psycho-social needs assessment and structured professional judgement (SPJ) (8). Further, STARS protocol and training includes comprehensive safety planning based on Stanley and Brown's (20) safety planning intervention. The STARS training is underpinned by the globally proposed SRA competencies by Cramer et al. (15), and therefore is informed by empirical evidence about what is expected to be the minimum standards of competency for undertaking comprehensive, person-centered psycho-social risk assessment today.

Conclusion

This study is the first to examine changes in clinician competencies in the context of STARS training. Our preliminary evaluation suggested that such training is associated with increased declarative knowledge, greater perceived capability and more positive attitudes toward suicide prevention. As a general finding, the participants most likely to benefit from STARS training had lower perceived capability or declarative knowledge about SRA prior to STARS or less prior supervision/mentoring experience. A long-term term follow-up of clinicians' use of and adherence to STARS protocol in practice is recommended in future research, along with a controlled evaluation of the impacts of STARS training relative to alternative training programs.

Data Availability Statement

The raw data supporting the conclusions of this article can be made available upon a reasonable request.

Ethics Statement

This study involving human participants was reviewed and approved by Griffith University Human Research Ethics Committee (Ref number: 2015/813/HREC). All participants provided their informed consent to participate in this study.

Author Contributions

JH conceptualized and designed the study and coordinated the STARS training and evaluation project. TO, SS, DD, and EA are supervisors of Ph.D candidate JH. JH and KK analyzed data. JH wrote the manuscript. All authors contributed to the article and approved the submitted version.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Acknowledgments

We would like to acknowledge Jasmine Koo, who aided in the original data analysis phase of this evaluation. We also thank the mental health professionals in the Australian suicide prevention sector who undertook the STARS training and participated in this evaluation. Finally, significant acknowledgment must go to the supervision team (TO, SS, DD, and EA) who supervised JH. Different organizations within the suicide prevention sector in Australia funded their staff to attend STARS training.

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpsyt.2021.827060/full#supplementary-material

References

1. World Health Organization. Suicide. World Health Organization. (2021). Available online at: https://www.who.int/news-room/fact-sheets/detail/suicide (accessed October 11, 2021).

Google Scholar

2. Gunnell D, Appleby L, Arensman E, Hawton K, John A, Kapur N, et al. Suicide risk and prevention during the COVID-19 pandemic. Lancet Psychiatry. (2020) 7:468–71. doi: 10.1016/S2215-0366(20)30171-1

PubMed Abstract | CrossRef Full Text | Google Scholar

3. Jahn DR, Drapeau CW, Wacha-Montes AL, Nadorff MR, Pusateri MJ Jr, Hannah DJ, et al. Characteristics of and barriers to suicidology training in undergraduate and clinically-oriented graduate-level psychology programs. Suicidology. (2017) 8:74–83.

Google Scholar

4. Schmitz WM, Allen MH, Feldman BN, Gutin NJ, Jahn DR, Kleespies PM, et al. Preventing suicide through improved training in suicide risk assessment and care: an American Association of suicidology task force report addressing serious gaps in U.S. mental health training. Suicide Life Threat Behav. (2012) 42:292–304. doi: 10.1111/j.1943-278X.2012.00090.x

PubMed Abstract | CrossRef Full Text | Google Scholar

5. Carter G, Spittal MJ. Suicide risk assessment. Crisis J Crisis Int Suicide Prev. (2018) 39:229–34. doi: 10.1027/0227-5910/a000558

PubMed Abstract | CrossRef Full Text | Google Scholar

6. Franklin JC, Ribeiro JD, Fox KR, Bentley KH, Kleiman EM, Huang X, et al. Risk factors for suicidal thoughts and behaviors: a meta-analysis of 50 years of research. Psychol Bull. (2017) 143:187–232. doi: 10.1037/bul0000084

PubMed Abstract | CrossRef Full Text | Google Scholar

7. Wittouck C, Munck SD, Portzky G, Rijsselberghe LV, Autreve SV, Heeringen KV. A comparative follow-up study of aftercare and compliance of suicide attempters following standardized psychosocial assessment. Arch Suicide Res. (2010) 14:135–45. doi: 10.1080/13811111003704746

PubMed Abstract | CrossRef Full Text | Google Scholar

8. Cramer RJ, Kapusta ND. A social-ecological framework of theory, assessment, and prevention of suicide. Front Psychol. (2017) 8:1756. doi: 10.3389/fpsyg.2017.01756

PubMed Abstract | CrossRef Full Text | Google Scholar

9. Timmins KLE, Evans L, Tully RJ. Inter-rater reliability of the short-term assessment of risk and treatability (START). J Forensic Psychiatry Psychol. (2018) 29:968–88. doi: 10.1080/14789949.2018.1523945

PubMed Abstract | CrossRef Full Text | Google Scholar

10. Ward T, Willis GM. Responsivity dynamic risk factors and offender rehabilitation: a comparison of the good lives model and the risk-need model. In: Laws D, O'Donohue W, editors. Treatment of Sex Offenders. Cham: Springer (2016). doi: 10.1007/978-3-319-25868-3_8

CrossRef Full Text | Google Scholar

11. Paterson B, Dowding D, Harries C, Cassells C, Morrison R, Niven C. Managing the risk of suicide in acute psychiatric inpatients: a clinical judgement analysis of staff predictions of imminent suicide risk. J Mental Health. (2008) 17:410–23. doi: 10.1080/09638230701530234

CrossRef Full Text | Google Scholar

12. Hawgood J, De Leo D. STARS: Screening Tool for Assessing Risk of Suicide. Brisbane: Australian Institute for Suicide Research and Prevention, Griffith University (2018).

Google Scholar

13. Hawgood J, Ownsworth T, Mason H, Spence S, Arensman E, De Leo D. A pilot study of clinicians' perceptions of feasibility, client-centredness and usability of the systematic tailored assessment for recognising suicidality (STARS) protocol. Crisis. (2021) 1–8. doi: 10.1027/0227-5910/a000796

PubMed Abstract | CrossRef Full Text | Google Scholar

14. Hawgood J, De Leo D. Suicide prediction – a shift in paradigm is needed. Crisis. (2016) 251–55. doi: 10.1027/0227-5910/a000440

PubMed Abstract | CrossRef Full Text | Google Scholar

15. Cramer RJ, Johnson SM, McLaughlin J, Rausch EM, Conroy MA. Suicide risk assessment training for psychology doctoral programs: core competencies and a framework for training. Train Educ Profess Psychol. (2013) 7:1–11. doi: 10.1037/a0031836

PubMed Abstract | CrossRef Full Text | Google Scholar

16. Pisani AR, Cross WF, Gould MS. The assessment and management of suicide risk: state of workshop education. Suicide Life Threat Behav. (2011) 41:255–76. doi: 10.1111/j.1943-278X.2011.00026.x

PubMed Abstract | CrossRef Full Text | Google Scholar

17. Hawgood J, Krysinska K, Mooney M, Ozols I, Andriessen K, Betterridge C, et al. Suicidology post graduate curriculum: Priority topics and delivery mechanisms for suicide prevention education. Int J Environ Res Public Health. (2021) 18:9926. doi: 10.3390/ijerph18189926

PubMed Abstract | CrossRef Full Text | Google Scholar

18. Liebling-Boccio DE, Jennings HR. The current status of graduate training in suicide risk assessment. Psychol Scho. (2013) 50:72–86. doi: 10.1002/pits.21661

PubMed Abstract | CrossRef Full Text | Google Scholar

19. Shea SC. Psychiatric Interviewing: The Art of Understanding : A Practical Guide for Psychiatrists, Psychologists, Counselors, Social Workers, Nurses, and Other Mental Health Professionals, with Online Video Modules. 3rd ed. Elsevier (2017). p. 876.

Google Scholar

20. Stanley B, Brown GK. Safety planning intervention: A brief intervention to mitigate suicide risk. Cogn Behav Pract. (2012) 19:256–64. doi: 10.1016/j.cbpra.2011.01.001

PubMed Abstract | CrossRef Full Text | Google Scholar

21. Bandura A. Social cognitive theory: an agentic perspective. Ann Rev Psychol. (2001) 52:1–26. doi: 10.1146/annurev.psych.52.1.1

PubMed Abstract | CrossRef Full Text | Google Scholar

22. Burnette C, Ramchand R, Ayer L. Gatekeeper training for suicide prevention: a theoretical model and review of the empirical literature. Rand Health Quart. (2015) 5:16. doi: 10.7249/j.ctt13x1ftq

PubMed Abstract | CrossRef Full Text | Google Scholar

23. Hawgood J, Woodward A, Quinnett P, De Leo D. Gatekeeper training and minimum standards of competency: essentials for the suicide prevention workforce. Crisis. (2021) 1–7. doi: 10.1027/0227-5910/a000794

PubMed Abstract | CrossRef Full Text | Google Scholar

24. Hegerl U, Mergl R, Havers I, Schmidtke A, Lehfeld H, Niklewski G, et al. Sustainable effects on suicidality were found for the Nuremberg alliance against depression. Eur Arch Psychiatry Clin Neurosci. (2010) 260:401–6. doi: 10.1007/s00406-009-0088-z

PubMed Abstract | CrossRef Full Text | Google Scholar

25. Mishara BL, Houle J, Lavoie B. Comparison of the effects of four suicide prevention programs for family and friends of high-risk suicidal men who do not seek help themselves. Suicide Life Threat Behav. (2005) 35:329–42. doi: 10.1521/suli.2005.35.3.329

PubMed Abstract | CrossRef Full Text | Google Scholar

26. Wakai S, Schilling EA, Aseltine RH, Blair EW, Bourbeau J, Duarte A, et al. Suicide prevention skills, confidence and training: results from the zero suicide workforce survey of behavioral health care professionals. SAGE Open Med. (2020) 8:205031212093315. doi: 10.1177/2050312120933152

PubMed Abstract | CrossRef Full Text | Google Scholar

27. Monahan MF, Crowley KJ, Arnkoff DB, Glass CR, Jobes DA. Understanding therapists' work with suicidal patients: an examination of qualitative data. OMEGA J Death Dying. (2020) 81:330–46. doi: 10.1177/0030222818775888

PubMed Abstract | CrossRef Full Text | Google Scholar

28. LoParo D, Florez IA, Valentine N, Lamis DA. Associations of suicide prevention trainings with practices and confidence among clinicians at community mental health centers. Suicide Life Threat Behav. (2019) 49:1148–56. doi: 10.1111/sltb.12498

PubMed Abstract | CrossRef Full Text | Google Scholar

29. Silverman MM, Berman AL. Training for suicide risk assessment and suicide risk formulation. Acad Psychiatry. (2014) 38:526–37. doi: 10.1007/s40596-014-0200-1

PubMed Abstract | CrossRef Full Text | Google Scholar

30. Roush JF, Brown SL, Jahn DR, Mitchell SM, Taylor NJ, Quinnett P, et al. Mental health professionals' suicide risk assessment and management practices: the impact of fear of suicide-related outcomes and comfort working with suicidal individuals. Crisis. (2018) 39:55–64. doi: 10.1027/0227-5910/a000478

PubMed Abstract | CrossRef Full Text | Google Scholar

31. Rothes IA, Henriques MR, Leal JB, Lemos MS. Facing a patient who seeks help after a suicide attempt: the difficulties of health professionals. Crisis. (2014) 35:110–22. doi: 10.1027/0227-5910/a000242

PubMed Abstract | CrossRef Full Text | Google Scholar

32. Jahn DR, Quinnett P, Ries R. The influence of training and experience on mental health practitioners' comfort working with suicidal individuals. Profess Psychol Res Practice. (2016) 47:130–8. doi: 10.1037/pro0000070

CrossRef Full Text | Google Scholar

33. Masuku S. Self-harm presentations in emergency departments: staff attitudes and triage. Brit J Nurs. (2019) 28:1468–76. doi: 10.12968/bjon.2019.28.22.1468

PubMed Abstract | CrossRef Full Text | Google Scholar

34. Reshetukha TR, Alavi N, Prost E, Kirkpatrick RH, Sajid S, Patel C, et al. Improving suicide risk assessment in the emergency department through physician education and a suicide risk assessment prompt. Gen Hosp Psychiatry. (2018) 52:34–40. doi: 10.1016/j.genhosppsych.2018.03.001

PubMed Abstract | CrossRef Full Text | Google Scholar

35. La Guardia AC, Cramer RJ, Brubaker M, Long MM. Community mental health provider responses to a competency-based training in suicide risk assessment and prevention. Comm Mental Health J. (2019) 55:257–66. doi: 10.1007/s10597-018-0314-0

PubMed Abstract | CrossRef Full Text | Google Scholar

36. O'Connor S, Glover L. Hospital staff experiences of their relationships with adults who self-harm: a meta-synthesis. Psychol Psychother Theory Res Practice. (2017) 90:480–501. doi: 10.1111/papt.12113

PubMed Abstract | CrossRef Full Text | Google Scholar

37. Osteen P, Frey JM, Woods MN, Ko J, Shipe S. Modeling the longitudinal direct and indirect effects of attitudes, self-efficacy, and behavioral intentions on practice behavior outcomes of suicide intervention training. Suicide Life-Threat Behav. (2017) 47:410–20. doi: 10.1111/sltb.12288

PubMed Abstract | CrossRef Full Text | Google Scholar

38. Rayner G, Blackburn J, Edward K, Stephenson J, Ousey K. Emergency department nurse's attitudes towards patients who self-harm: A meta-analysis. Int J Ment Health Nurs. (2019) 28:40–53. doi: 10.1111/inm.12550

PubMed Abstract | CrossRef Full Text | Google Scholar

39. Osafo J, Akotia CS, Boakye KE, Dickson E. Between moral infraction and existential crisis: exploring physicians and nurses' attitudes to suicide and the suicidal patient in Ghana. Int J Nurs Studies. (2018) 85:118–25. doi: 10.1016/j.ijnurstu.2018.05.017

PubMed Abstract | CrossRef Full Text | Google Scholar

40. Patridge EF, Bardyn TP. Research electronic data capture (REDCap). J Med Library Assoc JMLA. (2018) 106:142–4. doi: 10.5195/jmla.2018.319

CrossRef Full Text | Google Scholar

41. Regehr C, Bogo M, LeBlanc VR, Baird S, Paterson J, Birze A. Suicide risk assessment: clinicians' confidence in their professional judgment. J Loss Trauma. (2016) 21:30–46. doi: 10.1080/15325024.2015.1072012

CrossRef Full Text | Google Scholar

42. Airey ND, Iqbal Z. Are clinicians confident in the risk assessment of suicide?: a systematic literature review. Arch Suicide Res. (2020) 1–13. doi: 10.1080/13811118.2020.1792019

PubMed Abstract | CrossRef Full Text | Google Scholar

43. Herron J, Ticehurst H, Appleby L, Perry A, Cordingley L. Attitudes toward suicide prevention in front-line health staff. Suicide Life Threat Behav. (2001) 31:342–7. doi: 10.1521/suli.31.3.342.24252

PubMed Abstract | CrossRef Full Text | Google Scholar

44. Wyman PA, Brown CH, Inman J, Cross W. Randomized trial of a gatekeeper program for suicide prevention: 1-year impact on secondary school staff. J Consult Clin Psychol. (2008) 76:104–15. doi: 10.1037/0022-006X.76.1.104

PubMed Abstract | CrossRef Full Text | Google Scholar

45. Hays RD, Hayashi T, Stewart AL. A 5-item measure of socially desirable response set. Educ Psychol Meas. (1989) 49:629–36. doi: 10.1177/001316448904900315

CrossRef Full Text | Google Scholar

46. McCambridge J, De Bruin M, Witton J. The effects of demand characteristics on research participant behaviours in non-laboratory settings: a systematic review. PLoS ONE. (2012) 7:e39116. doi: 10.1371/journal.pone.0039116

PubMed Abstract | CrossRef Full Text | Google Scholar

47. Detry M, Ma Y. Analyzing repeated measurements using mixed models. JAMA. (2016) 315:407. doi: 10.1001/jama.2015.19394

PubMed Abstract | CrossRef Full Text | Google Scholar

48. Tabachnick BG, Fidell LS. Using Multivariate Statistics 6th ed. Pearson Education Ltd (2013).

Google Scholar

49. Kraemer H, Blasey C. Centring in regression analyses: a strategy to prevent errors in statistical inference. Int J Meth Psychiatric Res. (2004) 13:141–51. doi: 10.1002/mpr.170

PubMed Abstract | CrossRef Full Text | Google Scholar

50. Seltman HJ,. Experimental Design Analysis. Department of Statistics at Carnegie Mellon. (2018). Available online at: https://www.stat.cmu.edu/~hseltman/309/Book/Book.pdf (accessed August 9, 2021).

Google Scholar

51. Peters SA, Bots ML, Den Ruijter HM, Palmer MK, Grobbee DE, Crouse JR, et al. Multiple imputation of missing repeated outcome measurements did not add to linear mixed-effects models. J Clin Epidemiol. (2012) 65:686–95. doi: 10.1016/j.jclinepi.2011.11.012

PubMed Abstract | CrossRef Full Text | Google Scholar

52. Labouliere CD, Green KL, Vasan P, Cummings A, Layman D, Kammer J, et al. Is the outpatient mental health workforce ready to save lives? Suicide prevention training, knowledge, self-efficacy, and clinical practices prior to the implementation of a statewide suicide prevention initiative. Suicide Life Threat Behav. (2021) 51:325–33. doi: 10.1111/sltb.12708

PubMed Abstract | CrossRef Full Text | Google Scholar

53. Silva C, Smith AR, Dodd DR, Covington DW, Joiner TE. Suicide-related knowledge and confidence among behavioral health care staff in seven States. Psychiatric Serv. (2016) 67:1240–5. doi: 10.1176/appi.ps.201500271

PubMed Abstract | CrossRef Full Text | Google Scholar

54. Lund EM, Schultz JC, Nadorff MR, Galbraith K, Thomas KB. Experience, knowledge, and perceived comfort and clinical competency in working with suicidal clients among vocational rehabilitation counselors. Rehabil Couns Bull. (2017) 61:54–63. doi: 10.1177/0034355217695776

CrossRef Full Text | Google Scholar

55. Saunders KE, Hawton K, Fortune S, Farrell S. Attitudes and knowledge of clinical staff regarding people who self-harm: a systematic review. J Affect Dis. (2012) 139:205–16. doi: 10.1016/j.jad.2011.08.024

PubMed Abstract | CrossRef Full Text | Google Scholar

56. Cramer RJ, Bryson CN, Stroud CH, Ridge BE. A pilot test of a graduate course in suicide theory, risk assessment, and management. Teach Psychol. (2016) 43:238–42. doi: 10.1177/0098628316649483

CrossRef Full Text | Google Scholar

57. Jacobson JM, Osteen P, Jones A, Berman A. Evaluation of the recognizing and responding to suicide risk training. Suicide Life Threat Behav. (2012) 42:471–85. doi: 10.1111/j.1943-278X.2012.00105.x

PubMed Abstract | CrossRef Full Text | Google Scholar

58. Tompkins TL, Witt J. The short-term effectiveness of a suicide prevention gatekeeper training program in a college setting with residence life advisers. J Prim Prev. (2009) 30:131–49. doi: 10.1007/s10935-009-0171-2

PubMed Abstract | CrossRef Full Text | Google Scholar

59. Ayer L, Ramchand R, Geyer L, Burgette L, Kofner A. The influence of training, reluctance, efficacy, and stigma on suicide intervention behavior among NCOs in the army and Marine Corps. J Prim Prev. (2016) 37:287–302. doi: 10.1007/s10935-016-0419-6

PubMed Abstract | CrossRef Full Text | Google Scholar

60. Rudd MD, Cukrowicz KC, Bryan CJ. Core competencies in suicide risk assessment and management: implications for supervision. Train Educ Profess Psychol. (2008) 2:219–28. doi: 10.1037/1931-3918.2.4.219

PubMed Abstract | CrossRef Full Text | Google Scholar

61. Koivu A, Saarinen PI, Hyrkas K. Who benefits from clinical supervision and how? The association between clinical supervision and the work-related well-being of female hospital nurses. J Clin Nurs. (2012) 21:2567–78. doi: 10.1111/j.1365-2702.2011.04041.x

PubMed Abstract | CrossRef Full Text | Google Scholar

62. Snowdon DA, Leggat SG, Taylor NF. Does clinical supervision of healthcare professionals improve effectiveness of care and patient experience? A systematic review. BMC Health Serv Res. (2017) 17:786. doi: 10.1186/s12913-017-2739-5

PubMed Abstract | CrossRef Full Text | Google Scholar

63. Jager J, Putnick DL, Bornstein MH II. More than just convenient: the scientific merits of homogeneous convenience samples. Monog Soc Res Child Dev. (2017) 82:13–30. doi: 10.1111/mono.12296

PubMed Abstract | CrossRef Full Text | Google Scholar

64. Russell DJ, Zhao Y, Guthridge S, Ramjan M, Jones MP, Humphreys JS, et al. Patterns of resident health workforce turnover and retention in remote communities of the northern territory of Australia, 2013-2015. Human Res Health. (2017) 15:52. doi: 10.1186/s12960-017-0229-9

PubMed Abstract | CrossRef Full Text | Google Scholar