Evaluating the before operational stress program: comparing in-person and virtual delivery

Introduction: Public safety personnel (PSP) are at increased risk for posttraumatic stress injuries (PTSI). Before Operational Stress (BOS) is a mental health program for PSP with preliminary support mitigating PTSI. The current study compared the effectiveness of delivering BOS in-person by a registered clinician (i.e., Intensive) to virtually delivery by a trained clinician (i.e., Classroom).

Methods: Canadian PSP completed the Intensive (n = 118; 61.9% male) or Classroom (n = 149; 50.3% male) program, with self-report surveys at pre-, post-, 1 month, and 4 months follow-ups.

Results: Multilevel modelling evidenced comparable reductions in anxiety (p < 0.05, ES = 0.21) and emotional regulation difficulties (ps < 0.05, ESs = 0.20, 0.25) over time with no significant difference between modalities. Participants discussed benefits of the delivery modality they received.

Discussion: The results support virtual delivery of the BOS program (Classroom) as an accessible mental health training option for PSP, producing effects comparable to in-person delivery by clinicians.

1 Introduction

Canadians rely on diverse public safety personnel (PSP; e.g., police, firefighters, paramedics) to ensure their safety and well-being (Mendicino and Blair, 2022). PSP are at increased risk for posttraumatic stress injuries (PTSI) (Carleton et al., 2020a; Heber et al., 2023) resulting from operational and organizational stressors (Carleton et al., 2020a). Almost half (44.5%) of PSP screen positive for one or more mental health disorders (Vig et al., 2020) and many report lifetime suicidal ideation (27.8%), planning (13.3%), and attempts (4.6%) (Carleton et al., 2018a). PTSI are associated with frequent exposures to potentially psychologically traumatic events (PPTEs; i.e., direct or indirect exposures to actual or threatened death, serious injury, or sexual violence) (Carleton et al., 2019a; Heber et al., 2023). PTSI are a substantial concern for PSP mental health and the Canadian government has recently developed an action plan to address challenges related to these mental health difficulties (Public Safety Canada, 2019).

There are several effective treatments for PTSI, such as cognitive behavioural therapy (CBT), that appear beneficial for PSP (Foa and Rothbaum, 2001; Ponniah and Hollon, 2009; Carleton, 2021; Hadjistavropoulos et al., 2021). Proactive solutions to mitigate PTSI also exist, often focusing on providing training or peer support to bolster resilience, minimize stigma, and develop individual stress management skills (Carleton et al., 2020b; Stelnicki et al., 2021). There has been relatively little research on such proactive efforts, with the available evidence suggesting small time-limited effects, necessitating recommendations for larger longitudinal research efforts (Anderson et al., 2020; Di Nota et al., 2022).

1.1 Overview of the before operational stress program

The Before Operational Stress (BOS) program was designed to provide access to effective evidence-based mental health training for PSP (McElheran et al., 2020; Stelnicki et al., 2021). The BOS programming is based on teaching core CBT skills and providing evidence informed learning content, taught by a clinician trained in CBT principles and familiar with PSP culture and treatment. BOS program combines theoretical and experiential learning procedures designed to improve resiliency, strengthen interpersonal relationships, and mitigate the effects of operational stressors. Participants complete one module per week for 8 weeks. The first six modules involve teaching participants to identify, understand, and navigate the connection between thoughts, emotions, physiological sensations, and behaviour. The final two modules focus on enhancing interpersonal relationships by teaching communication skills and empathy. Throughout the program, participants explore various mental health related topics to expand their knowledge of OSI and effective coping strategies. The program also emphasizes the importance of PSP maintaining healthy, authentic relationships, which can provide important mental health benefits for PSP as previous research suggests they are more likely to seek support from spouses and friends than professionals (Carleton et al., 2019b, 2020b; Nisbet et al., 2023).

The BOS Intensive program is facilitated in-person by a trained clinician in a group setting, following the one module per week for eight weeks model. Each 2 hour module-session is approximately evenly divided between a facilitator-led didactic component communicating program content, and 1 hour of group processing where participants share thoughts on the program content and discuss its application in their lives. A prior evaluation of the BOS Intensive program evidenced small but statistically significant improvements in PTSD symptoms and quality of life measures, increases in perceived social support, and reductions in mental health stigma associated with the training (Stelnicki et al., 2021). This evaluation was conducted with a smaller sample of participants who completed the training through the Original BOS Intensive (in-person) delivery modality and surveyed participants at four time points (before and after the training, and at 1 month and 3 month follow-ups). The available outcomes were associated with improved communication skills and more positive behaviours toward family members, as evidenced by qualitative responses.

The BOS Classroom program, consisting of 1 h sessions delivered virtually by a trained clinician, was developed to enhance training accessibility for individuals in areas lacking BOS-trained clinicians. The BOS Classroom modality removes the group processing aspect of the Intensive training to solely focus on delivery of the didactic component, therein shortening each module-session to approximately one hour. BOS Classroom became particularly important for maintaining training accessibility during public gathering restrictions imposed during the COVID-19 pandemic. The virtual training also helped address logistical access barriers by providing flexible timing and avoiding stigma barriers still prevalent in PSP workplaces (Rice et al., 2019; Hadjistavropoulos et al., 2021).

To date there has been no evaluation of the effectiveness of the BOS Classroom program and, by extension, no comparative assessment with the BOS Intensive program. Extant research indicates internet-based Cognitive Behavioural Therapy (ICBT) can be as effective as face-to-face CBT in treating a range of mental health challenges, offering similar therapeutic benefits, but with the added convenience of accessibility and flexibility (Andersson et al., 2019; Hadjistavropoulos et al., 2021; Thew et al., 2022). Previous evaluations of mindfulness training with military personnel demonstrated in-person training as producing better improvements in sleepiness, pain, and energy, than virtual training (Rice and Schroeder, 2021). The researchers suggested the in-person training benefits were supported by having longer sessions, stronger interpersonal bonds between the instructor and classmates (Rice and Schroeder, 2021), face-to-face interactions, greater accountability, and in-class participation (Rice et al., 2019).

1.2 The current study

The current study was designed to evaluate the effectiveness of the BOS Classroom program for improving mental health. The study hypotheses were: (1) participation in the BOS program would be associated with reductions in mental health symptoms, substance use, and mental health stigma, as well as increases in perceived social support, emotional regulation, resilience, and quality of life; and (2) the outcomes of the BOS Classroom program would be largely comparable to the BOS Intensive program.

We anticipate that the effect sizes in the BOS Classroom will be smaller than those in the BOS Intensive, although the direction of the effects is expected to be consistent across both modalities. This difference in effect sizes is expected due to the distinct characteristics of the Classroom modality, such as reduced group processing and shorter session durations. Although previous research suggests in-personal and virtual CBT training and treatment programs can have similar effectiveness (Spek et al., 2007; Andersson et al., 2016; Carlbring et al., 2018; Rice et al., 2018; Zhang et al., 2022; Alavi et al., 2023), BOS Classroom also features a reduced session time and omits the group processing component, which may affect the overall effectiveness of the training. The current study builds upon the findings of the previous independent evaluation of the BOS program (Stelnicki et al., 2021). Our study extends this work by comparing delivery modalities and employing a comprehensive mixed-methods approach, with a specific focus on contrasting the effectiveness of in-person and virtual delivery modalities.

2 Method

2.1 Procedure

The current study was approved by the University of Regina Institutional Research Ethics board (2018-191). Data were collected for two training modalities: BOS Intensive (i.e., in-person with group processing components) and BOS Classroom (i.e., virtual, didactic components only). The quasi-experimental design resulted from convenience samples arising from BOS Intensive groups being scheduled in-person until implementation of COVID-19 pandemic gathering restrictions, with BOS Classroom groups being scheduled after onset of the gathering restrictions. All participants completed a standardized intake interview prior to starting the BOS program, during which they were informed about potential voluntary participation in an independent research study evaluating the BOS training. Participants were screened prior to training for acute mental health distress or severe PTSD symptoms, and those in need were referred to therapy intervention resources. Participants who expressed interest were emailed a consent form along with a link to the first survey. Due to confidentiality assured during BOS training, data was only recorded from participants who expressed interest in the study, and the total number of participants in the BOS training at the time outside of study participation is not available. The research participation was voluntary, anonymous, and did not impact eligibility for the BOS program. Neither the clinical facilitators nor other group members were aware which members chose to participate in the study. No questions were mandatory, and participants could withdraw at any time without consequences for their training. Surveys were administered at four time points: pre-training, post-training, 1 month follow-up, and 4 months follow-up. All surveys contained the same measures, with pre-training also including sociodemographic measures, and post-training having an additional five open-ended questions requesting feedback on participant experiences in the training program. Although training recruitment was aimed at early career recruits, training was open to all career stages. As PSP will likely experience multiple exposure to PPTEs in their careers, BOS is intended as a proactive measure for any future exposures a PSP may experience.

2.2 Participants

A total of 267 participants were included in the study, of 118 which received the BOS Intensive (in-person) modality and 149 received the BOS Classroom (virtual) modality. Participants generally identified as male (61.9% Intensive, 50.3% Classroom), between 40 and 49 years old (39.0% Intensive, 32.9% Classroom), and married (71.2% Intensive, 72.5% Classroom). Program participants primarily resided in Western (56.8% Intensive; 46.3% Classroom) and Eastern Canada (33.9% Intensive; 43.6% Classroom). Most participants had graduated from a college program (35.6% Intensive, 19.5% Classroom) or completed a university degree (11.0% Intensive, 36.2% Classroom). Participants were typically firefighters (39.8% Intensive, 18.1% Classroom), paramedics (18.6% Intensive, 10.7% Classroom), or police (13.6% Intensive, 19.5% Classroom). Of the participants who answered pre- and post-training surveys, an approximate 15% returned for the four-month follow-up.

2.3 Measures

Study measures were chosen based on module content of the BOS program and to facilitate comparison with the BOS pilot study evaluation (Stelnicki et al., 2021).

2.3.1 Alcohol use disorders identification test

The Alcohol Use Disorders Identification Test [AUDIT; Babor et al., 2001] is a 10-item self-report measure of potentially hazardous alcohol use. Participants are asked to rate each item on a 5-point Likert scale ranging from 0 (never) to 4 (daily or almost daily). Higher scores indicate more potentially hazardous alcohol use. The AUDIT is widely employed and has evidence of adequate psychometric properties (Reinert and Allen, 2002; de Meneses-Gaya et al., 2009; Peng et al., 2012). AUDIT scores can be used to screen for hazardous alcohol use (>7) and alcohol dependence (>15).

2.3.2 Brief resiliency scale

The Brief Resiliency Scale (BRS; Smith et al., 2008) is a 6-item self-report measure of resilience. Participants rate each item on a 5-point Likert scale ranging from 1 (strongly disagree) to 5 (strongly agree). Items are summed to produce a total score, with higher scores indicating higher resiliency. The BRS has evidence of adequate internal consistency and test–retest reliability (Windle et al., 2011).

2.3.3 Depression, anxiety, and stress scale

The Depression, Anxiety, and Stress Scale [DASS-21; Lovibond and Lovibond, 1995] is a 21-item self-report measure of designed to measure the negative emotional states of depression, anxiety, and stress. Each subscale consists of 7 items scored on a 5-point Likert scale ranging from 0 (did not apply to me at all) to 4 (applied to me very much or most of the time). Higher scores indicate greater symptom levels. The DASS-21 has evidence of adequate internal consistency, construct validity, and convergent and discriminant validity (Henry and Crawford, 2005). The DASS-21 subscale scores can be used to screen for clinically-significant depression (>20), anxiety (>14), and stress (>25).

2.3.4 Difficulties in emotional regulation scale

The Difficulties in Emotional Regulation Scale (DERS; Gratz and Roemer, 2004) is a 36-item self-report measure of difficulties with emotional regulation. Participants rate each item on a 5-point Likert scale ranging from 1 (almost never) to 5 (almost always). All items can be summed to produce a total score, with higher scores indicating a greater degree of emotional dysregulation. Item subsets can be summed to produce subscales related to emotional responses (i.e., nonacceptance of emotional responses, lack of emotional awareness, limited access to emotional regulation strategies, and limited emotional clarity). The DERS has evidence of adequate reliability, as well as construct and concurrent validity (Gratz and Roemer, 2004; Bardeen et al., 2012; Fowler et al., 2014; Hallion et al., 2018), and may help predict responses to cognitive-behavioural therapy (Hallion et al., 2018).

2.3.5 Opening minds survey for workplace attitudes

The Opening Minds Survey for Workplace Attitudes (OMSWA; Szeto et al., 2013) is an 11-item self-report measure of attitudes towards people with mental illness. Participants rate each item on a 5-point Likert scale ranging from 1 (strongly disagree) to 5 (strongly agree). Items are summed to produce a total score with higher scores indicating a higher degree of stigma in the workplace. The OMSWA has been widely employed by the Mental Health Commission of Canada to measure stigma (Krakauer et al., 2020) and with evidence of adequate factor validity for the 9-item version (Boehme et al., 2022).

2.3.6 PTSD checklist for DSM-5

The PTSD Checklist for DSM-5 (PCL-5; Weathers et al., 2013) PCL-5 is a 20-item self-report measure used to assess PTSD symptoms in the past month. Participants who identify exposure to at least one PPTE are then asked to select which exposure has caused them the most difficulty recently and answer questions regarding how much they have been bothered by different aspects of that event over the past month. Participants rate each item on a 5-point Likert scale ranging from 0 (not at all) to 4 (extremely). All item scores can be summed to produce a total score, and subsets of items can be summed to produce subscale scores reflecting DSM-5 symptom clusters B (re-experiencing/re-living), C (avoiding reminders of the incident), D (negative thoughts and mood), and E (hyper arousal/alertness). Higher scores indicate greater symptom levels. The PCL-5 has evidence of adequate reliability, as well as structural, convergent, and discriminant validity (Blevins et al., 2015; Ashbaugh et al., 2016). A positive screen for probable PTSD can be made for participants who endorse symptoms in each PTSD cluster and exceed the minimum clinical cutoff of 32 for the total PCL-5 score.

2.3.7 Social provisions scale

The Social Provisions Scale (SPS10; Cutrona and Russell, 1987) is a 10-item self-report measure of extent of social support. Participants rate each item on a 4-point Likert-type scale ranging from 1 (strongly disagree) to 4 (strongly agree). All items can be summed to produce a total score, and the SPS10 has evidence of adequate reliability and convergent validity (Gottlieb and Bergen, 2010).

2.3.8 WHO quality of life-BREF (WHOQOL-BREF)

The WHO Quality of Life-BREF (WHOQOL-BREF; WHOQOL Group, 1998) is a 25-item self-report measure of quality of life. Participants rate each item on a 5-point Likert scale ranging from 1 (not at all) to 5 (an extreme amount). The mean of all items is calculated to produce a total score, with item subset means calculated to produce subscale scores describing each of four domains: physical health, psychological health, social relationships, and environmental quality of life. We analyzed domains 2, 3, and 4, which relate to psychological and social quality of life. Higher mean scores indicate greater quality of life. The scale has evidence of adequate psychometric properties (Skevington et al., 2004).

2.4 Qualitative data and analyses

Participants were asked five open–ended questions at the end of the post-training survey (i.e., upon BOS program completion): (1) What has been the most helpful aspect of BOS for you?; (2) What has been the least helpful aspect of BOS for you?; (3) Has anything gotten better for you as a result of BOS? If so, please describe; (4) Has anything gotten worse for you as a result of BOS? If so, please describe; and (5) Please use the space below to provide any other comments you would like about your participation in BOS. Of the initial participant responses, 9 (n = 9) were excluded due to missing data, resulting in 30 Intensive and 45 Classroom responders, totaling 75 (n = 75) participants. Responses were anonymized and analyzed in NVivo 12. Two authors (MR and AW) used the coding procedures outlined in Miles et al. (2018) to separately inductively analyze the data. The same authors then collaborated to create broad themes by comparing their two code lists, producing a master coding framework, and recoding the data. A comparison query was run to determine the degree of overlap between coders (calculated at 80% agreement with a Cohen’s Kappa of 0.40, suggesting a fair level of agreement beyond what might be expected from chance) and to reach consensus on codes of disagreement (i.e., what theme best captured codes of discrepancy between authors). The authors then re-visited the coded data and addressed discrepancies in themes to achieve 100% agreement and co-constructed definitions to closely represent participant experiences by discussing the coded themes and the associated interrelationships. A matrix coding query was used to analyze the differences in theme references between Intensive and Classroom modalities and to clarify final participant counts for each theme.

2.5 Quantitative analyses

Descriptive analyses provided information about the sociodemographic characteristics of participants in the Intensive and Classroom modalities (Table 1). Descriptive analyses characterized mental health and resilience scale variables over time (Table 2). Generalized Estimating Equations were used to assess whether the number of participants screening positive for a particular mental health challenge changed statistically significantly over time (Table 3) (Andreski et al., 1998; Heine et al., 2011). Modality was included as a factor to assess whether the number of participants with a positive screen differed between Intensive and Classroom modalities. Interaction effects were initially tested, but none were statistically significant (all ps > 0.05), so none were retained for the final model.

Table 1

Table 1. Public safety personnel demographics by delivery modality.

Table 2

Table 2. Self-report mental health measure metrics by time.

Table 3

Table 3. Valid generalized estimating equations for psychological disorder criteria screeners by modality and timepoint, using intensive modality at T1 as baseline.

A multi-level modelling (MLM) approach was employed to assess for changes in mean self-report scores over time, and to assess whether any changes differed across delivery modalities (i.e., Intensive or Classroom, Table 4). The MLM approach provides accurate estimates when some individuals have missing timepoints (Heck et al., 2013; West et al., 2015), and also allowed for more close comparison to the previous BOS Intensive evaluation. Model fits were estimated using restricted maximum likelihood (REML) via the MIXED command in SPSS Version 28. The MLM strategy for each scale was as follows: analyses began with a two-level model consisting of a fixed effect for Time, a random intercept and slope for Time at the individual level, and the first order autocorrelation structure for the within individual covariance matrix at the repeated measures level. The random slope for Time was removed from the model due to non-convergence, which is consistent with previous BOS program evaluations using the same model criteria (Stelnicki et al., 2021). Modality was then included as a moderator variable by adding fixed effects for Modality and Time by Modality interaction. The Time by Modality interaction was not statistically significant for all scales and was subsequently removed from the fixed effects. Removing the Time by Modality interaction did not decrease model fit per a likelihood ratio test to assess the goodness of fit (−2 log-likelihood) between the reduced model without the interaction and the model that contained the interaction, as estimated with maximum likelihood. The final model reported in Table 4 contains fixed effects for Time and Modality, a random intercept at the individual level, and the autocorrelation structure at the repeated measures level estimated with REML.

Table 4

Table 4. Two-level MLM results for overall scale totals.

Random effects (i.e., between person variance in baseline level time τ00, and within individual variance σ²) were statistically significant for all models (ps < 0.05) and were omitted from the final table for brevity. The random effects estimates were used to calculate intraclass correlation coefficients included for interpretation. Standardized effect sizes (ES) and intraclass correlation coefficients (ICC) for the MLM analysis were calculated manually using estimates of fixed and random effects. Standardized effect sizes were calculated for each fixed effect using a pooled variance estimate from the final model for each scale. Using pooled variance to standardize effect size estimates accounts for the within and between individual variance in repeated measures designs (Pustejovsky et al., 2014; Westfall et al., 2014). The standardized effect sizes can be interpreted like Cohen’s d (Pustejovsky et al., 2014), in which 0.20 represents a small effect, 0.50 represents a medium effect, and 0.80 represents a large effect (Pleil et al., 2018). ICCs were calculated for each applicable scale or subscale using the between individual (random intercept) variance estimate and the residual variance estimate from the empty random intercept model. The empty random intercept model consisted of a random intercept at the individual level, and no fixed effects, estimated with REML. In the absence of model predictors, the ICC provides an estimate of the proportion of variance attributable to individual differences (Raudenbush and Bryk, 2002; Hox and De Leeuw, 2003).

3 Results

Sociodemographic characteristics of participants in both modalities were largely comparable (Table 1). The only statistically significant effect of modality for positive screens indicated Classroom modality participants at pre-training had a lower prevalence of positive screens for potentially hazardous alcohol use than Intensive modality participants (Table 3). The differences in prevalence of positive screens for mental health disorders between the modalities were largely attributable to individual differences between participants (Table 3).

There were no statistically significant differences between modalities for other measures, except that OMSWA total scores were statistically significantly higher (p < 0.05) among Intensive participants (Table 4). Average change estimates (Table 4) indicated PCL-5 Cluster E (hyper arousal/alertness) scores increased statistically significantly (p < 0.05) for all participants from pre- to post-training. Anxiety and environmental quality of life subscale scores decreased (p < 0.05) for all participants from pre-training to 1-month follow-up. Emotional regulation total scores decreased from pre-training to 1 month follow-up (p < 0.05), and pre-training to 4-month follow-up (p < 0.05) (Table 4). Emotional regulation subscale scores decreased (p < 0.05) for “Non Acceptance of Emotional Responses” and “Emotional Awareness” from pre-training to 4 month follow-up. “Emotional Clarity” subscale scores evidenced a statistically significant increase from pre- to post-training, and a statistically significant decrease from pre-training to 1 month follow-up. Decreases in resilience from pre- to post-training, and from pre-training to 4 month follow-up, and increases in resilience from pre-training to 1 month follow-up were not statistically significant.

All model random effects were statistically significant (ps < 0.05), indicating a large proportion of the variance observed in outcome measures across time was attributable to initial differences between individuals, as well as differences within each individual. The results were supported by medium to high ICC values associated with all outcome measures.

3.1 Qualitative analyses results

The most heavily emphasized theme across participant responses was insight and awareness changes, with many participants describing thinking about themselves and their experiences in new ways (cognitive); behaving in different ways or acting upon new awareness (behavioural); and understanding their feelings and emotional responses, including being more attuned to bodily sensations (emotional). Participants in both modalities described increased awareness or insight within cognitive (Intensive: n = 11; Classroom: n = 24), emotional (Intensive: n = 13; Classroom: n = 6), and behavioural domains (Intensive: n = 9; Classroom: n = 5). Intensive participants more frequently described increased emotional and behavioural awareness, whereas Classroom participants more frequently described increased cognitive awareness. For example, one Intensive participant described increased attention to and recognition of their emotional experiences: “I have been able to recognize certain emotions as they are occurring and work through them, opposed to before when I would just become angry because I did not understand the emotion” (Participant 17). Intensive (n = 5) and Classroom (n = 2) participants also described changes in their familial relationships: “I’ve started communicating better with my wife and being able to relate and explain some of her feelings and thoughts to what I’ve learned in the program as well” (Intensive Participant 15).

The stigma reduction theme included participant descriptions of feeling less different from others or feeling less alone with their mental health symptoms, either via the explicit learning content (i.e., learning about brain-based or physiological responses to PPTE or other stressful events) or via group discussion of said content (i.e., relating to others’ experiences, experiencing normalization, and validation of responses to trauma or stress by sharing and hearing others’ stories). Intensive (n = 14) and Classroom (n = 4) modality participants described reductions in stigma via the group dynamic: “I have found a small cadre of others in the same boat as me, people I can relate with who had also stepped forward to say ‘I’m broken’” (Intensive Participant 4). Classroom (n = 7) and Intensive (n = 2) modality participants described the learning content contributed to reductions in feelings of isolation: “The program has been very helpful, but the one thing that really was … [that] PTSD is not a mental illness but a brain injury.” (Classroom Participant 39). This variation indicates that while both modalities were perceived as beneficial, consistent with the program content emphasized in each modality, Intensive participants more frequently cited group discussions and Classroom participants discussed the learning content.

The delivery theme included participant descriptions of program accessibility, quality, and delivery of the program. Classroom modality participants (n = 17) expressed negativity towards the delivery method, indicating a preference for Intensive delivery. There were also Classroom modality participants (n = 18) who reported feeling they had missed out on the benefits of group sessions and described online participation as difficult: “… very hard to express your feelings and receive feedback… The program would be far more beneficial if it was in a group setting, face to face with others” (Classroom participant 52). Intensive modality participants (n = 8) more often described participation challenges such as scheduling of the sessions and time required to attend compared to Classroom participants (n = 2).

4 Discussion

The current study was designed to evaluate the effectiveness of the virtual modality of the BOS program (BOS Classroom) as compared to the existing in-person delivery (BOS Intensive). The current results support the Classroom modality as generally comparable to the Intensive modality in terms of changes in self-reported mental health symptom outcomes. Across both modalities, the BOS program was associated with small, but statistically significant, improvements in self-reported anxiety and emotional regulation. The current results are consistent with a previous evaluation of BOS Intensive (Stelnicki et al., 2021), but incrementally evidence some of the small, statistically significant improvements in anxiety and emotional regulation, some of which were sustained 4 months after completing the training.

Consistent with our hypotheses, outcomes following the BOS program were largely comparable across both Classroom and Intensive modalities, with no differences in effects observed between modalities. The only modality effect was for the OMSWA measure, which differed statistically significantly between groups at initial scores, indicating changes in outcome measures did not meaningfully differ between delivery types. The proportion of participants who screened positive for mental health disorders were also largely comparable across modalities, except for a higher prevalence of potentially hazardous alcohol use positive screens among Intensive participants at pre-training. There were no statistically significant decreases in the number of participants who screened positive for mental health disorders over time after controlling for individual differences, suggesting any changes reflected dynamic individual circumstances, differences in relative engagement with the BOS material, or other unmeasured variables. The results were consistent with previous research evidencing Mindfulness-based Stress Reduction training as comparable across virtual and in-person formats among US military personnel and Veterans (Rice et al., 2018, 2019).

There were small, statistically significant improvements in anxiety symptoms and difficulties with emotional regulation across both modalities, which were sustained 4 months after the training. Open-text responses corroborate and contextualize the quantitative results, with participants from both modalities reporting new insights into their own thoughts and emotions concerning stressful situations. There were no statistically significant reductions in mental health stigma and feelings of isolation for participants in either modality, but participants who provided qualitative feedback described stigma reductions as important elements of the BOS program experience. Classroom modality participants primarily attributed reductions in self-stigma to the didactic component of BOS, whereas Intensive modality participants attributed the same reductions to group sharing. The feedback is consistent with prior research regarding the perceived potential impact of psychoeducation (Ricciardelli et al., 2020) and social validation (Kosmicki and Glickauf-Hughes, 1997; Cox et al., 2017; Yalom and Leszcz, 2020) for reducing stigmatizing beliefs. Reports of reduced stigma may be particularly important for PSP populations, as previous research has identified stigma towards mental health as a primary barrier to help-seeking behaviors in PSP (Newell et al., 2022).

Open-text responses also indicated that Intensive participants valued the group discussions, whereas Classroom modality participants described technical difficulties with online discussions sufficient to describe the discussions as detrimental to the overall training. The results contrast previous evidence suggesting preferences for virtual discussions perceived as less anxiety-provoking and more accessible (Rochlen et al., 2004; Fortier et al., 2022). Despite also facing some accessibility barriers while attending the training, participants in the BOS Intensive modality collectively expressed a preference for in-person groups. The individual variability in results and preferences, juxtaposed with the comparable results across modalities, suggests PSP may be best served by self-selecting modality training options that meet their current needs and preferences.

There was evidence of statistically significant increases in hyperarousal/alertness and decreases in environmental quality of life from pre- to post-training, possibly due to increased PPTE exposures concurrent with COVID-19 pandemic onset (Heber et al., 2020). Participants reported that the BOS program increased their mental health self-awareness, which may have facilitated increased attention to, and reporting of, the progressive pandemic impacts. In general populations, 40 to 70% of those who met screening criteria for PTSD symptoms no longer screened positive after CBT treatments or interventions (Bradley et al., 2005). In military and Veteran populations, many participants still report residual symptoms after CBT and prolonged exposure therapy (Bradley et al., 2005; Steenkamp et al., 2015; Allan et al., 2017), which may indicate these particular symptoms are resistant to treatment or are more successfully addressed over longer periods of time. Hyperarousal symptoms in particular were reported to be treatment resistant to typical CBT in military and Veteran populations (Crawford et al., 2019; Schnurr and Lunney, 2019; Miles et al., 2023). Promising alternative solutions have been explored in Veteran populations, such as mediation-based intervention (Crawford et al., 2019) to address this category of symptoms. Consistent with previous research results for BOS (Stelnicki et al., 2021) there were no statistically significant changes in self-reported resilience among participants in either modality.

Existing time-limited training interventions for PSP have generally demonstrated limited effectiveness for improving mental health outcomes (Carleton et al., 2018b, 2019a; Anderson et al., 2020). The BOS program presents a promising option for PSP mental health training as a function of small, but statistically significant, improvements in measures of anxiety, emotional regulation, and mental health stigma, which appear comparable across in-person and virtual modalities. Participants in the BOS Intensive program reported challenges with accessibility, but still reported a preference for in-person engagement. In addition, the virtual BOS training may be a viable alternative for PSP who find in-person sessions less accessible or convenient. Allowing PSP to select the modality that best aligns with their current circumstances and learning preferences could lead to increased engagement and, potentially, more effective training outcomes.

4.1 Limitations and future directions

The current study has several limitations that can help inform future research directions. First, the Classroom modality was introduced in response to growing accessibility needs and COVID-19 pandemic safety measures, which meant that the current data were based on a quasi-experimental design without randomized assignment to each modality. Second, there is no way to differentiate the relative impact of the COVID-19 pandemic from the impact of the BOS program using data from the current sample (Yu et al., 2020; Combden et al., 2022; Bouza et al., 2023) with PSP and healthcare workers appearing to have been disproportionately impacted by the COVID-19 pandemic (Heber et al., 2020; Marchildon et al., 2020; Hossain and Clatty, 2021; Cadell et al., 2022; Xue et al., 2022; Patel et al., 2023). The comparable effects found in the two modalities despite this time delay suggest the effects associated with BOS may be sustained even in high-stress social environments. Third, there was considerable intrapersonal variability in many outcome measures, reducing the statistical power to detect additional small effects at 1 month and 4 month follow-up assessments. Future researchers should consider using explicit random assignment or participant self-selection and including measurements of skill acquisition and use as part of evaluating the relative impact of the BOS program. Fourth, there was substantial attrition during the study such that the sample size at the 4 month follow-up was modest and the associated results may reflect important self-selection biases. Fifth, the current qualitative evidence suggests important elements of participant experiences may not be entirely captured in quantitative surveys, highlighting the importance of nuanced assessment of participant perceptions of the program in program evaluations.

5 Conclusion

The current study evaluated BOS program delivery and expanded previous research on the BOS program by including a longer follow-up period, comparing the Intensive (i.e., in-person) and Classroom (i.e., virtual) delivery modalities, and including qualitative analyses of participant experiences. The current results were consistent with previous research and evidenced small, but statistically significant, improvements in anxiety and emotional regulation, some of which were sustained 4 months after training. Changes in mental health symptoms were largely comparable across Intensive and Classroom modalities; however, many participants reported a preference for the Intensive program despite acknowledging accessibility benefits of the Classroom modality. Participants reported perceiving stigma reductions as part of qualitative data collection that were not reflected in quantitative self-report analyses. The comparable results across modalities suggests PSP may be best served by self-selecting modality training options that meet their current needs and preferences.

Data availability statement

The datasets presented in this article are not readily available because the dataset will be made available only to persons with the necessary clearances as defined by the research ethics board at the University of Regina. Requests to access the datasets should be directed to RNC bmljay5jYXJsZXRvbkB1cmVnaW5hLmNh.

Ethics statement

The studies involving humans were approved by University of Regina Research Ethics Board. The studies were conducted in accordance with the local legislation and institutional requirements. The participants provided their written informed consent to participate in this study.

Author contributions

GI: Data curation, Formal analysis, Investigation, Methodology, Supervision, Writing – original draft, Writing – review & editing. NB: Data curation, Formal analysis, Investigation, Methodology, Writing – original draft, Writing – review & editing, Validation. MR: Data curation, Formal analysis, Investigation, Methodology, Writing – original draft, Writing – review & editing. AW: Formal analysis, Investigation, Writing – original draft, Writing – review & editing. AS: Formal analysis, Writing – original draft, Writing – review & editing. JD: Formal analysis, Investigation, Writing – review & editing. JK: Formal analysis, Investigation, Writing – original draft, Writing – review & editing. KB: Formal analysis, Investigation, Writing – review & editing. SW: Investigation, Writing – original draft. TC: Investigation, Writing – original draft, Writing – review & editing. TT: Data curation, Formal analysis, Investigation, Methodology, Writing – review & editing. JP: Writing – review & editing. KM: Conceptualization, Investigation, Writing – review & editing. RNC: Conceptualization, Investigation, Supervision, Writing – review & editing.

Funding

The author(s) declare financial support was received for the research, authorship, and/or publication of this article. This study has received funding from Wounded Warriors Canada to support program evaluation research, and from the Public Health Agency of Canada to support analysis and publication.

Acknowledgments

We appreciated the Wayfound Mental Health Group efforts for their contributions to the current study; specifically, delivering the Before Operational Stress program for public safety personnel, which was fundamental to our research. We also appreciate the Wayfound Mental Health Group efforts with respect to logistic supports for data collection.

Conflict of interest

MR is currently employed by Wayfound Mental Health Group but had not had any contact or affiliation with the company at the time of the data collection, analysis, and manuscript preparation. TC was affiliated with Wayfound Mental Health Group at the time of the manuscript preparation but did not have access to any participant data, formal analyses, or data interpretation, and only assisted with writing consultation.

The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

Alavi, N., Moghimi, E., Stephenson, C., Gutierrez, G., Jagayat, J., Kumar, A., et al. (2023). Comparison of online and in-person cognitive behavioral therapy in individuals diagnosed with major depressive disorder: a non-randomized controlled trial. Front. Psych. 14:1113956. doi: 10.3389/fpsyt.2023.1113956

PubMed Abstract | Crossref Full Text | Google Scholar

Allan, N. P., Gros, D. F., Myers, U. S., Korte, K. J., and Acierno, R. (2017). Predictors and outcomes of growth mixture modeled trajectories across an exposure-based PTSD intervention with veterans. J. Clin. Psychol. 73, 1048–1063. doi: 10.1002/jclp.22408

PubMed Abstract | Crossref Full Text | Google Scholar

Anderson, G. S., Di Nota, P. M., Groll, D., and Carleton, R. N. (2020). Peer support and crisis-focused psychological interventions designed to mitigate post-traumatic stress injuries among public safety and frontline healthcare personnel: a systematic review. Int. J. Environ. Res. Public Health 17:7645. doi: 10.3390/ijerph17207645

PubMed Abstract | Crossref Full Text | Google Scholar

Andersson, G., Titov, N., Dear, B. F., Rozental, A., and Carlbring, P. (2019). Internet-delivered psychological treatments: from innovation to implementation. World Psychiatry 18, 20–28. doi: 10.1002/wps.20610

PubMed Abstract | Crossref Full Text | Google Scholar

Andersson, G., Topooco, N., Havik, O., and Nordgreen, T. (2016). Internet-supported versus face-to-face cognitive behavior therapy for depression. Expert. Rev. Neurother. 16, 55–60. doi: 10.1586/14737175.2015.1125783

PubMed Abstract | Crossref Full Text | Google Scholar

Andreski, P., Chilcoat, H., and Breslau, N. (1998). Post-traumatic stress disorder and somatization symptoms: a prospective study. Psychiatry Res. 79, 131–138. doi: 10.1016/S0165-1781(98)00026-2

PubMed Abstract | Crossref Full Text | Google Scholar

Ashbaugh, A. R., Houle-Johnson, S., Herbert, C., El-Hage, W., and Brunet, A. (2016). Psychometric validation of the English and French versions of the posttraumatic stress disorder checklist for DSM-5 (PCL-5). PLoS One 11:e0161645. doi: 10.1371/journal.pone.0161645

PubMed Abstract | Crossref Full Text | Google Scholar

Babor, T.F., Higgins-Biddle, J.C., Saunders, J.B., and Monteiro, M.G. (2001). The alcohol use disorders identification test

Google Scholar

Bardeen, J. R., Fergus, T. A., and Orcutt, H. K. (2012). An examination of the latent structure of the difficulties in emotion regulation scale. J. Psychopathol. Behav. Assess. 34, 382–392. doi: 10.1007/s10862-012-9280-y

Crossref Full Text | Google Scholar

Blevins, C. A., Weathers, F. W., Davis, M. T., Witte, T. K., and Domino, J. L. (2015). The posttraumatic stress disorder checklist for DSM-5 (PCL-5): development and initial psychometric evaluation. J. Trauma. Stress. 28, 489–498. doi: 10.1002/jts.22059

PubMed Abstract | Crossref Full Text | Google Scholar

Boehme, B. A. E., Shields, R. E., Asmundson, G. J. G., Szeto, A. C. H., Dobson, K. S., and Carleton, R. N. (2022). A short version of the opening minds scale–workplace attitudes: factor structure and factorial validity in a sample of Canadian public safety personnel. Can. J. Behav. Sci. 37:e350. doi: 10.1037/cbs0000350

Crossref Full Text | Google Scholar

Bouza, E., Arango, C., Moreno, C., Gracia, D., Martin, M., Perez, V., et al. (2023). Impact of the COVID-19 pandemic on the mental health of the general population and health care workers. Rev. Esp. Quimioter. 36, 125–143. doi: 10.37201/req/018.2023

PubMed Abstract | Crossref Full Text | Google Scholar

Bradley, R., Greene, J., Russ, E., Dutra, L., and Westen, D. (2005). A multidimensional meta-analysis of psychotherapy for PTSD. Am. J. Psychiatry 162, 214–227. doi: 10.1176/appi.ajp.162.2.214

PubMed Abstract | Crossref Full Text | Google Scholar

Cadell, S., Ashcroft, R., Furtado, J., Adamson, K., McConnell, S. M., and Teichman, S. (2022). COVID-19 and social work in health care in Canada: what are the impacts? Soc. Work Health Care 61, 218–242. doi: 10.1080/00981389.2022.2104985

Crossref Full Text | Google Scholar

Carlbring, P., Andersson, G., Cuijpers, P., Riper, H., and Hedman-Lagerlof, E. (2018). Internet-based vs. face-to-face cognitive behavior therapy for psychiatric and somatic disorders: an updated systematic review and meta-analysis. Cogn. Behav. Ther. 47, 1–18. doi: 10.1080/16506073.2017.1401115

PubMed Abstract | Crossref Full Text | Google Scholar

Carleton, R. N. (2021). Collaborating to support the mental health of public safety personnel: the Canadian Institute for Public Safety Research and Treatment (CIPSRT). Can. Psychol. 62, 167–173. doi: 10.1037/cap0000267

Crossref Full Text | Google Scholar

Carleton, R. N., Afifi, T. O., Taillieu, T., Turner, S., Krakauer, R., Anderson, G. S., et al. (2019a). Exposures to potentially traumatic events among public safety personnel in Canada. Can. J. Behav. Sci. 51, 37–52. doi: 10.1037/cbs0000115

Crossref Full Text | Google Scholar

Carleton, R. N., Afifi, T. O., Taillieu, T., Turner, S., Mason, J. E., Ricciardelli, R., et al. (2020a). Assessing the relative impact of diverse stressors among public safety personnel. Int. J. Environ. Res. Public Health 17:1234. doi: 10.3390/ijerph17041234

PubMed Abstract | Crossref Full Text | Google Scholar

Carleton, R. N., Afifi, T. O., Turner, S., Taillieu, T., LeBouthillier, D. M., Duranceau, S., et al. (2018a). Suicidal ideation, plans, and attempts among public safety personnel in Canada. Can. Psychol. 59, 220–231. doi: 10.1037/cap0000136

Crossref Full Text | Google Scholar

Carleton, R. N., Afifi, T. O., Turner, S., Taillieu, T., Vaughan, A. D., Anderson, G. S., et al. (2019b). Health training, attitudes towards support, and screening positive for mental disorders. Cogn. Behav. Ther. 49, 55–73.

Google Scholar

Carleton, R. N., Afifi, T. O., Turner, S., Taillieu, T., Vaughan, A. D., Anderson, G. S., et al. (2020b). Mental health training, attitudes towards support, and screening positive for mental disorders. Cogn. Behav. Ther. 49, 55–73. doi: 10.1080/16506073.2019.1575900

PubMed Abstract | Crossref Full Text | Google Scholar

Carleton, R. N., Korol, S., Mason, J. E., Hozempa, K., Anderson, G. S., Jones, N. A., et al. (2018b). A longitudinal assessment of the road to mental readiness training among municipal police. Cogn. Behav. Ther. 47, 508–528. doi: 10.1080/16506073.2018.1475504

PubMed Abstract | Crossref Full Text | Google Scholar

Combden, S., Forward, A., and Sarkar, A. (2022). COVID-19 pandemic responses of Canada and United States in first 6 months: a comparative analysis. Int. J. Health Plann. Manag. 37, 50–65. doi: 10.1002/hpm.3323

PubMed Abstract | Crossref Full Text | Google Scholar

Cox, D. W., Owen, J. J., and Ogrodniczuk, J. S. (2017). Group psychotherapeutic factors and perceived social support among veterans with PTSD symptoms. J. Nerv. Ment. Dis. 205, 127–132. doi: 10.1097/NMD.0000000000000635

PubMed Abstract | Crossref Full Text | Google Scholar

Crawford, J. N., Talkovsky, A. M., Bormann, J. E., and Lang, A. J. (2019). Targeting hyperarousal: Mantram repetition program for PTSD in US veterans. Eur. J. Psychotraumatol. 10:1665768. doi: 10.1080/20008198.2019.1665768

PubMed Abstract | Crossref Full Text | Google Scholar

Cutrona, C. E., and Russell, D. W. (1987). The provisions of social relationships and adaptation to stress. Adv. Personal Relationships 1, 37–67.

Google Scholar

de Meneses-Gaya, C., Zuardi, A. W., Loureiro, S. R., and Crippa, J. A. S. (2009). Alcohol use disorders identification test (AUDIT): an updated systematic review of psychometric properties. Psychol. Neurosci. 2, 83–97. doi: 10.3922/j.psns.2009.1.12

Crossref Full Text | Google Scholar

Di Nota, P. M., Bahji, A., Groll, D., Carleton, R. N., and Anderson, G. S. (2022). Psychological interventions for posttraumatic stress injuries among public safety personnel: a systematic review and meta-analysis. Syst. Rev. 11:2112. doi: 10.1186/s13643-022-02112-1

PubMed Abstract | Crossref Full Text | Google Scholar

Foa, E. B., and Rothbaum, B. O. (2001). Treating the trauma of rape: Cognitive-behavioral therapy for PTSD. New York, USA: Guilford Press.

Google Scholar

Fortier, C. B., Currao, A., Kenna, A., Kim, S., Beck, B. M., Katz, D., et al. (2022). Online telehealth delivery of group mental health treatment is safe, feasible, and increases enrollment and attendance in post-9/11 US veterans. Behav. Ther. 53, 469–480. doi: 10.1016/j.beth.2021.11.004

PubMed Abstract | Crossref Full Text | Google Scholar

Fowler, J. C., Charak, R., Elhai, J. D., Allen, J. G., Frueh, B. C., and Oldham, J. M. (2014). Construct validity and factor structure of the difficulties in emotion regulation scale among adults with severe mental illness. J. Psychiatr. Res. 58, 175–180. doi: 10.1016/j.jpsychires.2014.07.029

PubMed Abstract | Crossref Full Text | Google Scholar

Gottlieb, B. H., and Bergen, A. E. (2010). Social support concepts and measures. J. Psychosom. Res. 69, 511–520. doi: 10.1016/j.jpsychores.2009.10.001

Crossref Full Text | Google Scholar

Gratz, K. L., and Roemer, L. (2004). Multidimensional assessment of emotion regulation and dysregulation: development, factor structure, and initial validation of the difficulties in emotion regulation scale. J. Psychopathol. Behav. Assess. 26, 41–54. doi: 10.1023/B:JOBA.0000007455.08539.94

Crossref Full Text | Google Scholar

Hadjistavropoulos, H. D., McCall, H. C., Thiessen, D. L., Huang, Z., Carleton, R. N., Dear, B. F., et al. (2021). Initial outcomes of transdiagnostic internet-delivered cognitive behavioral therapy tailored to public safety personnel: longitudinal observational study. J. Med. Internet Res. 23:e27610. doi: 10.2196/27610

PubMed Abstract | Crossref Full Text | Google Scholar

Hallion, L. S., Steinman, S. A., Tolin, D. F., and Diefenbach, G. J. (2018). Psychometric properties of the difficulties in emotion regulation scale (DERS) and its short forms in adults with emotional disorders. Front. Psychol. 9:539. doi: 10.3389/fpsyg.2018.00539

PubMed Abstract | Crossref Full Text | Google Scholar

Heber, A., Testa, V., Groll, D., Ritchie, K., Tam-Seto, L., Mulligan, A., et al. (2023). Glossary of terms: a shared understanding of the common terms used to describe psychological trauma, version 3.0. Health Promot. Chronic Dis. Prev. Can. 43, S1–S999. doi: 10.24095/hpcdp.43.10/11.09

Crossref Full Text | Google Scholar

Heber, A., Testa, V., Smith-MacDonald, L., Brémault-Phillips, S., and Carleton, R. N. (2020). Rapid response to COVID-19: addressing challenges and increasing the mental readiness of public safety personnel. J. Health Promot Chronic Disease Prevent Canada 40, 350–355. doi: 10.24095/hpcdp.40.11/12.04

PubMed Abstract | Crossref Full Text | Google Scholar

Heck, R. H., Thomas, S. L., and Tabata, L. N. (2013). Multilevel and longitudinal modeling with IBM SPSS : Routledge.

Google Scholar

Heine, J. J., Land, W. H., and Egan, K. M. (2011). Statistical learning techniques applied to epidemiology: a simulated case-control comparison study with logistic regression. BMC Bioinf. 12:37. doi: 10.1186/1471-2105-12-37

PubMed Abstract | Crossref Full Text | Google Scholar

Henry, J. D., and Crawford, J. R. (2005). The short-form version of the depression anxiety stress scales (DASS-21): construct validity and normative data in a large non-clinical sample. Br. J. Clin. Psychol. 44, 227–239. doi: 10.1348/014466505X29657

Crossref Full Text | Google Scholar

Hossain, F., and Clatty, A. (2021). Self-care strategies in response to nurses moral injury during COVID-19 pandemic. Nurs. Ethics 28, 23–32. doi: 10.1177/0969733020961825

PubMed Abstract | Crossref Full Text | Google Scholar

Hox, J.J., and de Leeuw, E.D. (2003). Multilevel models for meta-analysis, Multilevel modeling. Psychology Press, 87–104

Google Scholar

Kosmicki, F. X., and Glickauf-Hughes, C. (1997). Catharsis in psychotherapy. Psychother. Theory Res. Pract. Train. 34, 154–159. doi: 10.1037/h0087724

Crossref Full Text | Google Scholar

Krakauer, R. L., Stelnicki, A. M., and Carleton, R. N. (2020). Examining mental health knowledge, stigma, and service use intentions among public safety personnel. Front. Psychol. 11:949. doi: 10.3389/fpsyg.2020.00949

PubMed Abstract | Crossref Full Text | Google Scholar

Lovibond, P. F., and Lovibond, S. H. (1995). The structure of negative emotional states: comparison of the depression anxiety stress scales (DASS) with the Beck depression and anxiety inventories. Behav. Res. Ther. 33, 335–343. doi: 10.1016/0005-7967(94)00075-u

Crossref Full Text | Google Scholar

Marchildon, G. P., Allin, S., and Merkur, S. (2020). Canada: health system review. Health Syst. Transit. 22, 1–194

PubMed Abstract | Google Scholar

McElheran, M., Stelnicki, A. M., and Carleton, R. N. (2020). Before operational stress (BOS): a proposed program to support mental health and proactive psychological protection in public safety personnel [registered report protocol]. Front. Psychol. :3218.

Google Scholar

Mendicino, M.E.L., and Blair, W.S. (2022). Departmental Plan 2022-23, (Ed.) P.S. Canada

Google Scholar

Miles, S. R., Hale, W. J., Mintz, J., Wachen, J. S., Litz, B. T., Dondanville, K. A., et al. (2023). Hyperarousal symptoms linger after successful PTSD treatment in active duty military. Psychol. Trauma 15, 1398–1405. doi: 10.1037/tra0001292

PubMed Abstract | Crossref Full Text | Google Scholar

Miles, M. B., Huberman, A. M., and Saldaña, J. (2018). Qualitative data analysis: A methods sourcebook. Los Angeles, London, New Delhi, Singapore, Washington DC: Sage publications.

Google Scholar

Newell, C. J., Ricciardelli, R., Czarnuch, S. M., and Martin, K. (2022). Police staff and mental health: barriers and recommendations for improving help-seeking. Police Pract. Res. 23, 111–124. doi: 10.1080/15614263.2021.1979398

Crossref Full Text | Google Scholar

Nisbet, J., Jamshidi, L., Andrews, K. L., Stewart, S. H., Shields, R. E., Teckchandani, T. A., et al. (2023). Mental health and social support among Royal Canadian Mounted Police cadets. Front. Psychol. 14:1092334. doi: 10.3389/fpsyg.2023.1092334

Crossref Full Text | Google Scholar

Patel, H., Easterbrook, B., D’Alessandro-Lowe, A. M., Andrews, K., Ritchie, K., Hosseiny, F., et al. (2023). Associations between trauma and substance use among healthcare workers and public safety personnel during the SARS-CoV-2 (COVID-19) pandemic: the mediating roles of dissociation and emotion dysregulation. Eur. J. Psychotraumatol. 14:2180706. doi: 10.1080/20008066.2023.2180706

PubMed Abstract | Crossref Full Text | Google Scholar

Peng, C.-Z., Wilsnack, R. W., Kristjanson, A. F., Benson, P., and Wilsnack, S. C. (2012). Gender differences in the factor structure of the alcohol use disorders identification test in multinational general population surveys. Drug Alcohol Depend. 124, 50–56. doi: 10.1016/j.drugalcdep.2011.12.002

PubMed Abstract | Crossref Full Text | Google Scholar

Pleil, J.D., Wallace, M.A.G., Stiegel, M.A., and Funk, W.E. (2018). Human biomarker interpretation: The importance of intra-class correlation coefficients (ICC) and their calculations based on mixed models, ANOVA, and variance estimates. England: Taylor & Francis 21, 161–180

Google Scholar

Ponniah, K., and Hollon, S. D. (2009). Empirically supported psychological treatments for adult acute stress disorder and posttraumatic stress disorder: a review. Depress. Anxiety 26, 1086–1109. doi: 10.1002/da.20635

PubMed Abstract | Crossref Full Text | Google Scholar

Public Safety Canada (2019). Supporting Canada’s public safety personnel: An action plan on post-traumatic stress injuries, (Ed.) P.S.a.E. Preparedness. Ottawa, ON: Government of Canada.

Google Scholar

Pustejovsky, J. E., Hedges, L. V., and Shadish, W. R. (2014). Design-comparable effect sizes in multiple baseline designs: a general modeling framework. J. Educ. Behav. Stat. 39, 368–393. doi: 10.3102/1076998614547577

Crossref Full Text | Google Scholar

Raudenbush, S. W., and Bryk, A. S. (2002). Hierarchical linear models: applications and data analysis methods : Sage.

Google Scholar

Reinert, D. F., and Allen, J. P. (2002). The alcohol use disorders identification test (AUDIT): a review of recent research. Alcohol. Clin. Exp. Res. 26, 272–279. doi: 10.1111/j.1530-0277.2002.tb02534.x

PubMed Abstract | Crossref Full Text | Google Scholar

Ricciardelli, R., Carleton, R. N., Mooney, T., and Cramm, H. (2020). Playing the system: structural factors potentiating mental health stigma, challenging awareness, and creating barriers to care for Canadian public safety personnel. Health 24, 259–278. doi: 10.1177/1363459318800167

PubMed Abstract | Crossref Full Text | Google Scholar

Rice, V. J., Liu, B., Allison, S. C., and Schroeder, P. J. (2019). Mindfulness training offered in-person and in a virtual world—weekly self-reports of stress, energy, pain, and sleepiness among US military active duty and veteran personnel. Mindfulness 10, 1815–1827. doi: 10.1007/s12671-019-01129-3

Crossref Full Text | Google Scholar

Rice, V. J., Liu, B., and Schroeder, P. J. (2018). Impact of in-person and virtual world mindfulness training on symptoms of post-traumatic stress disorder and attention deficit and hyperactivity disorder. Mil. Med. 183, 413–420. doi: 10.1093/milmed/usx227

PubMed Abstract | Crossref Full Text | Google Scholar

Rice, V. J., and Schroeder, P. J. (2021). In-person and virtual world mindfulness training: trust, satisfaction, and learning. Cyberpsychol. Behav. Soc. Netw. 24, 526–535. doi: 10.1089/cyber.2019.0590

PubMed Abstract | Crossref Full Text | Google Scholar

Rochlen, A. B., Zack, J. S., and Speyer, C. (2004). Online therapy: review of relevant definitions, debates, and current empirical support. J. Clin. Psychol. 60, 269–283. doi: 10.1002/jclp.10263

PubMed Abstract | Crossref Full Text | Google Scholar

Schnurr, P. P., and Lunney, C. A. (2019). Residual symptoms following prolonged exposure and present-centered therapy for PTSD in female veterans and soldiers. Depress. Anxiety 36, 162–169. doi: 10.1002/da.22871

PubMed Abstract | Crossref Full Text | Google Scholar

Skevington, S. M., Lotfy, M., and O’Connell, K. A. (2004). The World Health Organization’s WHOQOL-BREF quality of life assessment: psychometric properties and results of the international field trial. A report from the WHOQOL group. Qual. Life Res. 13, 299–310. doi: 10.1023/B:QURE.0000018486.91360.00

PubMed Abstract | Crossref Full Text | Google Scholar

Smith, B. W., Dalen, J., Wiggins, K., Tooley, E., Christopher, P., and Bernard, J. (2008). The brief resilience scale: assessing the ability to bounce back. Int. J. Behav. Med. 15, 194–200. doi: 10.1080/10705500802222972

PubMed Abstract | Crossref Full Text | Google Scholar

Spek, V., Nyklicek, I., Smits, N., Cuijpers, P., Riper, H., Keyzer, J., et al. (2007). Internet-based cognitive behavioural therapy for subthreshold depression in people over 50 years old: a randomized controlled clinical trial. Psychol. Med. 37, 1797–1806. doi: 10.1017/S0033291707000542

Crossref Full Text | Google Scholar

Steenkamp, M. M., Litz, B. T., Hoge, C. W., and Marmar, C. R. (2015). Psychotherapy for military-related PTSD: a review of randomized clinical trials. JAMA 314:489. doi: 10.1001/jama.2015.8370

Crossref Full Text | Google Scholar

Stelnicki, A. M., Jamshidi, L., Fletcher, A. J., and Carleton, R. N. (2021). Evaluation of before operational stress: a program to support mental health and proactive psychological protection in public safety personnel. Front. Psychol. 12:511755. doi: 10.3389/fpsyg.2021.511755

PubMed Abstract | Crossref Full Text | Google Scholar

Szeto, A. C. H., Luong, D., and Dobson, K. S. (2013). Does labeling matter? An examination of attitudes and perceptions of labels for mental disorders. Soc. Psychiatry Psychiatr. Epidemiol. 48, 659–671. doi: 10.1007/s00127-012-0532-7

PubMed Abstract | Crossref Full Text | Google Scholar

Thew, G. R., Rozental, A., and Hadjistavropoulos, H. D. (2022). Advances in digital CBT: where are we now, and where next? Cogn Behav Therap 15:E44. doi: 10.1017/S1754470X22000423

PubMed Abstract | Crossref Full Text | Google Scholar

Vig, K. D., Mason, J. E., Carleton, R. N., Asmundson, G. J. G., Anderson, G. S., and Groll, D. (2020). Mental health and social support among public safety personnel. Occup. Med. 70, 427–433. doi: 10.1093/occmed/kqaa129

PubMed Abstract | Crossref Full Text | Google Scholar

Weathers, F.W., Litz, B.T., Keane, T.M., Palmieri, P.A., Marx, B.P., and Schnurr, P.P. (2013). The PTSD checklist for DSM-5 (PCL-5). Scale available from the National Center for PTSD. National Center for PTSD. Available: www.ptsd.va.gov. [accessed].

Google Scholar

West, B. T., Beer, L., Gremel, G. W., Weiser, J., Johnson, C. H., Garg, S., et al. (2015). Weighted multilevel models: a case study. Am. J. Public Health 105, 2214–2215. doi: 10.2105/AJPH.2015.302842

PubMed Abstract | Crossref Full Text | Google Scholar

Westfall, J., Kenny, D. A., and Judd, C. M. (2014). Statistical power and optimal Design in Experiments in which samples of participants respond to samples of stimuli. J. Exp. Psychol. Gen. 143, 2020–2045. doi: 10.1037/xge0000014

PubMed Abstract | Crossref Full Text | Google Scholar

WHOQOL Group (1998). Development of the World Health Organization WHOQOL-BREF quality of life assessment. Psychol. Med. 28, 551–558. doi: 10.1017/S0033291798006667

Crossref Full Text | Google Scholar

Windle, G., Bennett, K. M., and Noyes, J. (2011). A methodological review of resilience measurement scales. Health Qual. Life Outcomes 9:8. doi: 10.1186/1477-7525-9-8

Crossref Full Text | Google Scholar

Xue, Y., Lopes, J., Ritchie, K., D’Alessandro, A. M., Banfield, L., McCabe, R. E., et al. (2022). Potential circumstances associated with moral injury and moral distress in healthcare workers and public safety personnel across the globe during COVID-19: a scoping review. Front. Psych. 13:863232. doi: 10.3389/fpsyt.2022.863232

PubMed Abstract | Crossref Full Text | Google Scholar

Yalom, I. D., and Leszcz, M. (2020). The theory and practice of group psychotherapy. New York City: Basic Books.

Google Scholar

Yu, A., Prasad, S., Akande, A., Murariu, A., Yuan, S., Kathirkamanathan, S., et al. (2020). COVID-19 in Canada: a self-assessment and review of preparedness and response. J. Glob. Health 10:203104. doi: 10.7189/jogh.10.0203104

PubMed Abstract | Crossref Full Text | Google Scholar

Zhang, W., Du, Y., Yang, X., Wang, E., Fang, J., Liu, Z., et al. (2022). Comparative efficacy of face-to-face and internet-based cognitive behavior therapy for generalized anxiety disorder: a meta-analysis of randomized controlled trial. Front. Psych. 13:832167. doi: 10.3389/fpsyt.2022.832167

PubMed Abstract | Crossref Full Text | Google Scholar