Expanding access to evidence-based psychotherapy in VA settings: implementation of the brief cognitive behavioral therapy for depression program

Introduction: Evidence-based psychotherapies (EBPs) are effective for mental health conditions, but access to these services remains limited and rural Veterans are particularly underserved. Specialized implementation and dissemination programs are needed to improve access to known EBPs.

Methods: The current project sought to improve access to a known EBP—brief Cognitive Behavioral Therapy for depression (Brief CBT). Diverse Veterans and those from rural settings were a focus of this work. Aligned with the RE-AIM framework, a multifaceted implementation program was used to train and support VHA providers in their use of Brief CBT in VHA mental health settings, with specific outreach efforts made to providers at VHA Community-Based Outpatient Clinics (CBOCs) where rural Veterans often receive care. Evaluation included all facets of RE-AIM with a particular focus on adoption, effectiveness, and maintenance.

Results: During the first two years, over 40 VHA facilities adopted the program across four regional networks. Eighty-three providers were approached, and 54 (65.1%) providers completed the training and are delivering the intervention. A total of 688 Veterans, 174 rural (25.7%), received 2,186 sessions (average of 3.5 sessions per Veteran). Veterans receiving Brief CBT with elevated depression scores who completed three or more sessions were found to have significant symptom reductions of 4.6 points (first to last available evaluations).

Discussion: Implementation efforts of Brief CBT resulted in rapid uptake and significant clinical impact on Veterans. Rural outreach efforts, including targeted training for CBOC providers and use of tele-mental health, enhanced availability of EBP services for rural Veterans.

Introduction

The Veterans Health Administration (VHA) has invested in multiple national evidence-based psychotherapy (EBP) rollouts (1–6) to improve access to high-quality mental health services. EBP trainings often involve a multi-day workshop, post-workshop supervision/consultation, and fidelity assessments and session audio reviews. Although these efforts have led to an increased number of trained providers, delivery of EBPs remains limited and rural Veterans are often negatively affected (7–10).

Factors contributing to underutilization of EBPs are often complex and occur at the patient, provider, and organizational levels (11–13). Consequently, engaging clinical and operational leaders early in the implementation and dissemination process appears to be critical to ensuring EBP uptake and sustainability (14–18). Equally important is engaging providers early and often when implementing EBPs (19) as a critical breakdown in EBP application occurs between training and its use in clinical settings (7).

This paper reports findings from a demonstration project that used a multifaceted implementation program to train and support VHA providers in delivering a brief Cognitive Behavioral Therapy for depression, known as Brief CBT (20). Funding from VHA Office of Rural Health (ORH) allowed the project to target providers at Community-Based Outpatient Clinics (CBOCs), where rural Veterans often receive care. The RE-AIM framework (21) enabled us to capture and continuously monitor the impact of this demonstration project, specifically, on the program's ability to train and support providers in the delivery of Brief CBT (adoption), Veteran depression clinical outcomes (effectiveness), as well as continued use of the program at VHA facilities over time (maintenance). Although not primary evaluation factors, additional data on reach and implementation were collected.

Methods

Overview

The current project sought to enhance delivery of Brief CBT using a multi-faceted provider training and support program. Brief CBT (3–6 sessions) is an empirically supported individual therapy program, designed to improve Veteran depression (22, 23). With a focus on implementation and quality improvement, this project was deemed exempt from IRB review by the Houston VA's Research and Development Office. Data for the current manuscript included training and services delivered between February 1, 2020 and December 31, 2022.

Planned implementation sites

The Brief CBT implementation program was supported by two Veterans Integrated Service Networks (VISNs; VHA's regional organizational structure), in the South-Central United States. The first and second provider training cohorts focused on primary care mental health program providers in one VISN. Following this, a third training cohort targeted psychotherapy providers (e.g., psychologists, social workers) in primary care and CBOCs in both VISNs. A fourth cohort targeted rural and CBOC providers from either VISN. Among the planned implementation sites, there were 8 VHA Medical Centers and 29 CBOCs, for a total of 37 VHA facilities.

Multicomponent implementation strategy

The integrated-Promoting Action on Research Implementation in Health Services (i-PARIHS) framework was used to guide our implementation strategy (24). I-PARIHS posits successful implementation as a function of four components (i.e., facilitation, innovation, recipients, and context). Facilitation was the primary mechanism driving the Brief CBT provider and support program with focused attention to help providers adopt the program within their specific clinical context.

The resulting Brief CBT provider training and facilitation program was designed for primary care and CBOC settings. The multifaceted strategy included treatment materials (i.e., provider manuals and patient workbook), web-based provider training (i.e., online and modular based training), consultation and support, as well as clinical and delivery resources including an electronic progress note template and a web-based Brief CBT delivery dashboard. These components are detailed below.

Treatment materials

A hardcopy and electronic links to the treatment manual and patient workbook were given to all providers. While core components of the treatment have not been altered, contextual modifications to the treatment components were made as part of the lead up to the current demonstration project to increase the likelihood of provider adoption and ease of use. For example, an abbreviated treatment manual showing a condensed view of each session's content was created to assist providers in delivering the treatment with fidelity.

Web-based brief CBT provider training curriculum

Training consisted of five one-hour modules and included the opportunity to earn professional educational credits. Using a standardized process for assessing and collaboratively discussing training needs with each provider, training modules were selected in collaboration between providers and Brief CBT consultants to “right size” the training experience based on provider needs. Once the curriculum was determined, providers were encouraged to complete the selected trainings within 2–3 weeks.

Facilitation (hereafter referred to as consultation)

Providers met with a Brief CBT consultant through individual video conference calls (15–20 min) monthly for 4–5 months. Consultation focused on reinforcing learned information from training modules, resolving implementation challenges, and providing clinical consultation. Providers were encouraged to use the electronic progress note and deliver measurement-based care using the Patient Health Questionnaire-9 (PHQ-9) at each session.

Clinical and delivery resources

A Brief CBT electronic progress note was created for use in the VHA's electronic health records system. The template was designed to assist providers in quickly and accurately documenting Brief CBT sessions (e.g., pre-populated radio buttons to select content delivered and measurement-based care elements—i.e., PHQ-9 and Columbia Suicide Rating Scale).

A delivery dashboard was developed to capture and share real-time data collected through the Brief CBT electronic progress note with Brief CBT providers and other relevant operational partners. The dashboard included: number of unique patients served, number of Brief CBT sessions delivered, number of sessions with a PHQ-9 administered, and depression outcome change scores as assessed by the PHQ-9.

RE-AIM

The RE-AIM framework (21) was selected to guide evaluation across all dimensions, however, focused on adoption, effectiveness, and maintenance. Adoption was defined as the number of providers trained and using the program in clinical practice. Effectiveness was defined as within patient depression score (i.e., PHQ-9) changes during treatment. Maintenance was defined as the number of VHA facilities engaged that continued to use the program in the final six months data were captured for this paper. Secondary outcomes examined Reach (ability of the program to reach diverse Veterans, especially, those from rural settings subset of Veterans) and Implementation (degree to which the program was delivered as intended).

Data collection methods

Clinical and delivery data

Primary programmatic data were collected using information from the Brief CBT electronic progress note, including session type, session length, delivery modality, and specific Brief CBT content. In addition, note template data included information on PHQ-9 items and total scores as well as linking those scores to basic demographic data (e.g., age, race/ethnicity, gender), as reported in VHA electronic health records database.

Provider feedback: survey and interview data

Survey questions, and follow-up interviews, were used to solicit feedback from providers in the earlier training cohorts. Cohort 1 (n = 10 providers) were contacted via email and asked if they were willing to share “what worked well and what we can improve about our program to better meet the needs of … providers.” Once scheduled, providers received a pre-survey via e-mail to complete and return prior to the interview. The survey consisted of 16 questions divided into three sections: (1) effectiveness of provider training and consultation, (2) delivery (e.g., usability of intervention materials), and (3) program impact (e.g., overall impact on patients). Questions included Likert-scale response options, rating aspects of Brief CBT from 5 (No improvements needed/Excellent) to 1 (Major improvements needed/Poor). A brief (20–40 min) semi-structured interview after the survey allowed providers to elaborate on their ratings. The interviewer took notes, and immediately after, edited for clarity. Interview notes and qualitative responses included in the survey-portion were deductively coded using the RE-AIM framework.

The second cohort of providers (n = 9) were asked to complete the same survey using an online portal. Given data saturation from interviews completed with the first cohort, this cohort was only provided with open-ended text boxes to provide written feedback and an option to schedule an individual interview to share verbal feedback. Mean and ranges for survey items were computed.

Across the two provider training cohorts, 16 of the 19 providers (84.2%) completed the survey, and 9 of the 10 (90.0%) providers in the first cohort also completed the individual semi-structured interview. No providers from the second cohort requested a follow-up individual interview.

Results

Adoption

Brief CBT adoption occurred at 41 VHA facilities (9 Medical Centers and 32 CBOCs) across four VISNs. In addition to our planned implementation sites, sites at two non-planned VISNs (i.e., one VHA Medical Center and three CBOCs) were added to the current demonstration project after their leadership expressed interest in their providers using Brief CBT. Overall, 83 providers were approached, and from these, 54 (65.1%) completed the training and planned to adopt Brief CBT into their practice. Of these, Brief CBT was successfully adopted by 44 providers working across 37 VHA facilities (i.e., eight VHA Medical Centers and 29 CBOCs within four VISNs). In contrast, 10 of the 54 providers working at seven of the planned implementation sites (i.e., four VHA Medical Centers and three CBOCs, across two VISNs) did not complete the consultation process after completing the training.

Survey items about Adoption assessed providers evaluations about the 1) Brief CBT Needs Assessment and Tailored Curriculum Plan, 2) Web-based Training Portal, and 3) Consultation Sessions. Mean scores were 4.56, 4.63, and 4.63, respectively. Similarly, an item regarding preparedness in delivering Brief CBT to patients after the consultation program ended had a mean rating of 4.56. Providers viewed Brief CBT as a good fit for their specific clinical setting (x¯ = 4.63) and job responsibilities (x¯ = 4.44) and regarded the treatment materials positively (x¯ = 4.69). Provider survey open-ended responses suggested the satisfaction with training and consultation program (e.g., “I really enjoyed the program and the support the calls offered”; PC-MHI Provider #11), identified some challenges associated with completing the online training (e.g., “Sitting in front of TMS modules is pretty tough”; PC-MHI Provider #8), and provided suggestions for further adaptations (e.g., “I would suggest for the workbook to be shorter and for the templates to have more options on the check marks since some of them were repetitive and/or did not apply”; CBOC Provider #10)

Finally, providers shared information related to Adoption during individual interviews. Providers commented that the treatment worked well in their clinical settings. One provider stated, “This is now a part of my toolkit and I talk to patients about it on a consistent basis as part of a list of treatment options. The nice thing is that all the different options I also talk about tend to be already combined into the program in some way. So normally I would offer all the treatment options separately, but now the program incorporates most treatment options into one simple program so it's great” [PC-MHI Provider #3]. Other providers commented, “The flexibility was also great—certain [patients]… I’m seeing may have depressive symptoms due to chronic pain, or anxiety, or lack of sleep. I could flexibly use different parts of the treatment. Also the cards for the physical health component were great” [PC-MHI Provider #9]. This provider went on to comment about how their “…patients really like the workbook, it's easy to follow, the pictures, stories, and everything that's in there is great.” While the treatment was praised for its flexibility, one provider advised increasing its adoption potential by editing the manual “to allow for more room to adapt it or give multiple options for things so you can choose which one you like the most as the psychotherapist” [PC-MHI Provider #3].

Effectiveness

Treatment outcomes were captured using PHQ-9 total change scores (see Table 1). These outcome data were included for patients with two or more PHQ-9 scores, among those with an initial PHQ-9 score of 10 or greater, indicating the presence of at least mild-moderate depressive symptoms. Among these patients (n = 263), the impact of treatment was statistically significantly different among those who completed 3 or more sessions [χ2(1, N = 263) = 5.16, p = .02] or 4 or more [χ2(1, N = 263) = 4.66, p = .03] when compared to those with fewer sessions. Additionally, while this demonstration project was not powered to test for differences, data suggest equivocal outcomes across gender (p = .14), race (p = .08), ethnicity (p = .30), and rurality status (p = .90).

TABLE 1

Table 1. Treatment outcomes for patients with two or more PHQ-9 scores with an initial PHQ-9 score of 10 or greater.

Providers' ratings on survey items related to Effectiveness were positive. Specifically, the statement, Overall impact on patients had a mean rating of 4.25. The statement, Impact on patient depression scores had a mean rating of 4.13. One provider wrote, “I love these interventions and how easy and meaningful they are” [CBOC Provider #10]. Another provider commented about how the PHQ-9 did not capture all patient improvements on depressive symptoms, stating, “Sometimes people report functional improvement, but their PHQ-9 scores may stay the same. When discussing this with the patient, I don't really get any more clarity on why that is the case” [PC-MHI Provider #1].

During individual interviews, providers described how Brief CBT had a positive impact on patients during and after Brief CBT. One provider stated: “This training has improved my overall effectiveness with patients. In a couple of sessions, with some patients, they were good to go. For example, one patient realized after a couple sessions, ‘My feelings are not always representative of reality—now I see where I was making a mistake. I was letting my emotions lead,’ and that was enough for her” [PC-MHI Provider #3].

Maintenance

Of the planned implementation sites (37 facilities across 2 VISNS), all facilities (100%) continue to have one or more providers delivering Brief CBT in the final six months data were captured for this paper. Survey items related to Maintenance included provider responses to the question, Impact on your own professional development. This item had a mean rating of 4.44. In interviews, providers expressed their desire to continue to use Brief CBT over time. One provider wrote, “Once a tool, always a tool. I would never let go of anything like this [PC-MHI Provider #6], and another stated “Still using this program and still planning on using it and moving forward with it. I always keep checking with my patients to make sure they enjoy it” [PC-MHI Provider #7]. One provider expressed a desire to see Brief CBT implemented more broadly across the VHA, stating, “I believe it should be rolled out to other [VHAs], very useful. Would be great to roll out to other [VHAs] and other VISNS in terms of saving time for other providers. Sometimes we have 10–12 patients per day. Things like this program are key and are useful to make sure the patient is on board for treatment and benefitting from the treatment” [PC-MHI Provider #9]. Another provider described the expected impact of using Brief CBT going forward, saying, “I think it's great and definitely something that would be more helpful for providers by giving structure to treatment and having helpful things delivered every session. That way it cuts down on only supportive therapy” [PC-MHI Provider #1].

Reach

Patients receiving brief CBT were racially and ethnically diverse (see Table 2). Treatment was delivered to 688 unique patients (25.3% rural), with an average age of 49.5 (SD = 16.5) years, and consistent with the VHA patient population, mostly males (79.4%). While a majority were White (65.0%) and Non-Hispanic or Latina/x (41.6%), racial and ethnic diversity were represented, with 21.2% identified as African American and 33.6% as Hispanic or Latina/x.

TABLE 2

Table 2. Patient demographics.

Modality of Brief CBT delivery, a potentially critical factor in enhancing intervention reach, was also diverse, with over half of sessions delivered by video telehealth (52.6%). Other delivery modalities included face-to-face (37.6%) and telephone (9.8%). Also of note, some providers adapted Brief CBT for delivery in group format, represented by 4.4% of the total sessions delivered (see Table 3).

TABLE 3

Table 3. Brief CBT delivery data.

From the provider survey, items evaluating Reach assessed the usability of the intervention materials for patients, responses had a mean rating of 4.5 (rating between Good to No improvements needed/Excellent descriptive anchors). The broader applicability of Brief CBT was captured in provider feedback during interviews. One provider stated, “This is not intensive so it's great for [patients]… that are ambivalent towards therapy. Like dipping your toes in the water so I think every [patient]… could use this” [PC-MHI Provider #8]. Another provider wrote, “I’ve gotten feedback from patients that it's easy to follow. It's one of those treatments that they have actually shared with family or loved ones (do behavioral activation with them or getting them more involved)—gets patients excited which makes it useful. This is a very user-friendly program” [PC-MHI Provider #9]. Another provider commented about the benefits of the patient workbook, stating, “Patients won't read [a] very dense manual. Less words, lots of pictures, easy for them to open up and get the idea that way. The patient manual [i.e., workbook] was good like that” [PC-MHI Provider #3].

Implementation

Implementation outcomes focused on if treatment was delivered as intended, and what, if any, modifications were required to implement it into practice. While fidelity to the treatment protocol was not directly assessed, other delivery data were used as measures of fidelity. First, the target length of each session was designed to be delivered in as little as 30 min to accommodate providers in primary care settings. Consistent with this, delivery data indicate 63.5% of sessions were delivered in this amount of time (see Table 3). Additionally, the target number of Brief CBT sessions, as determined by prior randomized trials (22), was three or four. In the current sample, the average number of sessions across all patients was 3.5 (SD = 2.5) with 22.1% (n = 152) of all patients receiving exactly 3 or 4 sessions.

Provider use of measurement-based care practices was viewed as an additional construct of intervention fidelity. Delivery data indicate 85.5% (n = 588) of patients with Brief CBT sessions had at least one PHQ-9 collected. Of these patients, 68.0% (n = 400) had two or more PHQ-9 collected. Treatment fidelity can also be assessed in relation to the care provided to urban (x¯ = 3.6 (SD = 2.4) sessions) vs. rural (x¯ = 3.5 (SD = 2.6) sessions) patients (see Table 4). Chi-square test comparing clinically meaningful groupings of number of treatment sessions (i.e., 1–2, 3–4, or 5 or more sessions) for rural vs. urban patient groups was not significant (χ2(2, N = 676) = .057, p = .972); suggesting that rural Veterans received a similar amount of care relative to their urban counterparts.

TABLE 4

Table 4. Comparison of rural and urban patients.

Providers had generally positive ratings on the survey related to Implementation. Specifically, the item that assessed the usability of the CPRS note template had a mean rating of 4.31. Written comments from one provider describe a preference for the “workbook to be shorter” and shared ideas for making the electronic progress note template more efficient. Provider interviews suggested providers viewed Brief CBT as a “good fit” for PCMHI clinics. However, some providers reported experiencing some challenges around delivering treatment in a 30-min session. One provider described this challenge, particularly in relation to Session One, and the resulting impact, stating, “We only have 25–30 min to do each session, so the Session One has a lot of information to cover in there and trying to get all of that in is a little bit difficult, and if I can fit it all in I do, if I can't then I can't” [PC-MHI Provider #3]. Another provider highlighted challenges associated with delivering EBPs, stating how it was “…hard to keep the patient in a more active role—sometimes it can be difficult since we want treatment to be Veteran-centered, but also follow the protocol. If the patient comes in talking about anything and everything else, it can add some pressure” [PC-MHI Provider #7].

Appreciation was expressed for consultants’ responsiveness to feedback, noting how that made it easier to use Brief CBT in practice. One provider stated, “I would give feedback and have things updated shortly after by the team. For example, within the note template, suicide risk assessments are part of every session. The note template was updated so I could add that in easily. Consultants are responsive to what the providers need” [PC-MHI Provider #3]. Appreciation was also shared for how the program honored clinicians, with one provider stating, in contrast to other more structured EBP training programs, “…this program is more about ‘you are a professional, here's a guideline for the treatment and use it as you see fit’” [PC-MHI Provider #3].

Discussion

Implementation of Brief CBT resulted in rapid uptake and significant clinical impact on patients across a large healthcare system. The Brief CBT implementation program was quickly adopted (within two years) by over 40 VHA facilities spread across 4 regional networks and sustained over time. There was substantial interest and support for the program, as 54 providers were trained and delivered over 2,186 Brief CBT sessions to a diverse group of 688 patients (average of 3.5 sessions per patient). Importantly, patient depression scores indicate significant symptom reductions of 4.6 points (first to last available evaluations) on the PHQ-9 among those receiving three or more Brief CBT sessions.

To reach rural patients, the Brief CBT Resource Team targeted training for VHA CBOCs. Additionally, tele-mental health procedures were employed to enhance the availability of EBP for rural patients. Multiple options for how to receive care might be beneficial and/or needed for rural veterans to connect. Consequently, providers were encouraged to use whatever delivery modality was most convenient for their patients. Selection appears to be, at least in part, dependent upon patient preference/needs, as treatment was most commonly engaged via video sessions.

RE-AIM provided a useful evaluative framework for ongoing assessment of implementation efforts. Solicited feedback from stakeholders (e.g., advisory council) leading up to implementation, and providers during implementation, allowed the Brief CBT Resource Team to be responsive to identified needs and challenges and success in the Implementation and Reach domains. Adaptations were made to peripheral aspects of the treatment and how it was delivered, including the following:

• Treatment materials were made more accessible to support virtual delivery (e.g., patient workbook was modified to be a fillable PDF, allowing patients to mark on these forms without requiring that they be printed), and posted on a public facing website.

• A new patient vignette was added to the patient workbook to broaden the applicability of the workbook examples for a diverse patient population.

• A “Session 0” was added to the treatment manual to accommodate the initial intake session to increase the fit of Brief CBT within the context of PC-MHI.

• Columbia Suicide Risk Assessment measure was added to the electronic progress note template to comply with VA policy around suicide risk assessment.

A significant challenge to multisite implementation in a large healthcare system is accounting for expected turnover of providers, for example, when newly trained providers move to different clinics or accept positions at new locations. However, this also provides a potential opportunity to spread an EBP to new sites. In the current project, two recently trained providers accepted new positions at different VHA facilities; however, they continued to use Brief CBT and/or provided consultation/supervision of colleagues/trainees to deliver Brief CBT. The Resource Team reached out to these providers to offer assistance to support their efforts. Also of note, it is important to work with local and regional leadership to support an infrastructure capable of training new providers, as well as monitoring those already trained to encourage their continued use of the EBP.

We were pleased to see how Brief CBT spread by “word of mouth.” This provides another measure of Reach and Adoption. Resource Teams should develop flexible plans for how they will support unplanned implementation sites. Going forward, we are utilizing the World Health Organization ExpandNet Scaling Up Framework to guide our implementation efforts at planned and unplanned sites/regional networks (24).

Limitations should be noted. This was a clinical demonstration project, and consequently, effectiveness outcomes should be considered in this context (no control group). This project also did not allow for testing of implementation facets—so it is not clear which elements were more or less impactful on delivery and adoption.

Conclusion

This paper reported findings of a demonstration project using a multifaceted implementation program to train and support providers to deliver a brief psychotherapy for depression, that included focused efforts to support providers serving rural Veterans. RE-AIM provided a useful evaluative framework that allowed for ongoing assessment of implementation outcomes, particularly outcomes related to adoption, effectiveness, and maintenance. Program outcomes suggest that targeted implementation efforts to train and support VHA CBOC providers generated high rates of adoption, clear and positive impact on rural Veteran outcomes, and suggest that the program is sustainable over time.

Data availability statement

The datasets presented in this article are not readily available because patient electronic health records data are owned by the Veterans Health Administration and the corresponding author does not have the authority to share these raw anonymized data. The interview and survey data generated by providers are not publicly available because these data contain potentially identifying information, and providers were assured that the information they provided would be publicly available only in aggregate. Requests to access the datasets should be directed to Office of Research and Development, Department of Veterans Affairs.

Ethics statement

Ethical review and approval was not required for this study in accordance with the national legislation and the institutional requirements.

Author contributions

JC, JM, DB, DZ, and AR contributed to conception and design of the study. DZ, AR, RG, and SS coordinated, collected, and organized data. SS performed statistical analysis, and RG, JM, DZ and JAC coded qualitative data. All authors were involved in analysis and/or interpretation of data. JM, RG, and JC wrote the first draft of the manuscript. All authors contributed to the article and approved the submitted version.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

1. Karlin BE, Ruzek JI, Chard KM, Eftekhari A, Monson CM, Hembree EA, et al. Dissemination of evidence-based psychological treatments for posttraumatic stress disorder in the Veterans Health Administration: evidence-based psychological treatments for ptsd in vha. J Traum Stress. (2010) 23:663–73. doi: 10.1002/jts.20588

CrossRef Full Text | Google Scholar

2. Ruzek JI, Karlin BE, Zeiss A. Implementation of evidence-based psychological treatments in the Veterans Health Administration. In: McHugh RK, Barlow DH, editors. Dissemination and implementation of evidence-based psychological interventions. Oxford University Press (2015). p. 78–96. doi: 10.1093/med:psych/9780195389050.003.0005

CrossRef Full Text | Google Scholar

3. Foa EB, Gillihan SJ, Bryant RA. Challenges and successes in dissemination of evidence-based treatments for posttraumatic stress: lessons learned from prolonged exposure therapy for ptsd. Psychol Sci Public Interest. (2013) 14:65–111. doi: 10.1177/1529100612468841

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Resick PA, Monson CM, Chard KM. Cognitive processing therapy for PTSD: A comprehensive manual. New York: Guilford Press (2017).

5. Wenzel A, Brown G, Karlin B. Cognitive behavioral therapy for depression in veterans and military servicemembers: Therapist manual. Washington, DC: U.S. Department of Veterans Affairs (2011).

6. Department of Veterans Affairs. Strategic Plan: 2005–2010. Office of the Inspector General, Department of Veterans Affairs; (2005).

7. Rosen CS, Eftekhari A, Crowley JJ, Smith BN, Kuhn E, Trent L, et al. Maintenance and reach of exposure psychotherapy for posttraumatic stress disorder 18 months after training: maintenance and reach of prolonged exposure. J Trauma Stress. (2017) 30:63–70. doi: 10.1002/jts.22153

PubMed Abstract | CrossRef Full Text | Google Scholar

8. Finley EP, Garcia HA, Ketchum NS, McGeary DD, McGeary CA, Stirman SW, et al. Utilization of evidence-based psychotherapies in veterans affairs posttraumatic stress disorder outpatient clinics. Psychol Serv. (2015) 12:73–82. doi: 10.1037/ser0000014

PubMed Abstract | CrossRef Full Text | Google Scholar

9. Bruns EJ, Kerns SEU, Pullmann MD, Hensley SW, Lutterman T, Hoagwood KE. Research, data, and evidence-based treatment use in state behavioral health systems, 2001–2012. PS (Wash DC). (2016) 67:496–503. doi: 10.1176/appi.ps.201500014

CrossRef Full Text | Google Scholar

10. Mott JM, Grubbs KM, Sansgiry S, Fortney JC, Cully JA. Psychotherapy utilization among rural and urban veterans from 2007 to 2010: psychotherapy use among rural veterans. J Rural Health. (2015) 31:235–43. doi: 10.1111/jrh.12099

PubMed Abstract | CrossRef Full Text | Google Scholar

11. Cochrane LJ, Olson CA, Murray S, Dupuis M, Tooman T, Hayes S. Gaps between knowing and doing: understanding and assessing the barriers to optimal health care. J Contin Educ Health Prof. (2007) 27:94–102. doi: 10.1002/chp.106

PubMed Abstract | CrossRef Full Text | Google Scholar

12. Cabana MD, Rand CS, Powe NR, Wu AW, Wilson MH, Abboud P-AC, et al. Why don’t physicians follow clinical practice guidelines?: a framework for improvement. JAMA. (1999) 282:1458. doi: 10.1001/jama.282.15.1458

PubMed Abstract | CrossRef Full Text | Google Scholar

13. Proctor EK, Landsverk J, Aarons G, Chambers D, Glisson C, Mittman B. Implementation research in mental health services: an emerging science with conceptual, methodological, and training challenges. Adm Policy Ment Health. (2009) 36:24–34. doi: 10.1007/s10488-008-0197-4

PubMed Abstract | CrossRef Full Text | Google Scholar

14. Yano EM. A partnered research initiative to accelerate implementation of comprehensive care for women veterans: the VA women’s health create. Med Care. (2015) 53:S10–4. doi: 10.1097/MLR.0000000000000340

PubMed Abstract | CrossRef Full Text | Google Scholar

15. World Health Organization. Practical guidance for scaling up health service innovations. 2009; 54. Available at: https://apps.who.int/iris/handle/10665/44180

16. Charns MP, Egede LE, Rumsfeld JS, McGlynn GC, Yano EM. Advancing partnered research in the VA healthcare system: the pursuit of increased research engagement, responsiveness, and impact. J Gen Intern Med. (2014) 29:811–3. doi: 10.1007/s11606-014-3060-1

PubMed Abstract | CrossRef Full Text | Google Scholar

17. Kilbourne AM, Atkins D. Partner or perish: VA health services and the emerging bi-directional paradigm. J Gen Intern Med. (2014) 29:817–9. doi: 10.1007/s11606-014-3050-3

PubMed Abstract | CrossRef Full Text | Google Scholar

18. Midboe AM, Elwy AR, Durfee JM, Gifford AL, Yakovchenko V, Martinello RA, et al. Building strong research partnerships between public health and researchers: a VA case study. J Gen Intern Med. (2014) 29:831–4. doi: 10.1007/s11606-014-3017-4

PubMed Abstract | CrossRef Full Text | Google Scholar

19. Concannon TW, Meissner P, Grunbaum JA, McElwee N, Guise J-M, Santa J, et al. A new taxonomy for stakeholder engagement in patient-centered outcomes research. J Gen Intern Med. (2012) 27:985–91. doi: 10.1007/s11606-012-2037-1

PubMed Abstract | CrossRef Full Text | Google Scholar

20. Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health. (1999) 89:1322–7. doi: 10.2105/AJPH.89.9.1322

PubMed Abstract | CrossRef Full Text | Google Scholar

21. Cully JA, Stanley MA, Petersen NJ, Hundt NE, Kauth MR, Naik AD, et al. Delivery of brief cognitive behavioral therapy for medically ill patients in primary care: a pragmatic randomized clinical trial. J Gen Intern Med. (2017) 32:1014–24. doi: 10.1007/s11606-017-4101-3

PubMed Abstract | CrossRef Full Text | Google Scholar

22. Ecker AH, Johnson AL, Sansgiry S, Fletcher TL, Hundt N, Petersen NJ, et al. Brief cognitive behavioral therapy reduces suicidal ideation in veterans with chronic illnesses. Gen Hosp Psychiatry. (2019) 58:27–32. doi: 10.1016/j.genhosppsych.2019.02.002

PubMed Abstract | CrossRef Full Text | Google Scholar

23. Harvey G, Kitson A. PARIHS Revisited: from heuristic to integrated framework for the successful implementation of knowledge into practice. Implementation Sci. (2015) 11:33. doi: 10.1186/s13012-016-0398-2

CrossRef Full Text | Google Scholar

24. Boykin DM, Wray LO, Funderburk JS, Holliday S, Kunik ME, Kauth MR, et al. Leveraging the ExpandNet framework and operational partnerships to scale-up brief cognitive behavioral therapy in VA primary care clinics. J Clin Trans Sci. (2022) 6:e95. doi: 10.1017/cts.2022.430

CrossRef Full Text | Google Scholar