Pilot of a team-based quality improvement strategy to improve cardiovascular risk factors care in community mental health centers

Introduction: Populations with serious mental illness are less likely to receive evidence-based care for cardiovascular disease (CVD) risk factors. We sought to characterize the implementation of an adapted team-based quality improvement strategy to improve mental health providers’ delivery of evidence-based CVD risk factor care.

Methods: In a 12-month, single arm pre/post pilot study in four behavioral health homes embedded within psychiatric rehabilitation programs, sites implemented an adapted Comprehensive Unit Safety Program (CUSP). Primary measures examined changes in organizational quality improvement culture and provider self-efficacy for CVD risk factor care. Secondary measures examined changes in acceptability, appropriateness, and feasibility of CUSP and receipt of guideline-concordant care for hypertension, dyslipidemia, and diabetes.

Results: Provider self-efficacy to coordinate care for hypertension and diabetes improved, but organizational quality improvement culture did not change. Acceptability, appropriateness, and feasibility were rated highly but did not change pre/post CUSP. The percentage who reached goals per national guidelines improved for those with dyslipidemia but not for those with hypertension or diabetes. CUSP teams implemented processes to build staff capacity, standardize communication, elicit feedback, and deliver education on coordination for CVD risk factors.

Conclusion: This pilot study showed no effects of CUSP on organizational quality improvement culture or provider self-efficacy, the mechanisms by which CUSP is expected to improve care processes. Long term investments are needed to support organizational quality improvement work and providers’ efficacy to delivery - evidence-based CVD risk factor care delivery.

Clinical Trial Registration: http://www.ClinicalTrials.gov, identifier NCT04696653.

1 Introduction

Cardiovascular disease (CVD) is the leading cause of premature mortality in populations with serious mental illness (SMI), such as schizophrenia or bipolar disorder (1, 2). Populations with SMI have a high prevalence of CVD risk factors, including hypertension, diabetes, dyslipidemia, obesity, and tobacco use (3–6). However, this population with SMI is also less likely to receive evidence-based care for these CVD risk factors (3–6). Use of psychotropic medications further elevates their CVD risk (7, 8).

Within the United States, CVD risk factor management has mainly occurred in primary care. The American College of Cardiology and American Heart Association recommend a team-based care approach to manage CVD risk (9). They outline specific blood pressure, glycemic, and cholesterol goals and recommendations to prevent CVD, particularly for those with multiple risk factors (9, 10). Some recommendations, such as dietary modifications and increasing physical activity pertain to all individuals at risk for CVD. In addition, some medications are recommended across two or more CVD risk factors, such as statin therapy for diabetes and dyslipidemia treatment or ACE inhibitors/angiotensin receptor blockers for diabetes and hypertension treatment (10, 11).

Yet people with SMI are seen frequently in the specialty mental health settings (12). This historical separation, or siloing of care, has led to patients falling through the cracks in the U.S. health care system (12). Initial efforts to integrate specialty mental health and physical health settings were based in primary care (e.g. Primary Care Medical Home) (13, 14). In 2010, the U.S. Congress passed the Affordable Care Act, a comprehensive bill aimed at reforming healthcare delivery. One provision allowed the public health insurance, Medicaid, to fund a health home waiver program. This “health home” program was implemented across 18 states and the District of Columbia and allows specialty mental health centers to bill for care coordination and care management for physical health conditions, such as CVD and CVD risk reduction (15). Clinical trials using nurse care managers to conduct care coordination and population health strategies demonstrated positive outcomes in cardiometabolic care, measured as a composite score, and in cholesterol management (16–18). However, real-world behavioral health homes demonstrated limited success in improving outcomes, such as glycemic control (13, 16, 19, 20). Prior literature has reported that mental health staff express uncertainty of what evidence-based CVD risk factor care is, as many come from a clinical background in mental health and not CVD risk factor care (13, 21). The Medicaid health home waiver program provided financing option but did not provide knowledge and system-based resources for to put CVD risk factor care and care coordination into practice. For example, community mental sites have reported lack structured protocols to directly address CVD risk factor care (13, 21). In addition, a majority of mental health staff perceive that improving physical health should be part of their organization’s mission. However, the day to day work of delivering CVD risk factor falls to individual nurse care managers who often operate as a solo provider and not wider teams within the organizations (13, 21).

Team-based strategies have been successful in improving guideline-concordant care delivery in complex medical settings, such as reducing hospital-acquired infections within intensive care units, increasing uptake of preventive cancer screenings, and increasing vaccine uptake in outpatient clinics (22–25). These teams were comprised of individuals who were involved in all aspects of patient care, from the front line administrative assistants, to medical assistants, to social workers, and to nurses and physicians. Yet less is known about levering multi-disciplinary teams within in specialty mental health settings to improve physical health. In specialty mental health settings, patient care includes care managers, social workers, rehabilitation therapists, nurses, and psychiatrists (26, 26). This study sought to address CVD risk reduction in specialty mental health settings using a team-based quality improvement approach. Specifically, we pilot tested an adapted team-based implementation strategy, the Comprehensive Unit-Based Safety Program (CUSP) to support delivery of evidence-based care for diabetes, hypertension, and dyslipidemia. These three CVD risk factors were chosen as they have overlapping recommendations for lifestyle changes and medication management.

2 Methods

2.1 Study design

We conducted a single arm pilot study of a quality improvement (a QI) implementation strategy using a pre/post observational design between February 2021and April 2023. The Institutional Review Board at Johns Hopkins University approved the protocol. All site staff participants provided written/oral consent after receiving a complete description of the study. Client data was collected under a waiver of documentation of consent. The trial was registered at clinicaltrials.gov (#NCT04696653). No deviations from the protocol occurred.

2.2 Study setting

We enrolled four psychiatric rehabilitation programs who have implemented Medicaid-supported behavioral health home programs in Maryland. Psychiatric rehabilitation programs serve clients with significant functional impairment because of their mental illness and provide outpatient mental health and psychosocial services, including vocational training, employment, and mental health case management. These programs focus on helping clients work towards self-identified goals and to live more independently within the community (27). Staff at psychiatric rehabilitation programs included case managers, rehabilitation therapists, social workers, front-line administrative assistants, and nurses. Often, the case managers will liaise with psychiatrists. Within Maryland, psychiatrists refer patients to the psychiatric rehabilitation programs but are not directly involved in the day-to-day activities at the programs. Health home services for care coordination and care management for physical health conditions are billed on a per member, per month basis and are in addition to psychiatric rehabilitation services. Health homes may employ a nurse care manager (21).

2.3 Implementation strategy and evidence bundle

2.3.1 Comprehensive unit-based safety program implementation strategy

This study used the Comprehensive Unit Based Safety Program (CUSP) implementation strategy (28) to directly improve how programs systematically deliver evidence-based CVD risk factor care, a previously identified gap within behavioral health home programs (13). CUSP teams used an evidence bundle to identify what actions they were (or were not) doing for delivery of guideline-concordant care. Description of the evidence-bundle is described below.

The goal of CUSP is to provide healthcare teams with a structured process and toolkit to identify and to address barriers that interfere with their ability to delivery evidence-based care and to improve how they work together. It also seeks to foster a culture of continuous quality improvement (QI) and provider self-efficacy within inpatient and outpatient QI work (23, 28, 29). In this study, training was delivered at the site level and included provider training, external expert facilitation, and CUSP’s quality improvement (QI) process (30). CUSP’s core steps are: 1) all staff receive training in the science of QI; 2) CUSP teams create processes to identify barriers to evidence-based care; 3) CUSP teams create and implement processes for engaging senior leadership; 4) CUSP teams address barriers; and 5) CUSP teams work to improve teamwork and communication (28, 30). Steps 2-5 are conducted in parallel.

In this study, core CUSP components were implemented in alignment with the model. Teams identified barriers that presented challenge to their organization in delivering CVD risk factor care, key leaders within their organization, and potential strategies to move the project forward (30). CUSP materials were adapted to the mental health setting to align with existing organizational culture, workflow, and roles and responsibilities. For example, a CUSP tool for the inpatient setting (“learning from defects”) asks teams to identify causes and potential solutions for “defects” that can lead to adverse patient safety events (30). Language was adapted to “learning from challenges” to reflect anticipated barriers in the outpatient setting.

2.3.2 Evidence bundle

The intervention focused on hypertension, diabetes, and dyslipidemia in populations with SMI as these CVD factors share overlapping care processes. Compared with other CVD risk factors, hypertension, diabetes, and dyslipidemia have overlapping lifestyle counseling and medication management recommendations. Medication management is a key area where care coordination is needed across specialty mental health and primary care (12, 13). Care coordination was emphasized as it fit a team-based intervention and is a Medicaid-reimbursed service in behavioral health homes. Some cardiovascular risk factors, such as tobacco use, were not included as counseling approach (e.g. identifying triggers for smoking, coping strategies to avoid cigarette use) are unique to tobacco use. Moreover, medications for smoking cessation (e.g. varenicline) are distinct from medications used for hypertension, diabetes, and dyslipidemia.

An evidence bundle was assembled to outline what constitutes guideline-concordant care for hypertension, diabetes, and dyslipidemia. Prior CUSP interventions have used an evidence bundle to guide CUSP teams who may not have known the guidelines (23, 28, 29). Assembly of the bundle was conducted using the TRiP framework, which outlines how to synthesize evidence and then implement clinical practice (Appendix) (31).

The bundle summarized evidence-based screening practices and management target goals according to the American College of Cardiology/American Heart Association (ACC/AHA), and American Diabetes Association (ADA) (9, 11, 32). It also included best practice guidelines for populations with SMI, according to the American Psychiatric Association and literature, such as screening for metabolic sequalae of antipsychotic medications and caring for individuals with cognitive dysfunction (33–35). The bundle included best practices for population health, care management, and care coordination as cross-cutting care processes according to the Agency for Healthcare Research and Quality (AHRQ) and Patient Centered Medical Home criteria (36, 37). Population health identifies and track changes to a group’s health over time (38). Care management addresses an individual patient’s health needs, while care coordination aligns health services with the individual’s needs (37). Bundle assembly was led by two general internists, a nurse with experience in caring for patients with SMI, and an expert in motivational interviewing, all part of the study team. It was reviewed for clinical accuracy and feedback by three clinical experts (external members of study team) and for acceptability and feedback by two health home nurse care managers (external members of study team).

2.3.3 Study phases

We implemented CUSP in three phases: pre-implementation (2-months), implementation (12-months), and sustainment (3-months), for a total of 17 months at each site (Table A1) (30). Each site formed an interdisciplinary CUSP team with staff from the psychiatric rehabilitation programs and behavioral health homes.

In the 2-month pre-implementation phase, CUSP teams received training in advanced CUSP processes (four 1-hour sessions), evidence-based care management and coordination for CVD risk factors (two 2-hour sessions), motivational interviewing (two 2-hour sessions), and sustaining CUSP teams (one 1-hour session). During training, CUSP team members received the evidence bundle by both digital and print format. The training reflected the content of the evidence bundle. The motivational interview training focused on how to engage clients with their own care needs for CVD risk factors (e.g. self-management strategies for diabetes). It was tailored to staff working in specialty mental health clinics (39). Staff (CUSP and non-CUSP team members) filled out pre-implementation surveys. CUSP members then identified initial implementation barriers and reviewed pre-implementation survey data.

During the 12-month implementation phase, CUSP teams implemented the CUSP process. They met monthly as a team and with a CUSP expert facilitator from the study team. CUSP teams held a training on QI (1 hour duration) for staff. Sites selected which CVD risk factors to address, which barriers and how many potential solutions or practice changes to address at any given time based on their site’s available resources. They also identified which clients had diagnosis of hypertension, diabetes, or dyslipidemia (e.g. size of population with a CVD risk factor), what clinical data they had or needed to obtain on these clients’ CVD risk factor, and then who they needed to coordinate care with for a client (e.g. primary care physician). They then implemented activities to address self-identified barriers to evidence-based CVD risk factor coordination.

Finally, in the 3-month sustainment phase, the CUSP teams continued to address self-identified barriers but did not meet with the facilitator.

2.4 Measures

Our primary outcomes focused on changes in a) QI culture at the organizational level and b) provider self-efficacy to delivery CVD care coordination during the 12-month implementation phase. Our secondary outcomes focused on changes in c) acceptability, appropriateness, and feasibility of CUSP implementation strategy and use of the evidence bundle for CVD risk factor and d) the percentages receipt of guideline-concordant care for hypertension, diabetes, and dyslipidemia for clients enrolled in behavioral health homes. Data was collected during the pre-implementation phase and end of the implementation phase.

2.4.1 Quality improvement culture

We used a modified Survey on Patient Safety, a validated thirty-four item instrument that measures QI culture across nine domains (40). It examines teamwork, supervisor promotion of QI work, organizational learning, management support for QI, feedback and communication, communication openness, reporting mistakes, staffing capacity, and overall capacity for QI. On a 5-point Likert scale, higher scores indicate organizations that are more conducive to QI activities. All staff (CUSP team members and non-CUSP team members) were asked to complete this survey. As CUSP team members received a longer training session on QI, responses were stratified by CUSP team membership status.

2.4.2 Provider self-efficacy

We used a modified Compeau and Higgins’ scale, on a nine-item instrument, to measure provider self-efficacy to coordinate evidence-based care for hypertension, hyperlipidemia, or diabetes (41). On a 10-point Likert scale, higher scores indicate greater self-efficacy. As only CUSP team members were given training on hypertension, hyperlipidemia, and diabetes, only CUSP team members were asked to complete this survey.

2.4.3 Acceptability, appropriateness, feasibility

We used the following validated survey measures: Acceptability of Intervention Measure (AIM), Intervention Appropriateness Measure (IAM), and Feasibility of Intervention Measure (FIM) (42). Each measure includes four items using a 5-point Likert scale, with higher scores signifying greater acceptability, appropriateness, or feasibility. Only CUSP team members were asked to complete these surveys on the CUSP strategy and use of the evidence bundle.

2.4.4 Evidence-based care for CVD risk factors

Finally, we measured the percentage of individuals who reached the recommended evidence-based targets for each CVD risk factor as recommended by the U,S. guidelines from the ACC/AHA and ADA (9, 11, 32). The ACC/AHA recommend a target blood pressure of less than 130/80 mm Hg (strict goal) or less than 140/90 mm Hg (more lenient goal) among individuals with hypertension. The ADA recommends a target hemoglobin A1c of less than 7% (strict goal) or hemoglobin A1c of less than 8% (lenient goal) among individuals with diabetes. Finally, the ACC/AHA recommends that individuals be prescribed a statin drug as a first-line medication for treatment of hyperlipidemia and a target low density lipoprotein (LDL) <100 mg/dL (strict goal) or <130 mg/dL (lenient goal).

2.5 Data collection

Survey data was collected using REDCap. Staff and CUSP team members were surveyed during the pre-implementation phase and at the end of the 12-month implementation phase. Sites self-reported client demographics and presence of CVD risk factor from records, and if had capability, measured blood pressure. One site had access to existing health system data; all other sites requested notes and laboratory data from external primary care clinics. Consistent with population health practice (38), sites tracked clients’ blood pressure, hemoglobin A1c, and lipid panel results across the implementation phase. Client data was provided to the study team during the transition from the pre-implementation phase to implementation phase as pre-implementation data and data available at the end of the 12-month implementation phase was provided as post-implementation data.

2.6 Statistical analysis

We used mixed-effect repeated-measure regression models to measure changes in outcomes that used Likert scales (QI culture and provider self-efficacy, and perceptions of acceptability, appropriateness, feasibility) and generalized estimating equations (GEE) to measure changes in probability of clients receiving guideline-concordant care. For both approaches, the pre-post comparisons were evaluated with the corresponding 95% confidence intervals (CIs) derived while adjusted for study sites. The QI culture outcome model also included an additional binary covariate indicating participation on the CUSP Team (Yes/No). For the mixed-effects modeling, an unstructured variance-covariance model was used for the longitudinally measured outcomes to allow different outcome variance at different timepoints and address the within-subject (staff) correlation between the repeated outcomes across timepoints. For GEE analyses, an exchangeable working correlation was used, and the sandwich-based robust estimates were used for statistical inferences. Available data for all enrolled participants was included in these modeling-based analyses, including those with only pre or post data (e.g., due to staff turnover).

To explore for site specific effects, mixed-effects models included additional cross-product interaction terms between study site variables and the timepoint variable, all as fixed effects, were used (43). For the exploratory analyses using the GEE approach employing visit by site interactions, the sites with identical outcome responses (e.g., 100% yes) pre or post intervention within site were excluded due to numerical challenge of model convergence. These sites were noted and described in the results with descriptive statistics. As a sensitivity analysis, overall effect estimates were also modeled using only data from sites that could be included in the model for site-specific effect estimation. We also reported effect size based on Cohen’s d statistic, with interpretation of d in 0.00-0.19 as very small; 0.20-0.49 as small; 0.50-0.79 as medium; and d ≥ 0.80 as large effect (43).

3 Results

3.1 Study sites, staff, and client participants

Participating sites provided virtual care or hybrid of virtual and in-person care in the first six months of the study. As COVID-19 pandemic restrictions evolved, one site transitioned to to fully in-person care, one site remained fully virtual, and two sites delivered care in a hybrid format (Table 1).

Table 1

Table 1. Characteristics of participating sites, staff, and client populations. All data is pre-implementation unless otherwise noted.

Across all sites, 85 staff members participated, of which 19 were a part of a CUSP team and 66 were not a part of a CUSP team. Sites had 16-26 staff members, and CUSP teams comprised of 3-6 members. Staff were predominately female-identifying (74%), white race (72%), college-educated (91%), and employed full-time (91%) (Table 1, Appendix). Staff worked as mental health coordinators and rehabilitation therapists (57%) or as front-line providers, such as employment specialists or residential program staff (14%). Three out of four sites had nurses who were affiliated with the behavioral health home and participated on CUSP teams. Site directors or managers participated in CUSP teams at all sites.

The pilot included 442 clients, and individual sites served 83-149 clients (Table 1, Appendix). Sites reported their client population had a mean age 47.5 years (SD=14.2 years), were 53% male, 51% white race and 45% Black race. Approximately 45% of clients were diagnosed with schizophrenia, 25% had bipolar disorder, and 29% had major depressive disorder, with the prevalence of each mental health diagnoses varying across sites. Approximately 33% of clients had a history of alcohol or substance use disorder, and 68% of clients received disability. At baseline. At pre-implementation, there were 44% of clients with a history of hypertension, of whom 70% had uncontrolled hypertension. Of the 29% of clients with a history of diabetes, 42% had poor glycemic control. Of the 26% of clients with a history of dyslipidemia, 21% had elevated cholesterol levels. In addition, 76% clients were overweight or obese (body mass index>25) and 43% of clients smoked tobacco.

3.2 Quality improvement culture

CUSP and non-CUSP staff reported on how they perceived organizational QI culture between pre-implementation (N=84) and post-implementation (N=64). Staff perceived no difference across time (3.8 [SD=0.7] vs. 3.8 [SD=0.6]; estimate change 0.0 [95% CI: -0.2-0.1]. The effect size was very small, measured by Cohen’s d, was d=-0.05 (Table 2). Responses were similar between CUSP and non-CUSP team members and across sites. In addition, no differences across the 12-month measurement period were observed when we looked at sub-measures of quality improvement culture (Table A4). Lowest ratings were given to the categories of staffing capacity (2.9 [SD=0.9] vs. 2.8 [SD=2.9; estimate change -0.2 [95% CI: -0.4- 0.0]; d=-0.19) and openness of communication (3.5 [SD=0.8] vs. 3.5 [SD=0.8]; estimate change 0.1 [95% CI: -0.1-0.3]; d=0.11).

Table 2

Table 2. Summary of changes across the 12-month implementation period for primary outcomes: quality improvement culture and perceived provider self-efficacy for cardiovascular disease risk factor coordination.

3.3 Provider self-efficacy

CUSP team members reported on perceived self-efficacy for coordinating for each CVD risk factor from pre-implementation (N=22) to post-implementation (N=19). Perceived self-efficacy increased significantly for two topics: hypertension (6.0 [SD=2.3] vs. 7.1 [SD=1.2]; estimate change 1.0; (95% CI: 0.2-1.8); d=0.44, small effect) and diabetes (6.2 [SD=2.6] vs. 7.3 [SD=1.4]; estimate change 1.0 [95% CI: 0.1-2.0]; d=0.4, small effect). Self-efficacy also increased for dyslipidemia management but did not reach statistical significance (5.9 [SD=2.4] vs. 7.0 [SD=1.4]; estimate 1.1 [95% CI: 0.0-2.1]; d=0.44, small effect). Self-efficacy scores to coordinate care were driven by large effect sizes at Sites 1 and 3, as measured by Cohen’s d (Table A3).

3.4 Acceptability, Appropriateness, and Feasibility

CUSP team members also reported pre-implementation (N=19) and post-implementation (N=19) on the acceptability, appropriateness, and feasibility of the CUSP implementation strategy and evidence-bundle outlining CVD risk factor care guidelines (Table 3). Scores for the acceptability of the CUSP implementation strategy did not change significantly over time (4.4 [SD=0.5] vs. 4.2 [SD=0.6]; estimate change -0.2 [95% CI: -0.6-0.1], d= -0.49). Similar scores were observed for appropriateness and feasibility of the CUSP strategy. Scores for acceptability of the evidence bundle also did not change significantly across time (4.4 [SD=0.5] vs. 4.2 [SD=0.5]; estimate change -0.3 [95% CI: -0.6-0.0], d= -0.59). Scores were similar for appropriateness and feasibility of the evidence bundle (Table 3).

Table 3

Table 3. Summary of outcomes of acceptability, appropriateness, and feasibility of the CUSP implementation strategy and evidence bundle to improve delivery of cardiovascular risk factor care coordination.

3.5 Receipt of evidence-based care for CVD risk factors

Hypertension: Across all sites, the likelihood of having blood pressure strictly controlled at <130/80 mm Hg did not change over time (OR 1.4, 95% CI: 0.9-2.2) (Table 4). Clients at site 1 were more likely to have achieved a more lenient goal of blood pressure < 140/90 mm Hg after the intervention (OR 3.0, 95% CI: 1.2-7.8); however, this was not observed at other sites (Table A7). In addition, no difference was seen over time in the percentage of the client population with an available blood pressure in the prior 12-months.

Table 4

Table 4. Number and percentage of clients with hypertension, diabetes, and dyslipidemia with available clinical data and those who have reached the recommended target for blood pressure control, glycemic control as measured by hemoglobin A1c, and cholesterol control across the 12-month implementation stage (pre-implementation vs. post).

Diabetes: Across all sites, we observed no change in the likelihood of clients having achieved strict glycemic control (Hemoglobin A1c < 7%) (OR 1.2; 95% CI: 0.7-2.2) or more lenient glycemic control (Hemoglobin A1c < 8%) (OR 1.2; 95% CI: 0.7-2.2) (Table A7). There was also no difference in the percentage of clients who had a hemoglobin A1c value available in the prior 12-months.

Dyslipidemia: Across all sites, clients were more likely to have an LDL-C value of < 100 mg/dl after the intervention (OR 1.4; 95% CI: 1.1-1.9) (Table 4). However, only clients at Site 1 had an increased likelihood in reaching this cholesterol measure (OR 1.8; 95% CI-1.1-2.9) (Table A7). Notable, Sites 2 and 4 had available cholesterol laboratory data for 54-68% of their client population.

3.6 Activities of CUSP team

CUSP teams held monthly meetings at least 80% of the time during implementation phase (Table 5), but only two sites held formal meetings during the sustainment phase (when external facilitator was not present). All sites addressed at least one CVD risk factor. The three sites that delivered in-person care addressed hypertension.

Table 5

Table 5. Activities of CUSP team during the 12-month implementation and 3-month sustainment phases.

All CUSP teams sought staff-wide feedback about how to improve CVD risk factor coordination during the implementation phase, and two teams conducted an additional survey during the sustainment phase. Site 1 opened meetings to all staff and received feedback on CUSP activities from direct care staff. Three teams also delivered CVD risk factor education for staff and/or clients. Site 3 implemented this staff-wide training twice yearly, while Site 2 sponsored staff to become certified in tobacco cessation counseling.

Finally, CUSP teams created processes to improve intra-organization communication and to build staff capacity for coordination for CVD risk factor care. For example, direct care staff did not have a standardized process to communicate when clients had medical appointments to the nurse. Three teams implemented a shared electronic calendar of client appointments between staff and the nurse(s). This process took three to eight months and required buy-in from administrators for technology resources, training, and workflow changes.

Sites also identified differential access to clients’ health data, and this lack of information was a barrier that they sought to address. They felt that they did not know which clients were receiving guideline-concordant care and which clients to prioritize for CVD risk factor coordination. Site 1 had access to electronic health data for CVD risk factors as many of their clients saw primary care physicians within a shared electronic health record. However, they did not have a system to obtain the data. Site 4 had no access to existing laboratory data or a system to know when clients had last seen a primary care physician. Thus, these two sites implemented a new standardized workflow for direct care staff to obtain laboratory data and health records from primary care physicians and specialists. This was incorporated as part of client intake and meetings with case managers. Site 4 also implemented a process where clients could grant permission for their mental health team to access results directly from an external clinical laboratory network. This process allowed the nurse to monitor chronic disease measures (e.g. hemoglobin A1c). Both sites then planned to use the information to coordinate appointments for clients who had not seen a primary care physician in over a year or who had a CVD risk factor that was poorly controlled.

4 Discussion

In this pilot, four community mental health behavioral health home programs that served populations with SMI implemented a team-based quality improvement strategy focused on CVD risk factor care quality. We observed a medium effect increase in provider self-efficacy score to coordinate CVD risk factor care but no significant change in the organizations’ QI culture. In secondary outcomes, use of an evidence bundle and the team-based QI strategy had high scores for acceptability, appropriateness, and feasibility at the start and end of the implementation, phase, but no interval change was observed.

This pilot study’s findings reflect the challenges of implementation within specialty mental health settings, addressing provider and organizational behavior and culture within a 12-month time frame and the complexity of the intervention. All sites served a high-need, vulnerable population with SMI. The client population had high prevalence of disability (likely because of impairment from mental illness) who had been referred by their psychiatrist for vocational training and community-support services. While the study team did not collect data on socioeconomic status, clients likely had low income as they were enrolled in public Medicaid insurance (health home programs were supported by Medicaid). Moreover, our findings echo prior literature that Medicaid-funded health home programs have struggled with implementing CVD risk factor management due to lack of knowledge and resources (23, 28, 29). None of the sites had existing population-based approach for managing clients’ CVD risk factors where they knew which groups of clients had which CVD risk factors. No sites also had pre-existing resources to identify whether clients had achieved CVD risk factor control. Rather, they relied on addressing CVD risk factors on an ad hoc basis and if an individual client came to the nurse with a question. We also observed the limited resources and staff turnover at these organizations, which further suggested why organizations struggled to deliver services.

In the intervention, CUSP teams were charged with developing and initiating new processes for coordination of CVD risk factor care for their client population with SMI. By doing so, they started shifting CVD risk factor care from the sole responsibility of a nurse to a team-based care model. Three out of four sites focused on improving how teams worked together and highlighting infrastructure barriers. The prioritized addressing foundational team processes (intra-team communication, obtaining client population data, education) that are needed to support CVD risk factor care. These site-identified barriers mirror the barriers that have been reported in other behavioral health home programs (13). This suggests that community mental health sites need time and resources to put key infrastructure and team processes in place before they can successfully focus on improving specific CVD risk factor care processes.

Our study is the first to pilot CUSP in specialty mental health programs and to use CUSP for a topic (for CVD risk factors) not routinely addressed by providers in these settings. CUSP implementation strategy was chosen as it fit into a team-based care model, which is the underlying model at psychiatric rehabilitation programs. This may help to explain why sites gave high ratings to the acceptability, appropriateness, and feasibility of the CUSP implementation strategy. Prior CUSP implementation projects have occurred in acute care or primary care and were focused on clinical measures that were within the team’s existing scope of practice (e.g., prevention of line-associated infections in acute care) (23, 28, 29). Prior to this study, no behavioral health home site had previously focused on QI, and improvement in QI culture – which exists at the unit, department, and system levels and is often referred to as “the way we do things here” (44) - is expected over two to three years (45–47). It takes time for culture change to occur and requires administrative support (23). Our results are consistent with this literature. While there is a natural inclination to look for change in any intervention study, this data provides meaningful baseline information about the quality improvement culture at community mental health sites for future studies. In addition, no behavioral health home site had previously focused on QI. In addition, the composite score and responses to individual QI survey questions provide insight about specific areas of strengths and weaknesses (48).

Importantly, our findings reflect similar organizational drivers that have been described in implementation of behavioral health homes and the wider implementation science literature. The sites that saw the greatest increase in self-efficacy to coordinate CVD risk factors had regular support/buy-in from leadership (Sites 1 and 3), greater access to technology (Site 1), and less staff turnover (Sites 1 and 3).

One possible organizational driver is the need for buy-in from organizational members, from top to bottom. CUSP focuses on engagement of staff and senior leadership because leadership engagement is critical for successful implementation and sustainment (30, 49). Each site was asked to have at least one director or manager on the CUSP team. However, only Site 1 had a director regularly attend CUSP meetings. Site 3 had regular check-ins with site leadership and a manager served as the CUSP team lead. Other sites had managers with differing levels of decision-making power. In practice, this translated to teams waiting for administrative approval and delays in implementing activities, such as the shared calendar. Future efforts to build upon this work will need continued support from leadership to address workflow processes and organizational culture. Moreover, CUSP is designed to be a longitudinal strategy for healthcare settings. Yet only half the sites held CUSP meetings during the 3-month sustainment period. The study team provided facilitation support for CUSP strategy during the 12-month intervention period but not during the sustainment period. Future work may be needed on whether additional facilitation support and what type of support is needed to sustain CUSP teams within mental health settings. The study team also did not offer additional incentives for staff to participate in the CUSP team and it was up to CUSP teams to incentivize staff participation for specific activities. Part of the CUSP team’s role is to identify how to engage their colleagues and address practice changes based on their site’s available resources. We did not collect specific data on whether sites offered incentives or not, and future studies may wish to examine how successful mental health sites engaged with staff and leadership. This information will be helpful to understand how to sustain CUSP teams in community mental health settings.

Sites also had different available resources, which may have influenced implementation and sustainability. For example, sites differed on technology, which is a known facilitator (or barrier) to implementation work (13, 49). Site 1 had access to a shared electronic health record with primary care clinicians but at the start of the study, did not know which laboratory measures were relevant to CVD risk factor care. Sites 2, 3, and 4 did not have a shared electronic medical record with primary care clinics and therefore had to request clinical data for their patient populations by fax, email, and phone. This resulted in multiple outreach efforts to primary care offices and greater frustration when no communication was received back. For community mental health sites to be able to conduct population health management and to help coordinate care for populations with SMI, whom they serve, they need accessible data.

In addition, the staff composition, responsibilities, relationships, and morale may have differed across sites, and these internal personnel factors influence success or failure of implementation projects (49). Site 3 was the only site that did not have a nurse onsite and likely a steeper learning curve to implement care processes for CVD risk factors. However, the site team lead was highly engaged, had a manager role and used the evidence-bundle to guide the team. This may also help to explain why the self-efficacy scores rose at Site 3. Sites were asked to have representative staff as part of the CUSP team from across their organization. All four sites had front-line staff and staff who had been with the organization of different lengths of time. Sites with full-time behavioral health home staff (sites 1 and 3) were more successful at obtaining complete client-level data by end of study. This may reflect differential organizational working conditions and structure (50). However, our data did not capture the quality of relationships among staff. Moreover, high staff turnover may have negatively impacted organization culture and ability for teams to implement durable change. Turnover increases individual workloads, slows momentum, and may make process changes harder to sustain (51).

We also gathered data on the implementation outcomes of acceptability, feasibility, and appropriateness and availability and measurement of CVD risk factor clinical data. The relatively high scores at the start and at the end of the study suggest that providers viewed the intervention and QI implementation strategy as a good fit when adopting the study and after implementing it over twelve months (52). While we did not have data on psychiatric medications prescribed to clients across all sites, it is likely that the CUSP teams knew that their clients were prescribed antipsychotic medications, were receiving blood draws, but did not have access to the laboratory values. Therefore, their perception of this intervention is likely consistent with their organizational mission in caring for populations with SMI bu. Consistency with organizational mission is a known facilitator in implementation of physical healthcare delivery within specialty mental health settings (13). However, it is unlikely to have impacted our study’s results.

Our study has several limitations. First, it was conducted in a single state within a small number of psychiatric rehabilitation programs, which may limit its generalizability. Sites also had to agree to participate in a 12-month organizational-wide intervention that focused on addressing physical health in their clients. Sites had 3-6 staff members participate in the CUSP teams, which limited our ability to detect differences over time. No psychiatrists were involved at any site. In Maryland where our study was conducted, psychiatrists refer patients with SMI to psychiatric rehabilitation programs for community services and vocational training. These programs may not be affiliated with their psychiatric clinic. Given the role of psychiatrists in selecting antipsychotic medications that influence CVD risk, future studies may wish to actively include psychiatrists in CUSP team activities.

Second, this study was conducted during the COVID-19 pandemic, which led to virtual meetings for staff and clients. Virtual interactions may also have impacted staff morale and organizational culture. In the second year of the pandemic, sites had differing capacities about resources and time to devote to a complex pilot intervention. Our pre-specified data collection plan in the protocol also did not include collecting detailed data on, so sites’ data did not capture detailed information about sites’ activities. Sites did not have pre-existing services for lifestyle counseling, such as diet and physical activity. We did not collect data on how often obesity management strategies (e.g. diet/exercise counseling, starting Metformin or other adjunctive medication) were offered or received. We also did not collect data on other care coordination process measures, such as referrals or appointments scheduled, attendance rate). This type of data would be helpful in future studies. Future implementation work for this pilot also will need to consider how sites deliver care, as all sites delivered telehealth care with some staff working remotely and its impact on organizational culture and care delivery (53). Third, we had approximately 25% staff turnover across the sites. However, this level of turnover is not dissimilar to what has been reported in the literature prior to the pandemic (54).

Finally, it is likely that this pilot study lacked adequate power to detect large changes in primary outcomes. Primary outcomes were chosen based on prior CUSP studies and reflected the purpose of CUSP to address the quality of care delivered within a given organization (28). Interpretations of QI culture may be also limited by ceiling effect as responses started between 3 or 4 (out of 5). Moreover, by measuring QI culture and its components over time during the pilot study, the study team and participating sites were able to study the approach of implementing CUSP and to identify specific areas of QI culture to target in the future: staff turnover and openness of communication.

Overall, our pilot study observed improvement in self-efficacy to manage CVD risk factor coordination but no significant change in organizational quality improvement culture. The four mental health sites implemented and used the CUSP strategy to focus on improving infrastructure and foundational team processes and focused less on a individual CVD risk factor care processes. Our study suggests that mental health organizations need administrative and personnel support to conduct quality improvement work, a new direction for many, and an information technology infrastructure to conduct population health for CVD risk factor care among populations with SMI. Caring for populations with SMI requires teams of clinicians within mental and physical health settings. This pilot study offers a novel strategy to deliver complex care coordination for CVD risk factors to a marginalized population with SMI, but much work remains to improve quality improvement culture and self-efficacy for CVD risk factor coordination among mental health providers.

Data availability statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics statement

The studies involving humans were approved by Johns Hopkins University School of Medicine. The studies were conducted in accordance with the local legislation and institutional requirements. The participants provided their written informed consent to participate in this study.

Author contributions

KM: Formal analysis, Investigation, Writing – original draft, Writing – review & editing. JG: Data curation, Investigation, Project administration, Writing – review & editing. AD: Investigation, Writing – review & editing. CC: Data curation, Formal analysis, Methodology, Writing – review & editing. SG: Investigation, Writing – review & editing. TF: Formal analysis, Writing – review & editing. GD: Conceptualization, Funding acquisition, Investigation, Supervision, Writing – review & editing. DT: Conceptualization, Investigation, Supervision, Writing – review & editing. EM: Conceptualization, Supervision, Writing – original draft, Writing – review & editing. NW: Formal analysis, Writing – review & editing.

Funding

The author(s) declare that financial support was received for the research, authorship, and/or publication of this article. This work was supported by NIMH P50 MH115842 (PI: Daumit) and NHBLI grant UG3154280 (PI: Daumit).

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpsyt.2025.1446985/full#supplementary-material

References

1. Colton CW, Manderscheid RW. Congruencies in increased mortality rates, years of potential life lost, and causes of death among public mental health clients in eight states. Preventing Chronic Dis. (2006) 3:A42.

PubMed Abstract | Google Scholar

2. Olfson M, Gerhard T, Huang C, Crystal S, Stroup TS. Premature mortality among adults with schizophrenia in the United States. JAMA Psychiatry. (2015) 72:1172–81. doi: 10.1001/jamapsychiatry.2015.1737

PubMed Abstract | Crossref Full Text | Google Scholar

3. Osborn DP, Wright CA, Levy G, King MB, Deo R, Nazareth I. Relative risk of diabetes, dyslipidaemia, hypertension and the metabolic syndrome in people with severe mental illnesses: systematic review and metaanalysis. BMC Psychiatry. (2008) 8:84. doi: 10.1186/1471-244x-8-84

PubMed Abstract | Crossref Full Text | Google Scholar

4. Dixon L, Weiden P, Delahanty J, Goldberg R, Postrado L, Lucksted A, et al. Prevalence and correlates of diabetes in national schizophrenia samples. Schizophr Bull. (2000) 26:903–12. doi: 10.1093/oxfordjournals.schbul.a033504

PubMed Abstract | Crossref Full Text | Google Scholar

5. McGinty EE, Baller J, Azrin ST, Juliano-Bult D, Daumit GL. Quality of medical care for persons with serious mental illness: A comprehensive review. Schizophr Res. (2015) 165:227–35. doi: 10.1016/j.schres.2015.04.010

PubMed Abstract | Crossref Full Text | Google Scholar

6. Cather C, Pachas GN, Cieslak KM, Evins AE. Achieving smoking cessation in individuals with schizophrenia: special considerations. CNS Drugs. (2017) 31:471–81. doi: 10.1007/s40263-017-0438-8

PubMed Abstract | Crossref Full Text | Google Scholar

7. De Hert M, Detraux J, van Winkel R, Yu W, Correll CU. Metabolic and cardiovascular adverse effects associated with antipsychotic drugs. Nat Rev Endocrinol. (2011) 8:114–26. doi: 10.1038/nrendo.2011.156

PubMed Abstract | Crossref Full Text | Google Scholar

8. Tschoner A, Engl J, Laimer M, Kaser S, Rettenbacher M, Fleischhacker WW, et al. Metabolic side effects of antipsychotic medication. Int J Clin Pract. (2007) 61:1356–70. doi: 10.1111/j.1742-1241.2007.01416.x

PubMed Abstract | Crossref Full Text | Google Scholar

9. Arnett DK, Blumenthal RS, Albert MA, Buroker AB, Goldberger ZD, Hahn EJ, et al. ACC/AHA guideline on the primary prevention of cardiovascular disease: executive summary. Circulation. (2019) :Cir0000000000000677. doi: 10.1161/cir.0000000000000677

PubMed Abstract | Crossref Full Text | Google Scholar

10. American Diabetes Association Professional Practice Committee Prevention or delay of diabetes and associated comorbidities: standards of care in diabetes-2024. Diabetes Care. (2024) 47:S43–s51. doi: 10.2337/dc24-S003

PubMed Abstract | Crossref Full Text | Google Scholar

11. Whelton PK, Carey RM, Aronow WS, Casey DE Jr, Collins KJ, Dennison Himmelfarb C, et al. 2017 ACC/AHA/AAPA/ABC/ACPM/AGS/APhA/ASH/ASPC/NMA/PCNA guideline for the prevention, detection, evaluation, and management of high blood pressure in adults: A report of the american college of cardiology/american heart association task force on clinical practice guidelines. J Am Coll Cardiol. (2018) 71:e127–248. doi: 10.1016/j.jacc.2017.11.006

PubMed Abstract | Crossref Full Text | Google Scholar

12. Horvitz-Lennon M, Kilbourne AM, Pincus HA. From silos to bridges: meeting the general health care needs of adults with severe mental illnesses. Health Affairs (Project Hope). (2006) 25:659–69. doi: 10.1377/hlthaff.25.3.659

PubMed Abstract | Crossref Full Text | Google Scholar

13. Murphy KA, Daumit GL, Stone E, McGinty EE. Physical health outcomes and implementation of behavioural health homes: a comprehensive review. Int Rev Psychiatry (Abingdon England). (2019) 6:1–18. doi: 10.1080/09540261.2018.1555153

PubMed Abstract | Crossref Full Text | Google Scholar

14. Swietek KE, Domino ME, Beadles C, Ellis AR, Farley JF, Grove LR, et al. Do medical homes improve quality of care for persons with multiple chronic conditions? Health Serv Res Dec. (2018) 53:4667–81. doi: 10.1111/1475-6773.13024

PubMed Abstract | Crossref Full Text | Google Scholar

15. Medicaid Health Homes: SPA Overview (Centers for Medicare and Medicaid Services). Medicaid Health Homes: SPA Overview. Baltimore, MD: Centers for Medicare and Medicaid Services (2019).

Google Scholar

16. Druss BG, von Esenwein SA, Glick GE, Deubler E, Lally C, Ward MC, et al. Randomized trial of an integrated behavioral health home: the health outcomes management and evaluation (HOME) study. Am J Psychiatry. (2017) 174:246–55. doi: 10.1176/appi.ajp.2016.16050507

PubMed Abstract | Crossref Full Text | Google Scholar

17. Daumit GL, Dalcin AT, Dickerson FB, Miller ER, Evins AE, Cather C, et al. Effect of a comprehensive cardiovascular risk reduction intervention in persons with serious mental illness: A randomized clinical trial. JAMA Network Open. (2020) 3:e207247–e207247. doi: 10.1001/jamanetworkopen.2020.7247

PubMed Abstract | Crossref Full Text | Google Scholar

18. Murphy KA, Sarker E, Stuart EA, Cook C, Goldsholl S, Daumit GL. Effect of care management on cholesterol for individuals with serious mental illness: a secondary analysis of an RCT. J Gen Internal Med. (2023) 39(2):354–6. doi: 10.1007/s11606-023-08510-y

PubMed Abstract | Crossref Full Text | Google Scholar

19. Scharf DM, Eberhart NK, Hackbarth NS, Horvitz-Lennon M, Beckman R, Han B, et al. Evaluation of the SAMHSA primary and behavioral health care integration (PBHCI) grant program: final report. Rand Health Quarterly. (2014) 4:6.

PubMed Abstract | Google Scholar

20. McGinty EE, Stone EM, Kennedy-Hendricks A, Bandara S, Murphy KA, Stuart EA, et al. Effects of maryland’s affordable care act medicaid health home waiver on quality of cardiovascular care among people with serious mental illness. J Gen Internal Med Nov. (2020) 35:3148–58. doi: 10.1007/s11606-020-05690-9

PubMed Abstract | Crossref Full Text | Google Scholar

21. McGinty EE, Kennedy-Hendricks A, Linden S, Choksy S, Stone E, Daumit GL. An innovative model to coordinate healthcare and social services for people with serious mental illness: A mixed-methods case study of Maryland’s Medicaid health home program. Gen Hosp Psychiatry. (2018) 51:54–62. doi: 10.1016/j.genhosppsych.2017.12.003

PubMed Abstract | Crossref Full Text | Google Scholar

22. Pronovost PJ, Berenholtz SM, Goeschel C, Thom I, Watson SR, Holzmueller CG, et al. Improving patient safety in intensive care units in Michigan. J Crit Care. (2008) 23:207–21. doi: 10.1016/j.jcrc.2007.09.002

PubMed Abstract | Crossref Full Text | Google Scholar

23. Timmel J, Kent PS, Holzmueller CG, Paine L, Schulick RD, Pronovost PJ. Impact of the Comprehensive Unit-based Safety Program (CUSP) on safety culture in a surgical inpatient unit. Jt Comm J Qual Patient Saf. (2010) 36:252–60. doi: 10.1016/s1553-7250(10)36040-5

PubMed Abstract | Crossref Full Text | Google Scholar

24. Pitts SI, Maruthur NM, Wang X, Sawyer MD, Grimes R, Nigrin C, et al. Team-based health information exchange use increased mammography documentation and referral in an academic primary care practice: an interrupted time series. J Gen Internal Med. (2018) 33:710–4. doi: 10.1007/s11606-017-4259-8

PubMed Abstract | Crossref Full Text | Google Scholar

25. Samaan ZM, Brown CM, Morehous J, Perkins AA, Kahn RS, Mansour ME. Implementation of a preventive services bundle in academic pediatric primary care centers. Pediatrics. (2016) 137:e20143136. doi: 10.1542/peds.2014-3136

PubMed Abstract | Crossref Full Text | Google Scholar

26. Mueser KT, McGurk SR. Schizophrenia. Lancet. (2004) 363:2063–72. doi: 10.1016/s0140-6736(04)16458-1

PubMed Abstract | Crossref Full Text | Google Scholar

27. Dalcin AT, Jerome GJ, Appel LJ, Dickerson FB, Wang NY, Miller ER, et al. Need for cardiovascular risk reduction in persons with serious mental illness: design of a comprehensive intervention. Front Psychiatry. (2018) 9:786. doi: 10.3389/fpsyt.2018.00786

PubMed Abstract | Crossref Full Text | Google Scholar

28. Pronovost P, Weast B, Rosenstein B, Sexton JB, Holzmueller CG, Paine L, et al. Implementing and validating a comprehensive unit-based safety program. J Patient Safety. (2005) 1:33–40. doi: 10.1097/01209203-200503000-00008

Crossref Full Text | Google Scholar

29. Pitts SI, Maruthur NM, Luu NP, Curreri K, Grimes R, Nigrin C, et al. Implementing the comprehensive unit-based safety program (CUSP) to improve patient safety in an academic primary care practice. Jt Comm J Qual Patient Saf. (2017) 43:591–7. doi: 10.1016/j.jcjq.2017.06.006

PubMed Abstract | Crossref Full Text | Google Scholar

30. Agency for. Healthcare Research and Quality. The CUSP Method (Accessed September 7, 2023).

Google Scholar

31. Pronovost PJ, Berenholtz SM, Needham DM. Translating evidence into practice: a model for large scale knowledge translation. BMJ (Clinical Res Ed). (2008) 337:a1714. doi: 10.1136/bmj.a1714

PubMed Abstract | Crossref Full Text | Google Scholar

32. American Diabetes Association. Classification and diagnosis of diabetes: standards of medical care in diabetes-2018. Diabetes Care. (2018) 41:S13–s27. doi: 10.2337/dc18-S002

PubMed Abstract | Crossref Full Text | Google Scholar

33. Marder SR, Cannon TD. Schizophrenia. N Engl J Med. (2019) 381:1753–61. doi: 10.1056/NEJMra1808803

PubMed Abstract | Crossref Full Text | Google Scholar

34. Firth J, Siddiqi N, Koyanagi A, Siskind D, Rosenbaum S, Galletly C, et al. The Lancet Psychiatry Commission: a blueprint for protecting physical health in people with mental illness. Lancet Psychiatry. (2019) 6(8):675–712. doi: 10.1016/S2215-0366(19)30132-4

PubMed Abstract | Crossref Full Text | Google Scholar

35. American Diabetes Association, American Psychiatric Association, American Association of Clinical Endocrinologists, North American Association for the Study of Obesity. Consensus development conference on antipsychotic drugs and obesity and diabetes. J Clin Psychiatry. (2004) 65:267–72. doi: 10.4088/JCP.v65n0219

PubMed Abstract | Crossref Full Text | Google Scholar

36. Meyers D, Peikes D, Genevro J, Peterson G, Taylor EF, Lake T, et al. AHRQ publication no. 11-M005-EF the roles of patient-centered medical homes and accountable care organizations in coordinating patient care. Agency Healthcare Res Qual. (2010).

Google Scholar

37. McDonald KM, Schultz E, Albin L, Pineda N, Lonhart J, Sundaram V, et al. Care Coordination Atlas Version 4 (Prepared by Stanford University under subcontract to American Institutes for Research on Contract No. HHSA290-2010-00005I). Agency Healthcare Res Qual. (2014).

Google Scholar

38. Sieck CJ, Wickizer T, Geist L. Population health management in integrated physical and mental health care. Article Adv Health Care Management. (2014) 16:139–50. doi: 10.1108/s1474-823120140000016006

PubMed Abstract | Crossref Full Text | Google Scholar

39. Dalcin AT, Yuan CT, Jerome GJ, Goldsholl S, Minahan E, Gennusa J, et al. Designing practical motivational interviewing training for mental health practitioners implementing behavioral lifestyle interventions: protocol for 3 pilot intervention studies. JMIR Res Protoc. (2023) 12:e44830. doi: 10.2196/44830

PubMed Abstract | Crossref Full Text | Google Scholar

40. Sorra J, Gray L, Streagle S, Famolaro T, Yount N, Behm J. Pilot study: validity and reliability of the hospital survey on patient safety. Agency Healthcare Res Quality: Rockville MD USA. (2004).

Google Scholar

41. Compeau DR, Higgins CA. Computer self-efficacy: development of a measure and initial test. MIS Quarterly. (1995) 19:189–211. doi: 10.2307/249688

Crossref Full Text | Google Scholar

42. Weiner BJ, Lewis CC, Stanick C, Powell BJ, Dorsey CN, Clary AS, et al. Psychometric assessment of three newly developed implementation outcome measures. Implementation Sci. (2017) 12:108. doi: 10.1186/s13012-017-0635-3

PubMed Abstract | Crossref Full Text | Google Scholar

43. Cohen J. Statistical Power Analysis for the Behavioral Sciences. 2nd. New York: Lawrence Erlbaum Associates (1988).

Google Scholar

44. Mathews SC, Demski R, Hooper JE, Biddison LD, Berry SA, Petty BG, et al. A model for the departmental quality management infrastructure within an academic health system. Acad Med. (2017) 92:608–13. doi: 10.1097/acm.0000000000001380

PubMed Abstract | Crossref Full Text | Google Scholar

45. Sexton JB, Holzmueller CG, Pronovost PJ, Thomas EJ, McFerran S, Nunes J, et al. Variation in caregiver perceptions of teamwork climate in labor and delivery units. J Perinatol. (2006) 26:463–70. doi: 10.1038/sj.jp.7211556

PubMed Abstract | Crossref Full Text | Google Scholar

46. Pronovost P, Holzmueller CG, Needham DM, Sexton JB, Miller M, Berenholtz S, et al. How will we know patients are safer? An organization-wide approach to measuring and improving safety. Crit Care Med. (2006) 34:1988–95. doi: 10.1097/01.Ccm.0000226412.12612.B6

PubMed Abstract | Crossref Full Text | Google Scholar

47. Sexton JB, Berenholtz SM, Goeschel CA, Watson SR, Holzmueller CG, Thompson DA, et al. Assessing and improving safety climate in a large cohort of intensive care units. Crit Care Med. (2011) 39:934–9. doi: 10.1097/CCM.0b013e318206d26c

PubMed Abstract | Crossref Full Text | Google Scholar

48. Marsteller JA, Wen M, Hsu YJ, Bauer LC, Schwann NM, Young CJ, et al. Safety culture in cardiac surgical teams: data from five programs and national surgical comparison. Ann Thorac Surg Dec. (2015) 100:2182–9. doi: 10.1016/j.athoracsur.2015.05.109

PubMed Abstract | Crossref Full Text | Google Scholar

49. Keith RE, Crosson JC, O’Malley AS, Cromp D, Taylor EF. Using the Consolidated Framework for Implementation Research (CFIR) to produce actionable findings: a rapid-cycle evaluation approach to improving implementation. Implementation Sci. (2017) 12:15. doi: 10.1186/s13012-017-0550-7

PubMed Abstract | Crossref Full Text | Google Scholar

50. Hsu YJ, Marsteller JA. Who applies an intervention to influence cultural attributes in a quality improvement collaborative? J Patient Saf Mar. (2020) 16:1–6. doi: 10.1097/pts.0000000000000223

PubMed Abstract | Crossref Full Text | Google Scholar

51. Baron AN, Hemler JR, Sweeney SM, Tate Woodson T, Cuthel A, Crabtree BF, et al. Effects of practice turnover on primary care quality improvement implementation. Am J Med Qual. (2020) 35:16–22. doi: 10.1177/1062860619844001

PubMed Abstract | Crossref Full Text | Google Scholar

52. Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Administration Policy Ment Health. (2011) 38:65–76. doi: 10.1007/s10488-010-0319-7

PubMed Abstract | Crossref Full Text | Google Scholar

53. Khalid A, Dong Q, Chuluunbaatar E, Haldane V, Durrani H, Wei X. Implementation science perspectives on implementing telemedicine interventions for hypertension or diabetes management: scoping review. J Med Internet Res. (2023) 25:e42134. doi: 10.2196/42134

PubMed Abstract | Crossref Full Text | Google Scholar

54. Brabson LA, Harris JL, Lindhiem O, Herschell AD. Workforce turnover in community behavioral health agencies in the USA: A systematic review with recommendations. Clin Child Fam Psychol Rev. (2020) 23:297–315. doi: 10.1007/s10567-020-00313-5

PubMed Abstract | Crossref Full Text | Google Scholar