Evaluation of COVID-19 vaccine implementation in a large safety net health system

Objectives: To evaluate rapid COVID-19 vaccine clinic implementation from January-April 2021 in the Los Angeles County Department of Health Services (LACDHS), the second-largest US safety net health system. During initial vaccine clinic implementation, LACDHS vaccinated 59,898 outpatients, 69% of whom were Latinx (exceeding the LA County Latinx population of 46%). LACDHS is a unique safety net setting to evaluate rapid vaccine implementation due to system size, geographic breadth, language/racial/ethnic diversity, limited health staffing resources, and socioeconomic complexity of patients.

Methods: Implementation factors were assessed through semi-structured interviews of staff from all twelve LACDHS vaccine clinics from August-November 2021 using the Consolidated Framework for Implementation Research (CFIR) and themes analyzed using rapid qualitative analysis.

Results: Of 40 potential participants, 25 health professionals completed an interview (27% clinical providers/medical directors, 23% pharmacist, 15% nursing staff, and 35% other). Qualitative analysis of participant interviews yielded ten narrative themes. Implementation facilitators included bidirectional communication between system leadership and clinics, multidisciplinary leadership and operations teams, expanded use of standing orders, teamwork culture, use of active and passive communication structures, and development of patient-centered engagement strategies. Barriers to implementation included vaccine scarcity, underestimation of resources needed for patient outreach, and numerous process challenges encountered.

Conclusion: Previous studies focused on robust advance planning as a facilitator and understaffing and high staff turnover as barriers to implementation in safety net health systems. This study found facilitators that can mitigate lack of advance planning and staffing challenges present during public health emergencies such as the COVID-19 pandemic. The ten identified themes may inform future implementations in safety net health systems.

Introduction

When COVID-19 vaccines attained U.S. Food and Drug Administration Emergency Use Authorization (EUA) and were made available to U.S. health systems in December 2020, safety net health systems were challenged to implement widespread vaccination in resource-limited environments during a time of peak COVID-19 transmission (1, 2). Vaccination implementation entailed understanding the evolving regulations of vaccine eligibility and availability, then distributing vaccine accordingly to vulnerable communities experiencing significant racial and economic inequities due to the COVID-19 pandemic. Implementation of safety net initiatives has been associated with challenges including limited staffing, lack of organizational financial investment, and the need to address patients' biopsychosocial complexities. Facilitators to implementation have included advance planning, redundancy in communication, knowledge of patient needs, desire to perform well, personnel commitment to reducing health inequities, and multidisciplinary teams to drive implementation (3–6). More research is needed to understand the role of implementation factors in the safety net, particularly for primary care-led vaccine distribution approaches (7). To date, there are few published qualitative studies of COVID-19 vaccine delivery in safety net health systems (8, 9), and none in a large safety net health system with coordination across many sites.

The Los Angeles County Department of Health Services (LACDHS) faced challenges in rapid vaccine implementation due to the size of the system, geographic breadth, language/racial/ethnic diversity and socioeconomic complexity of patients (low-income, publicly insured, and/or uninsured). LACDHS is the second largest public health system in the United States, serving over half a million unique patients annually across eight health center groups and four hospitals (Figure 1). Facilities span a geographic area greater than 4,000 square miles, including rural, urban, suburban, and exurban populations. The LACDHS empaneled patient population comprises approximately 60% Latinx and 12% Black/African American patients, compared to the overall LA County population with 46% Latinx and 8% Black/African-American people (10, 11). These populations were disproportionately affected during the pandemic as Latinx and Black people living in LA County had death rates nearly twice that of non-Hispanic white people (12–14). Persons in extremely poor or high-poverty census tracts had the highest COVID-19 case and death rates in LA County (14, 15).

FIGURE 1

Figure 1. Los Angeles county department of health services (LACDHS) organization structure of COVID-19 vaccine clinics.

In January 2021, LACDHS launched twelve COVID-19 vaccine clinics at each hospital and health center group, geographically spread across the county (Figure 1). The goal of the LACDHS COVID-19 vaccine implementation was to vaccinate as many patients as quickly as possible in the setting of limited access to vaccines, rapidly evolving eligibility guidelines, and staffing shortages related to the concurrent COVID-19 surge. We aimed to identify determinants impacting implementation of a systemwide COVID-19 vaccine intervention using the Consolidated Framework for Implementation Research (CFIR) in a safety net health system under circumstances where advance planning was limited. This evaluation could inform future population-level implementation efforts in safety net health systems, especially during public health emergencies.

Methods

This qualitative study evaluated determinants of LACDHS COVID-19 vaccine clinic implementation during the initial period of phased vaccine availability from January 2021 to the end of April 2021. The LA County Department of Public Health (LACDPH) Institutional Review Board approved the study before the initiation of the research. We report our work using the Standards for Reporting Qualitative Research (16, 17).

Setting and organization of LACDHS COVID-19 vaccine clinics

In January 2021, LACDHS leveraged its experience with and infrastructure from prior dedicated influenza clinic implementation to set up twelve COVID-19 vaccine clinics geographically spread across the county (Figure 1). LACDHS convened a multidisciplinary vaccine steering committee and primary care vaccine strategy workgroup to engage key stakeholders across disciplines and sponsor systemwide vaccination planning (Figure 2 “Central Leadership”). Each clinic site formed multidisciplinary leadership teams (Figure 2 “Site Leadership”) to oversee local COVID-19 vaccine clinic implementation.

FIGURE 2

Figure 2. LACDHS COVID-19 vaccine clinic implementation leadership and communications structure.Blue-shaded boxes summarize leadership infrastructure and yellow-shaded boxes summarize communications infrastructure.

Central leadership determined which sites within LACDHS would house its limited supply of ultra-low temperature freezers and met regularly to decide on system-level strategies impacting COVID-19 vaccine allocation and administration (Figure 2). Central leadership also made the critical decision to focus initial vaccination efforts on LACDHS-empaneled patients rather than the general public. Given that empaneled patients are disproportionately Latinx and Black, low-income, and undocumented people compared to the general LA County population, this was an intentional system-level decision to combat inequities in COVID-19 care.

COVID-19 vaccine clinics offered appointment-based and walk-in access. Some sites also offered drive-up services. To target eligible patients most effectively during the early phases of CDC vaccine eligibility, data-driven patient outreach lists were generated based on patient age and information on chronic conditions from the EMR. Utilizing outreach lists and scheduling scripts, clinic and call center staff called patients to schedule vaccine appointments, using bilingual staff when available. Additional strategies to reach vulnerable patients included automated phone calls, texts, and emails for high-volume population outreach. Staff also scheduled vaccine clinic appointments when patients presented to clinic sites for other reasons (i.e., primary care appointments, pharmacy medication pick-ups, and laboratory testing). Vaccine clinics were mostly staffed using existing staffing resources, with little additional hiring of contractors.

Summary of LACDHS COVID-19 vaccine administration during implementation

From January 21, 2021, through April 30, 2021, LACDHS COVID-19 vaccine clinics administered 101,222 COVID-19 vaccinations. This number excludes doses administered in hospital inpatient wards, emergency departments, homeless sites, or correctional facilities not administrated by the COVID-19 vaccine clinics described herein. Of the 59,898 unique outpatients LACDHS vaccinated during this initial implementation period, 29.8% were aged 65 years or over, 69.7% were aged 18–64, and 0.5% were aged 16–17. 57.5% of vaccinated identified as female, 42.4% as male, and less than 0.1% as other or unknown. The race and ethnicity breakdown of those vaccinated during implementation was: Hispanic/Latinx 69.2%, Black/African American 8.1%, Asian 6.7%, White 3.6%, American Indian/Alaskan Native 0.1%, Native Hawaiian/Pacific Islander 0.1%, Multi-Race 0.3%, Other/Unknown 11.9%. The majority of COVID-19 vaccinations LACDHS administered (85%) went to individuals in the lowest two quartiles of the Healthy Places Index (accounting for social determinants of health including education, job opportunities, environmental factors) which surpassed overall LA County performance in this regard (18). Vaccine administration data and demographic information were gathered from the LACDHS electronic medical record and analyzed using Statistical Analysis Software.

Interview guide

We utilized the CFIR to design an interview guide for COVID-19 vaccine clinic stakeholders. CFIR domains covered included inner setting, outer setting, individuals, process, and intervention characteristics (19, 20). The guide included open-ended primary questions and prompts to elicit thorough responses (see Supplementary Material A for the interview guide mapped to CFIR domains).

Interview recruitment and methods

From August 2021 through November 2021, potential participants were recruited via purposive sampling. The research team asked the twelve vaccine clinic directors to identify a cross-section of personnel who played integral roles in the planning, managing, and/or daily operations of the local vaccine clinic from January-April 2021. These potential participants were either members of “Site Leadership” or front-line staff in COVID-19 vaccine clinics. Each potential participant was emailed up to three times. Interested participants were sent an electronic pre-interview demographic survey and attitude questions (see Supplementary Material B for pre-survey) and scheduled for an interview. Participants provided written informed consent. Interviews were recorded via HIPAA-compliant internet phone or video call and lasted 30–45 min. A professional transcription service transcribed interview recordings and de-identified proper names, clinic names, and locations.

Rapid qualitative analysis

The research team applied a rapid analysis approach to analyzing all 25 stakeholder interviews (21–24). The team developed a master transcript summary template based on the original interview guide. This template was adapted upon the team trialing the template with six transcripts to improve standardization of data entry until consensus was reached. Researcher pairs then independently took notes, selected exemplar quotations from each de-identified interview transcript, compared notes, and edited a single high-level summary for each interview. Researchers met weekly to discuss findings, resolve discrepancies, and build consensus on transcript summaries. Each transcript summary was entered into an Excel matrix (24–27). Each row captured an individual interview and each column represented a topic area from the summary template (see Supplementary Material C for the transcript summary template). Then, the team identified and summarized major themes and representative quotes across interviews, which mapped to four CFIR domains. A synthesized summary of findings was presented to participants for comment and correction. An audit trail was kept throughout the analysis, including survey and interview guide drafts, scheduling logistics, raw data, field notes, rapid analysis summaries, weekly meeting minutes, and other process notes documenting key steps in methodological decision-making.

Results

Of 40 potential participants, 32 responded to initial email requests, 26 completed the participant demographic survey, and 25 ultimately completed an interview (see Table 1 for Survey Participant demographics). The survey also included two questions related to the experience of participating in the implementation. When asked to rate the ease or difficulty of the COVID-19 vaccine implementation at their site on a Likert scale from 1 (easiest) to 5 (hardest), 31% (n = 8) of participants indicated implementation was easy or very easy and 38% (n = 10) participants indicated implementation was difficult or very difficult. Ninety-two percent of participants (n = 24) endorsed they would agree to be part of the COVID-19 vaccine clinic if asked again.

TABLE 1

Table 1. Survey participant demographics*.

Ten narrative themes emerged as determinants of the LACDHS COVID-19 vaccine implementation. These determinants are presented as they correspond to four CFIR domains: Innovation Characteristics, Outer Setting, Inner Setting, and Process. Themes and exemplar quotes are presented in Table 2.

TABLE 2

Table 2. Themes and exemplar quotes from the Los Angeles County Department of Health Services COVID-19 vaccine implementation evaluation.

Innovation characteristics—implementation of COVID-19 vaccine clinics

Theme 1. LACDHS central leadership guidance and local site flexibility supported problem-solving during rapid implementation.

The LACDHS leadership communication infrastructure included structured weekly and ad hoc meetings between central and site leadership (Figure 2). This served as a platform for the bidirectional exchange of ideas between central and site leadership and across sites. These meetings provided a forum to clarify rapidly evolving information, coordinate and align around promoting health equity, and share best practices and lessons learned which site leaders could bring back to vaccine clinics to adapt local workflows quickly. Sites had leeway to adapt workflows based on local needs and resources while aligning with central guidelines. Participants viewed central leadership as informative and transparent, communicating regularly to inform clinics of the latest federal, state, and county policies.

Standardization worked effectively to an extent, but it was ultimately up to the clinics to adapt the implementation to best meet the needs of their local site and teams, particularly related to staffing and space availability. Participants described variations across vaccine clinics regarding infrastructure, demonstrating local flexibility in implementing central guidance. Sites strategized staffing solutions in the context of a concurrent winter COVID surge with nursing shortages due to frequent staff sick calls and redeployment of outpatient nursing to inpatient settings. As a result, some sites used staff from pandemic-closed clinics (e.g., dentists), some paid overtime for staff to work additional hours, and others used registries, volunteers, or students to staff vaccine clinics. Some sites pulled staffing from primary care or urgent care clinics, leaving those clinics short-staffed, sometimes leading to staff resentment. Each site had different vaccine clinic floor plans with varying accessibility to host the vaccine clinics. Some sites held the vaccine clinics indoors in temporarily closed clinics or repurposed spaces, including lobbies and auditoriums, and others held clinics outdoors on sidewalks, patios, and in covered garages.

Theme 2. Multidisciplinary teams facilitated vaccine implementation and vaccine clinic activities.

LACDHS assembled a multidisciplinary leadership team at the central level which included physician, nursing, pharmacy, and patient access leadership (Figure 2). This team designed a vaccine implementation toolkit to provide integrated guidance for the vaccine clinics. For example, physician leadership digested and communicated clinical and public health updates, nursing leadership addressed workflow and informatics needs related to vaccine administration, and patient access leads designed scripts and workflows for outreach and patient registration. Due to the complexity of inventory and allocation, storage, handling, and preparation of the vaccines, pharmacy leadership coordinated the distribution of large direct vaccine shipments across the system and monitored utilization across vaccine clinics. Pharmacy leads managed re-distribution of vaccine between sites to accommodate daily patient volume and minimize waste associated with short expiration dates.

Based on central leadership toolkit guidelines, site leadership assembled local multidisciplinary teams to problem solve and optimize workflows across staff types, and adapt workflows in real time. The strategic choice to designate a lead pharmacist role in the COVID-19 vaccine clinics was identified as an essential enabler of vaccine clinic efficiency as site pharmacists had knowledge of LACDHS vaccine resources and could mix, draw, administer, and counsel on the vaccine. Central leadership toolkit materials were designed to allow for workflow and role flexibility. For example, pharmacists could administer vaccine if there were nursing shortages, and nursing could register new patients when there was a clerical shortage.

Outer setting—macro-level factors that originate outside the LACDHS system

Theme 3. Initial COVID-19 vaccine scarcity and rigid eligibility tiers led to ethical dilemmas.

During the early weeks of vaccine scarcity and strict adherence to state and federal eligibility tiers, avoiding vaccine wastage was one of the participants' most significant concerns and even a source of anxiety. This felt most weighty at the end of a clinic session when the time came to draw the vaccine from the last multi-dose vial for the day, and there were more doses than patients remaining. Some participants expressed this as an ethical challenge: avoiding administering vaccines to patients outside the eligibility tiers meant doses might be wasted. Participants noted it was challenging to stay within eligibility tiers as tiers rapidly shifted. For example, at one point, there was discordance between the CDC and LACDPH guidance on the definition of chronic conditions and how to vet eligibility by occupation instead of age. Participants also experienced moral conflict when withholding vaccine from high-risk patients close to meeting eligibility criteria but did not fall into current eligibility tiers. Participants noted this was a tense time—balancing a reluctance to turn patients away with the risk of vaccine wastage resulted in extraordinary efforts to find patients to use the last remaining doses which could not be stored. One participant described this undertaking:

“We felt very strongly that given this small, scarce supply, it was irresponsible for us as healthcare providers to let a dose go to waste. And so, my entire objective and purpose from the onset and being involved was to try to ensure that no dose was wasted.”—Other staff member (administrative, scheduling, health education, physical therapy) from Site 4

To administer all remaining doses, participants performed last-minute outreach including overhead announcements, finding vulnerable staff such as environmental services and dietary workers to vaccinate, or going to Urgent Care and the emergency department to find patients before the vaccine had to be wasted.

Theme 4. Initial COVID-19 vaccine scarcity and infrastructure limitations made coordination of vaccine delivery across LACDHS complex.

Initial scarce COVID-19 vaccine supply necessitated complex coordination of vaccine distribution across our large health system. LACDHS received vaccine shipments weekly only at select sites with ultra-low temperature freezers. Vials then had to be re-distributed to sites without ultra-low freezers. Limited and variable weekly vaccine allocations restricted how far in advance patients could be scheduled. This resulted in complicated staffing and outreach planning, and sometimes led to site pharmacists driving long distances across the county to pick up doses from another LACDHS site. Participants also commented on the challenges of dealing with unpredictable and variable vaccine availability and the differences across multiple vaccine brands, including dosing intervals, expiration dates, and community preferences.

Inner setting—pertaining to the infrastructure, resources, and culture of the LACDHS system

Theme 5. Underestimated time and resources required to overcome vaccine concerns and misinformation.

Participants observed that additional time and resources were needed to overcome vaccine hesitancy and misinformation at all points of patient contact. This included encounters with the primary care provider, the nurse advice line, at the time of vaccine scheduling, while waiting in line at the vaccine clinic and at the time of vaccine administration. Local sites performed most of the patient-level vaccine outreach and scheduling mostly using non-clinical call center staff. However, non-clinical staff felt unprepared to answer vaccine questions from patients. Some sites reassigned clinical staff to make individualized calls to vaccine-hesitant patients or answer questions on-site at the vaccine clinics. Participants desired more community education and outreach and perceived a lack of consistent scripting for staff, especially in addressing complex vaccine conversations during a clinic visit with several competing priorities. Patients presenting to the vaccine clinic intending to get vaccinated still had questions about allergies, interactions, what to expect after the vaccine, and other concerns. Some sites created their own patient education and FAQ materials. It was viewed as a barrier that LACDHS central leadership did not provide more support in this area. A few staff expressed concern for their own safety from exposure to patients with COVID-19, and compared the COVID-19 pandemic to the HIV epidemic

Theme 6. Broader adoption and use of standing nurse order protocols enabled rapid capacity-building in COVID-19 vaccine clinics.

Participants recognized that the urgency and breadth of COVID-19 vaccine implementation warranted a transformation of existing workflows to improve efficiency for widespread vaccination. A meaningful change was delegating provider vaccine ordering authority to nursing staff for quicker vaccine ordering. LACDHS had prior experience with standing nurse orders, however an important change was made to the nurse training process for the COVID-19 vaccine implementation. Training for the standing nurse orders shifted from periodic in-person training to on-demand recorded virtual training for nurse vaccinators, which allowed hundreds of vaccinators to be trained in a short amount of time. Additionally, electronic post-training proficiency testing allowed for real-time calculation of results, which were posted to the staff portal where an up-to-date master roster of staff ready to vaccinate was maintained. Participants thought completing the online training before arriving to work at the vaccine clinic facilitated orientation and same-day onboarding while staffing was in flux.

Theme 7. A shared sense of purpose fostered a positive team culture

Participants noted a fellowship with their vaccine clinic co-workers when asked about site-level engagement. Staff had a strong sense of purpose and a feeling of responsibility to match the moment and be a part of history fighting the pandemic. Participants agreed a robust process for communication and collaboration amongst the local site team was a key factor in success. Participants desired to reach as many patients as possible with an “all hands-on deck” approach and willingness to do whatever it took to “make it work.” One participant described the team approach:

“That … concept of ‘It takes a village’ and our administration, everyone enacted an approach and commitment to getting our patients vaccinated.”—Clinical provider/medical director from Site 3

Multiple participants expressed feeling proud that they were making a difference. Openness to feedback and continuous improvement fostered a culture of multidisciplinary teamwork and collaboration, which stemmed from shared investment in the work. Sites were keenly aware of the safety net patient population, which led to many discussions at the local level about historical and contemporary marginalization and vaccine hesitancy as barriers to COVID-19 health equity. This strong sense of purpose facilitated buy-in for COVID-19 vaccine clinic implementation. Additionally, participants praised site leaders who showed gratitude and appreciation for vaccine clinic staff. Many leaders were present on the front lines to quickly address staffing and supply issues, effectively promoting a teamwork culture.

Process—means by which LACDHS COVID-19 vaccine clinics were implemented

Theme 8. Active and passive communication structures enabled sites to adapt to evolving demands.

Participants discussed the site-specific rapid decision-making related to implementation of brand-new vaccine clinic workflows and expressed the feeling that “we were building the plane as we flew it.” Site leaders realized they had to develop site-specific tools and infrastructure to support real-time communication between local team members. Debriefing with frontline staff promoted staff engagement in continuous improvement and enabled sites to walk back from stalled innovations. Sites with effective communication used various tools (e.g., emails and remote meeting platforms) and built redundancy in their communication structure (e.g., daily clinic huddles, weekly meetings, and workstations in proximity to leaders). Sites without timely, broad, and multidisciplinary communication structures felt challenged. Participants cited ad hoc meetings, frequent updates relaying messages from central leadership, and openness to feedback from frontline staff as effective communication methods used by site leaders.

Theme 9. Developed patient-centered engagement strategies for COVID-19 vaccine clinic scheduling and vaccine administration.

Participants enthusiastically described the novel ways their sites engaged patients to get vaccinated.

LACDHS central leadership created low-literacy vaccine Frequently Asked Questions documents in English and Spanish for use in the vaccine clinics. Site leaders were intentional about staffing vaccine clinics with diverse and multilingual staff and interpreters, along with providing appropriate educational materials when available. Sites used data-driven patient outreach lists provided by central leadership to schedule patients. Motivational interviewing, clinic staff sharing their vaccine stories, and face-to-face patient communication were important tools that helped engage patients. Sites prioritized direct patient education and communication; providers and clinic staff engaged patients while waiting in line and during and after vaccine administration to answer questions.

Sites provided broad access to vaccine appointments by offering evening and weekend clinics, accepting walk-ins, and performing patient-centered scheduling to combine a vaccination visit with another clinic visit. Online self-scheduling was also available systemwide for first doses. Participants expressed a desire for expanded self-scheduling for subsequent vaccine doses. Efforts to recruit patients for vaccine scheduling extended beyond phone outreach to every touch patients had with the clinics, e.g., picking up medications at the pharmacy or getting labs done. Participants described strategies to meet limited-mobility patient needs by providing wheelchairs and walkers on-site, vaccinating at curbside, and coordinating home vaccination referrals. One participant described these efforts:

“So the patients who needed to be in their cars, so they were handicapped, they weren’t dressed for the weather, they had a child—and this are all that has happened—or they were helping someone who was not ambulatory or they themselves weren’t ambulatory. We set up, like, reserved parking spaces as close to the vaccine station as possible and they would just tell the registration staff when they drove in that they were not able to walk up.”—Other staff member from Site 7

Additionally, patient safety, comfort, and experience were of paramount importance. Participants reported designating places for patients to lie down, socially distanced observation areas, and providing free personal protective equipment and outdoor heating.

Theme 10. Sites encountered a variety of process challenges implementing COVID-19 vaccine clinics.

Central leadership designed social media and broadcast message campaigns in English and Spanish to encourage vaccination and created patient outreach lists stratified by language for sites to schedule eligible patients. While most sites agreed with this outreach approach, one site refrained from performing language-concordant outreach for fear of prioritizing that ethnic group over English-speaking patients. The live outreach calls required extensive effort, yielded mixed results, and sometimes seemed to be wasted effort. Non-clinical scheduling staff worked outreach lists that were thousands of patients long, making multiple attempts and leaving voicemail messages if patients did not initially answer. Site staff also leveraged previously infrequently used robocall technology to perform automated outreach. Staff accommodated variable incoming call volume by adjusting staffing shifts, modifying the interactive voice response (phone tree branching structure), and continuously monitoring calls. In addition to this outreach, sites fielded a high volume of incoming calls from patients requesting to schedule vaccine appointments, many of whom were not yet eligible per county eligibility tiers.

LACHDS COVID-19 vaccine clinics were based in primary care settings. The hospitals and larger health centers also deliver specialty care and varied in how much their vaccine clinic collaborated with specialty care. These sites varied in their workflows of how patients in specialty care were directed to vaccine clinics, and how limited staffing was distributed between primary care, vaccine, and specialty care clinics. During the early phases of implementation in the setting of vaccine scarcity, central leadership focused initial vaccination efforts on LACDHS-empaneled primary care patients rather than the public to promote vaccine equity. This led to some confusion and tension at sites when non-empaneled patients receiving specialty care at LACDHS sites could not be vaccinated even when meeting vaccine eligibility criteria.

Participants were forthcoming that not all site-level operational decisions were adaptive. Examples of workflow decisions that were not sustained or served as barriers to efficient vaccination included: not hiring temporary nursing staff which resulted in staffing shortages, mixing vaccines in pharmacy hoods rather than at the vaccine clinic which required additional staff runners to transport vaccines, limiting Janssen vaccine administration for women over age 50 due to concern for thrombus despite no such FDA guideline, and not opening vaccine clinic on county holidays despite available staffing. Some participants noted that central leadership could have helped anticipate some logistical needs of the sites, such as coordinating bulk printing of vaccine clinic signage and purchasing of cold cubes for vaccine storage and tents for outdoor vaccine administration.

Discussion

Participants’ experience of the LACDHS COVID-19 vaccine implementation converged on ten themes related to four CFIR domains: Innovation Characteristics, Inner Setting, Outer Setting, and Process. These themes illustrate how our large safety net health system rapidly mobilized to launch broad-scale COVID-19 vaccination during a public health emergency. Limited resources necessitated LACDHS leadership and staff to be resourceful by leveraging bi-directional communication, quickly adapting to local site and patient needs, and promoting teamwork, all while aligning work to evolving COVID-19 vaccine guidelines.

Our study adds to the literature by providing a qualitative assessment of a large-scale implementation in a safety net health system where coordination across many sites was a core feature. LACDHS had implemented systemwide programs in the past (28, 29); however, no previous intervention was as far-reaching or had to be rolled out as quickly under such a systemwide strain on resources. To date, there have been few publications detailing COVID-19 vaccine implementation in safety net systems. DiVirgilio et al. emphasized community education and targeting by zip code to focus on communities disproportionately affected by COVID-19 morbidity and mortality in their Chicago study (8). San Francisco's safety net health network highlighted drop-in hours as the most effective way to lower barriers to COVID-19 vaccine access (9). Both studies operated on a smaller scale (approximately 5,000 patients in the San Francisco study and 11,000 patients in the Chicago study), with less geographic breadth.

A key facilitator of implementation seen in some safety net health system studies is an emphasis on advance planning for implementation (5, 6). However, health systems did not have the luxury of advance planning with COVID-19 vaccine implementation during the pandemic. The LACDHS case of COVID-19 vaccine clinic implementation under time pressure suggests that real-time frontline staff input into implementation design and balancing system standardization and local site adaptations are important facilitators in scenarios where advance planning is not possible. The rapid stand-up of LACDHS COVID-19 vaccine clinics demonstrates how a traditionally rigid system can be agile and adaptive to meet the moment.

A recent review of qualitative implementation studies in safety net settings found that understaffing and high staff turnover rates were the most common reason for the lack of acceptability of interventions (6). LACDHS, like other safety net systems, experienced high staff turnover and understaffing rates during the pandemic due to the inpatient COVID-19 surge and COVID-19-related sick callouts. Our study identified the use of multidisciplinary teams, bidirectional communication across leadership and sites, and the broad use of standing nurse orders as facilitators which helped overcome the barriers of staff turnover and understaffing. Similar to Crable's findings (3) where stakeholders' personal commitment to reduce health inequities was a facilitator of implementation, a takeaway of this evaluation is that a positive work culture and a clear shared goal helped mitigate a pressured work environment with high demands. Our themes of the importance of bidirectional and frequent communication and addressing patients' biopsychosocial complexities also aligned with previous studies of implementation in safety net settings (3).

The LACDHS vaccine clinic implementation deepened participants' and the research team's appreciation for the social complexity of the LA county safety net population. Frontline staff served as the best advocates for identifying and addressing social needs to lower barriers to patient vaccine access (30); however, beyond providing operational accommodations such as expanded vaccine clinic hours, bilingual staff, and assistance for those with limited mobility, sites had varying interpretations of how to promote health equity in vaccine clinics. Most sites and staff naturally focused on addressing the social needs of individual patients, rather than on the root causes of health inequities. Participants experienced moral discomfort when asked to focus on empanelment status and vaccine eligibility criteria, rather than vaccinating all-comers. These scenarios illustrate how additional training promoting a deeper understanding of health equity vs. equality is needed. To build on a commitment to inclusive care, LACDHS should offer staff training to further develop structural competency in health equity, an important step in the journey to advance health equity (31–33).

The LACDHS COVID-19 vaccine implementation experience highlighted the need to develop a comprehensive patient education strategy at the system level, encompassing outreach communications as well as education at the point of care. Safety net patient populations comprise diverse groups with different sociocultural, education, and outreach needs to combat vaccine misinformation and promote vaccine uptake (34–37). Participants expressed that central leadership did not provide enough support in patient education. LACDHS central leadership included low-literacy vaccine FAQ documents and scheduling scripts in its implementation toolkit, however, additional support and a robust infrastructure for patient education was needed. Some sites opted to create their own scheduling scripts, FAQs, and signage, reallocated clinical staff to address questions, and attempted to use language-concordant and culturally concordant staff to promote optimal health communication. Additional patient education materials in the vaccine clinics and resources to address vaccine hesitancy might have increased vaccine acceptance and vaccination rates. Allocating funding for and investing resources in building patient education, engagement, and communications infrastructure as a core service for safety net systems would be an important step to laying the foundation for future successful safety net implementations.

Study limitations include potential participant recall bias. Interviews were conducted from September to November 2021, months after the defined early vaccine implementation period from January to April 2021. Interviews were intentionally conducted mainly with vaccine clinic team members rather than central leadership, leading to a perspective focused more on site and frontline experiences. Finally, this study lacked patient perspectives on the LACDHS vaccine clinic implementation.

This comprehensive qualitative analysis of the LACDHS rapid implementation of COVID-19 vaccine clinics yielded important lessons for safety net health systems caring for populations experiencing disproportionate disease burden due to societal inequities. This analysis fostered a deepened understanding of facilitators which can help overcome understaffing and a lack of advance planning. Key facilitators included using robust communication between all levels of the organization and balancing workflow standardization with local site flexibility. Additional lessons included the importance of building system capacity for health equity work and regarding patient engagement and communications infrastructure as a core necessity for safety net health systems. Applying these lessons in future implementations can benefit staff, patients, and safety net communities.

Data availability statement

The original contributions presented in the study are included in the article/Supplementary Material, further inquiries can be directed to the corresponding author.

Ethics statement

The studies involving human participants were reviewed and approved by Los Angeles County Public Health, Ambulatory Care Network and Health Services Administration Institutional Review Board. The participants provided their written informed consent to participate in this study.

Author contributions

JC and CZ conceived of the study design and developed the interview guide, with expert guidance from SC and review by JT, GG, AT, RK, and KK. RK conducted interviews and data acquisition. JC, JT, GG, AT, KK, RK conducted data analysis and interpretation. JC, JT, GG, AT, and KK created tables and figures. SC provided expertise in qualitative research and rapid analysis techniques. AB provided expertise in qualitative research during conception and design of the research and funding for research support. JC, JT, GG, AT, KK, RK, CZ drafted sections of the paper, with SC and AB providing critical revisions, and JC and KK responsible overall for the paper's content and accuracy and responsibility for accuracy and integrity of all aspects of this research. All authors contributed to the article and approved the submitted version.

Funding

This paper was supported by NIH/NHLBI CEAL grant #21-312-0217571-66106l and NIH/NCATS grant #UL1TR001881.

Acknowledgments

The authors thank Paul Giboney for support of this study, Breena Taira for assistance with study design and advice on conducting semi-structured interviews. Research reported in this publication was supported by the Community Engagement Alliance Against COVID-19 Disparities (CEAL) of the National Heart Lung and Blood Institute (NHLBI) of the National Institutes of Health under the UCLA Clinical and Translational Science Institute grant number 21-312-0217571-66106l and the National Center for Advancing Translational Science (NCATS) of the National Institutes of Health under the UCLA Clinical and Translational Science Institute grant number UL1TR001881.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/frhs.2023.1152523/full#supplementary-material.

References

1. Chatterjee P, Sommers BD, Maddox KEJ. Essential but undefined—reimaging how policymakers identify safety-net hospitals. N Engl J Med. (2020) 383:2593–5. doi: 10.1056/NEJMp2030228

PubMed Abstract | CrossRef Full Text | Google Scholar

2. Knudsen J, Chokshi DA. COVID-19 and the safety net—moving from straining to sustaining. N Engl J Med. (2021) 385:2209–11. doi: 10.1056/NEJMp2114010

PubMed Abstract | CrossRef Full Text | Google Scholar

3. Crable EL, Biancarelli D, Walkey AJ, Drainoni M. Barriers and facilitators to implementing priority inpatient initiatives in the safety net setting. Implement Sci Commun. (2020) 1:35. doi: 10.1186/s43058-020-00024-6

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Moore SL, Fischer I, Havranek EP. Translating health services research into practice in the safety net. Health Serv Res. (2016) 51(1):16–31. doi: 10.1111/1475-6773.12341

PubMed Abstract | CrossRef Full Text | Google Scholar

5. Aggarwal M, Kokorelias KM, Glazier RH, Katz A, Shiers-Hanley JE, Upshur REG. What is the role of primary care in the COVID-19 vaccine roll-out and the barriers and facilitators to an equitable vaccine roll-out? A rapid scoping review of nine jurisdictions. BMJ Open. (2023) 13:e065306. doi: 10.1136/bmjopen-2022-065306

PubMed Abstract | CrossRef Full Text | Google Scholar

6. UCLA Center for Health Policy Research. Still Left Behind: Health Insurance Coverage and Access to Care Among Latinos in California. (August 2019). Available at: https://healthpolicy.ucla.edu/publications/Documents/PDF/2019/LatinoInsurance-policybrief-aug2019.pdf

7. Office of Patient Data and Analytics, LA County Department of Health Services. Data from: LACDHS county data file on COVID-19 vaccine delivery. Los Angeles, CA (2021).

8. Martinez LE, Vargas-Bustamante A, Anaya YB, Dominguez-Villegas R, Santizo-Greenwood S, Diaz SFM, et al. COVID-19 in vulnerable communities: an examination by race & ethnicity in Los Angeles and New York City (2020). Available at: https://latino.ucla.edu/research/covid-19-in-vulnerable-communities/

9. Casillas E, Wu G, Ning WV, Choi J, Chan P, Lee ML. COVID-19: highlighting health disparities in the Los Angeles latinx community. Clin Med Res. (2021) 19(4):161–8. doi: 10.3121/cmr.2021.1654

PubMed Abstract | CrossRef Full Text | Google Scholar

10. Barboza T, Poston B, Menezes R. Coronavirus ravages poorer L.A. communities while slowing in wealthier ones, data show. Los Angeles Times. (May 28, 2020). Available at: https://www.latimes.com/california/story/2020-05-28/coronavirus-surge-in-poor-l-a-county-neighborhoods-reveals-two-americas

11. Los Angeles County Department of Public Health, Chief Science Office. Report on LA County COVID-19 Data Disaggregated by Race/Ethnicity and Socioeconomic Status. (April 28, 2020).

12. DiVirgilio L, Boshara A, Hunt BR, Jacobs J, Just K, Johnson AK. A community-informed approach to COVID-19 vaccine roll-out in under-served areas in Chicago. Front Public Health. (2022) 10:863125. doi: 10.3389/fpubh.2022.863125

PubMed Abstract | CrossRef Full Text | Google Scholar

13. Stern RJ, Raffery HF, Robert AC, Taniguchi C, Gregory B, Khoong EC, et al. Concentrating vaccines in neighborhoods with high COVID-19 burden. NEJM Catalyst. (2021). doi: 10.1056/CAT.21.0056

PubMed Abstract | CrossRef Full Text | Google Scholar

14. Kegler MC, Beasley DD, Liang S, Cotter M, Phillips E, Hermstad A, et al. Using the consolidated framework for implementation research to understand safety net health system efforts to increase colorectal cancer screening rates. Health Educ Res. (2018) 33:315–26. doi: 10.1093/her/cyy019

PubMed Abstract | CrossRef Full Text | Google Scholar

15. Lyles CR, Handley MA, Ackerman SL, Schillinger D, Williams P, Westbrook M, et al. Innovative implementation studies conducted in US safety net health care settings: a systematic review. Am J Med Qual. (2019) 34(3):293–306. doi: 10.1177/1062860618798469

PubMed Abstract | CrossRef Full Text | Google Scholar

16. Nowell LS, Norris JM, White DE, Moules NJ. Thematic analysis: striving to meet the trustworthiness criteria. Int J Qual Methods. (2017) 16:1–13. doi: 10.1177/1609406917733847

CrossRef Full Text | Google Scholar

17. O’Brien BC, Harris IB, Beckman TJ, Reed DA, Cook DA. Standards for reporting qualitative research: a synthesis of recommendations. Acad Med. (2014) 89(9):1245–51. doi: 10.1097/ACM.0000000000000388

CrossRef Full Text | Google Scholar

18. Data from: LA County Department of Health COVID-19. Retrieved from LA County Daily COVID-19 Data. Available at: http://publichealth.lacounty.gov/media/Coronavirus/data/index.htm (April 29, 2021).

19. Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. (2009) 4(1):50. doi: 10.1186/1748-5908-4-50

PubMed Abstract | CrossRef Full Text | Google Scholar

20. Damschroder LJ, Reardon CM, Widerquist MA, Lowery J. The updated consolidated framework for implementation research based on user feedback. Implement Sci. (2022) 17:75. doi: 10.1186/s13012-022-01245-0

PubMed Abstract | CrossRef Full Text | Google Scholar

21. Hamilton A. Qualitative methods in rapid turn-around health services research: vA HSR&D national cyberseminar series. Los Angeles: Spotlight on Women’s Health (2013).

22. Nevedal AL, Reardon CM, Opra Widerquist MA, Jackson GL, Cutroma SL, White BS, et al. Rapid versus traditional qualitative analysis using the consolidated framework for implementation research (CFIR). Implement Sci. (2021) 16:67. doi: 10.1186/s13012-021-01111-5

PubMed Abstract | CrossRef Full Text | Google Scholar

23. Hamilton AB, Finley EP. Qualitative methods in implementation research: an introduction. Psychiatry Res. (2019) 280:112516. doi: 10.1016/j.psychres.2019.112516

PubMed Abstract | CrossRef Full Text | Google Scholar

24. Averill JB. Matrix analysis as a complementary analytic strategy in qualitative inquiry. Qual Health Res. (2002) 12(6):855–66. doi: 10.1177/104973230201200611

PubMed Abstract | CrossRef Full Text | Google Scholar

25. Miles MB, Huberman AM. Qualitative data analysis: an expanded sourcebook. Thousand Oaks, CA: Sage Publications, Inc. (1994).

26. Abraham TH, Finley EP, Drummond KL, Haro EK, Hamilton AB, Townsend JC, et al. A method for developing trustworthiness and preserving richness of qualitative data during team based analysis of large data sets. American Journal of Evaluation. (2021) 42(1):139–56. doi: 10.1177/1098214019893784

CrossRef Full Text | Google Scholar

27. Gale RC, Wu J, Erhardt T, Bounthavong M, Reardon CM, Damschroder LJ, et al. Comparison of rapid depth qualitative analytic methods from a process evaluation of academic detailing in the veterans health administration. Implement Sci. (2019) 14(1):11. doi: 10.1186/s13012-019-0853-y

PubMed Abstract | CrossRef Full Text | Google Scholar

28. Barnett ML, Yee HF, Mehrotra A, Giboney P. Los Angeles Safety-Net program eConsult system was rapidly adopted and decreased wait times to see specialists. Health Aff. (2017) 36(3):492–9. doi: 10.1377/hlthaff.2016.1283

CrossRef Full Text | Google Scholar

29. Daskivich LP, Vasquez C, Martinez C Jr., Tseng C-H, Mangione CM. Implementation and evaluation of a large-scale teleretinal diabetic retinopathy screening program in the Los Angeles county department of health services. JAMA Int Med. (2017) 177(5):642–9. doi: 10.1001/jamainternmed.2017.0204

CrossRef Full Text | Google Scholar

30. Grumbach K, Carson M, Harris OO. Achieving racial and ethnic equity in COVID-19 vaccination: from individual readiness to health system readiness. JAMA Health Forum. (2021) 2(7):e211724. doi: 10.1001/jamahealthforum.2021.1724

PubMed Abstract | CrossRef Full Text | Google Scholar

31. Manchanda R, Do R, Miles N. A toolkit to advance racial health equity in primary care improvement. San Francisco, CA: California Improvement Network, California Health Care Foundation, Healthforce Center at UCSF (2022).

32. American Medical Association. Advancing health equity: a guide to language, narrative, and concepts. Chicago, IL: American Medical Association. (2021) 21:587460:pdf:10/21. Available at: https://www.ama-assn.org/system/files/ama-aamc-equity-guide.pdf

33. Centers for Disease Control. Health equity guiding principles for inclusive communication. CS 328419-A. Atlanta, GA (2022). Available at: https://www.cdc.gov/healthcommunication/HealthEquityGuidingPrinciples.pdf

34. Carson SL, Casillas A, Castellon-Lopez Y, Mansfield LN, Morris D'Ann, Barron J, et al. COVID-19 Vaccine decision-making factors in racial and ethnic minority communities in Los Angeles, California. JAMA Network Open. (2021) 4(9):e2127582. doi: 10.1001/jamanetworkopen.2021.27582

PubMed Abstract | CrossRef Full Text | Google Scholar

35. Malik AA, McFadden SM, Elharake J, Omer SB. Determinants of COVID-19 vaccine acceptance in the US. EClinicalMedicine. (2020) 26:100495. doi: 10.1016/j.eclinm.2020.100495

PubMed Abstract | CrossRef Full Text | Google Scholar

36. Corbie-Smith G. Vaccine hesitancy is a scapegoat for structural racism. JAMA Health Forum. (2021) 2(3):e210434. doi: 10.1001/jamahealthforum.2021.0434

PubMed Abstract | CrossRef Full Text | Google Scholar

37. Savoia E, Piltch-Loeb R, Goldberg B, Miller-Idriss C, Hughes B, Montrond A, et al. Predictors of COVID-19 vaccine hesitancy: socio-demographics, co-morbidity and past experience of racial discrimination. Vaccines (Basel). (2021) 9(7):767. doi: 10.3390/vaccines9070767

PubMed Abstract | CrossRef Full Text | Google Scholar