The quick pivot: Capturing real world modifications for the re-implementation of an early psychosis program transitioning to virtual delivery

Background: Team-based Early Psychosis Intervention (EPI) services is standard of care for youth with psychosis. The COVID-19 pandemic required most EPI services to mount an unplanned, rapid pivot to virtual delivery, with limited guidance on how to deliver virtual clinical services or whether quality of re-implementation and treatment outcomes would be impacted. We used a structured approach to identify essential modifications for the delivery of core components and explored facilitators and barriers for re-implementation and fidelity of a virtually delivered EPI intervention.

Materials and methods: NAVIGATE is a structured approach to team-based EPI. It provides detailed modules to guide delivery of core components including medication management, psychoeducation and psychotherapies, supported employment/education, and family education. Having initially implemented NAVIGATE at the Centre for Addiction and Mental Health (CAMH) in 2017, the EPI service transitioned to virtual delivery amid the COVID pandemic. Using a practice profile developed to support implementation, we detailed how core components of NAVIGATE were rapidly modified for virtual delivery as reported in structured group meetings with clinicians. The Framework for Reporting Adaptations and Modifications for Evidence-Based Interventions (FRAME) was used to describe modifications. Fidelity to the EPI standards of care was assessed by the First Episode Psychosis Fidelity Scale (FEPS-FS). Re-implementation barriers and facilitators and subsequent mitigation strategies were explored using structured clinician interviews guided by the Consolidated Framework for Implementation Research (CFIR).

Results: Identified modifications related to the intervention process, context, and training. We identified contextual factors affecting the re-implementation of virtually delivered NAVIGATE and then documented mitigating strategies that addressed these barriers. Findings can inform the implementation of virtual EPI services elsewhere, including guidance on processes, training and technology, and approaches to providing care virtually.

Discussion: This study identified modifications, impacts and mitigations to barriers emerging from rapid, unplanned virtual delivery of EPI services. These findings can support delivery of high-quality virtual services to youth with psychosis when virtual care is indicated.

1. Introduction

Early psychosis intervention (EPI) is an evidence-based treatment that has become the standard of care for youth with psychosis (1). EPI care is provided by a multidisciplinary team who provide comprehensive treatment including psychoeducation and psychotherapy for psychosis (most commonly, cognitive behavioural therapy), case management, individual psychopharmacological intervention, family education and support, and support for education and employment (1, 2). Previous EPI effectiveness studies demonstrated superior outcomes including reduced mortality, decreased risk of relapse, fewer hospital readmissions, and increased employment rates relative to care as usual (3–7). Furthermore, evidence shows that a manualized package of EPI services called NAVIGATE results in improved functional outcomes compared to care as usual. Clients receiving NAVIGATE showed greater improvement in quality of life and psychopathology, greater involvement in work and school, and remained in treatment longer compared to clients receiving community care (2).

EPI models of care, such as NAVIGATE, are designed for in-person delivery, emphasizing frequent contacts and community outreach. The COVID-19 pandemic prompted an abrupt shift to virtual delivery of EPI care to ensure continuity in the face of public health restrictions (8–10). However, little was known about the modifications required to provide EPI care virtually or their impacts. The abrupt need for virtual care delivery without suspending service meant there was no time for planning or training to prepare for this shift. Clinicians and clients had to quickly adapt to a new delivery method with ongoing adjustments occurring over time.

The impacts of these modifications and whether virtual delivery of EPI care would achieve the same benefits as the in-person intervention were unknown. With the shift to virtual delivery, it is important to better understand the nature of the modifications that are made and their impact on treatment delivery and outcomes. Modifications, especially if unplanned, may or may not align with the core components required to ensure the intervention is effective (11). For instance, modifications that alter or remove core components of the EPI model, or fail to align with population needs may reduce the effectiveness of virtual EPI compared to the original, in-person intervention (11, 12).

Previous work on investigating modifications of evidence-based interventions led to the development of frameworks that can be used to systematically describe and evaluate modifications to evidence-based interventions, including the Framework for Modification and Adaptations (11, 13). The FRAME captures characteristics of modifications and was recently updated to include broader aspects of the implementation process, such as reasons for the modifications (e.g., to improve feasibility, engagement, outcome), level of the modifications (client, clinician, program), timing of the modifications (prior, during, for scale up), and fidelity to the original intervention (consistent or inconsistent) (11). This detailed framework facilitates understanding of the relationships between the modification and key outcomes that can be tested in implementation studies (14, 15). This is important because modifications that remove or alter core components of an intervention may be less effective. Despite significant developments to identify and classify modifications and their impact on outcomes using structured frameworks, there is little guidance on how to systematically document (ad hoc) modifications in a dynamic setting, how to assess the impacts of these modifications over time, and how contextual factors relate to modifications and outcomes.

The Centre for Addiction and Mental Health (CAMH) in Toronto, Ontario, is home to the largest EPI program in Canada, providing assessment and ongoing services to people aged 14–29 years who present with early psychosis. CAMH implemented the NAVIGATE model for EPI service delivery in 2017 for all clients attending the EPI outpatient clinic, and is currently leading a multisite implementation effectiveness study of NAVIGATE across EPI programs in the province of Ontario (16). NAVIGATE is expected to increase consistency of delivery and improve program fidelity to EPI practice standards (16). CAMH has a dedicated Virtual Mental Health and Outreach program that provides telepsychiatry to clients in remote and rural areas. During the COVID-19 pandemic, this program expanded to support other CAMH programs in their delivery of virtual care.

Soon after the onset of the pandemic, CAMH's Slaight Centre Early Intervention Service (SCEIS) was awarded COVID-19-related research funding to investigate the re-implementation of NAVIGATE from in-person to virtual delivery. The aims of this study are (1) to identify the modifications required to re-implement and deliver the NAVIGATE model virtually, (2) to assess whether these modifications affected fidelity to the EPI practice standards, (3) to explore implementation facilitators and barriers related to re-implementation, a term coined here to reflect a second implementation effort following the earlier, full implementation of an intervention, (4) to examine satisfaction with virtual delivery of NAVIGATE among clients, family members and clinicians, and (5) to investigate service engagement with virtual delivery of NAVIGATE. To address these aims, we conducted a mixed methods study using a convergent study design to investigate the unplanned shift to virtual delivery of EPI (17). The current manuscript addresses aims 1, 2 and 3, and illustrates the application and utility of a practice profile (18), the FRAME framework for identifying and documenting model adaptations and unanticipated impacts (11, 13), and the Consolidated Framework for Implementation Research (CFIR) for identifying barriers to re-implementation (19). Objectives 4 and 5 related to outcomes will be reported separately.

2. Materials and methods

2.1. Design

The study used a mixed methods, pragmatic, implementation and evaluation design described in more detail elsewhere (20). Youth and family members with lived experience, front-line clinicians, and clinical administrators were engaged in a structured, stepwise approach to track adaptations needed to provide NAVIGATE care virtually. Structured approaches were used to evaluate re-implementation outcomes as measured by fidelity, and to explore implementation facilitators and barriers. Throughout this manuscript we refer to “virtual” delivery of care when care is provided via phone or tele/videoconference.

3. Study setting and population

This study was conducted at SCEIS, the outpatient EPI program at the Centre for Addiction and Mental Health (CAMH) in Toronto, Canada. SCEIS serves people aged 14–29 years old who present with early psychosis (schizophrenia, schizoaffective disorder, schizophreniform disorder, bipolar I disorder or major depressive disorder with psychotic features, substance-induced psychotic disorder, unspecified psychotic disorder). Located in downtown Toronto, Canada, this EPI service is staffed by approximately 40 clinicians who assess approximately 600 new clients annually.

The Ontario Ministry of Health provides coverage for all medically necessary services including EPI to residents through the Ontario Health Insurance Plan (OHIP), and this coverage was maintained in the transition to virtual care.

SCEIS provides EPI services according to the NAVIGATE model, a highly structured program of coordinated specialty care with clearly defined roles for staff (21). Initially implemented at CAMH in late 2017, the model consists of four core clinical roles: Individual Resiliency Training (IRT), Supported Employment and Education (SEE), Family Education Program (FEP), and individualized medication management (21). Additional core components that are fundamental to the NAVIGATE program include: Team Lead who facilitates monitoring; Practice Feedback and Training; and Caseloads small enough to allow for the intensity and frequency of required contact. Manualized protocols are used to operationalize current EPI standards, and all clients are systematically offered all treatment components with regular team meetings to review client progress, fidelity, and need for adjustments. All clients receive substance use support as part of the IRT manual. Where there is additional need for substance use support beyond the general manual, clients can receive specialized support from a clinical psychologist at SCEIS or from additional programs at the substance use disorder services at CAMH.

3.1. Stakeholders

This study engaged youth and family members with lived experience, front-line clinicians and administrators according to current best practices (22). Stakeholders contributed meaningfully to the study design, data collection, integration of findings and knowledge dissemination. We held monthly meetings with the principal investigators, operational research staff, youth and family members with lived experience, front-line clinicians and clinical leads (“steering committee”) to review the progress of re-implementation and data collection, and to plan for knowledge dissemination. Monthly “knowledge user meetings” were held during the first phase of the study with front-line clinicians and clinical leads to discuss program modifications and their impacts, barriers to virtual care delivery, clinician resources and training.

3.2. Context: the COVID-19 pandemic

The shift to virtual delivery of care occurred abruptly in March 2020 due to COVID-19-related public health directives to stay at home during the first COVID-19 lockdown in Toronto. The first COVID-19 lockdown lasted from March to June 2020 (with ongoing restrictions persisting to varying degrees until the time of submission) and prompted a hospital-wide transition to virtual delivery for most outpatient services. Exceptions were made to allow in-person appointments for a small number of clients for whom virtual assessment and treatment was not feasible (e.g., clients in crisis and/or requiring a hospital admission, those receiving intramuscular injections, and/or those lacking access to virtual care).

The abrupt shift in the modality of care delivery pre-empted any preparation and planning for this transition. Fortuitously, several facilitating events occurred. Prior to March 2020, CAMH had taken steps towards integrating a digital platform to enhance capability for virtual meetings and enable the use of virtual care throughout the organization. After an extensive process, a digital platform (Cisco Webex) was chosen that met the Ministry of Health's privacy and confidentiality requirements including safeguarding Personal Health Information of clients. Proof-of-concepts in clinical and non-clinical settings had been conducted with this digital platform prior to the COVID-19 pandemic (23). Other enabling factors at CAMH that predated the pandemic included exclusive use of electronic medical records, and the transition to using laptops instead of desktop computers in order to facilitate remote and mobile work.

Once the pandemic triggered the shift to virtual care, CAMH rapidly scaled the deployment of the Cisco communications platform and initiated organization-wide training for clinicians in the use of Cisco Webex and the Ontario Telemedicine Network (OTN), two provincially approved digital platforms for providing virtual care. This training was provided to over 400 CAMH clinicians.

CAMH developed and implemented a virtual care policy and protocol that covered procedures for providing care in a virtual setting such as privacy, confidentiality, documentation practices, and practical instructions for providing virtual care. Subsequently, the Virtual Mental Health and Outreach team developed digital mental health training for clinicians on delivering virtual care in clinical settings. Training content included the context and evidence base for virtual care; clinical experiences; individual and group settings; safety and confidentiality procedures; technology; and the therapeutic relationship in a virtual setting (24). Other tools for facilitating virtual care delivery were made available across CAMH including a communications application allowing for instant messaging and phone calls with other team members and clients (Cisco Jabber); a secure file transfer platform to share files; and applications for faxing and scanning documents remotely. CAMH EPI clinicians were provided with mobile phones to facilitate voice communications and text reminders with clients.

Virtual care was enabled across Ontario by a shift in the Ontario Ministry of Health billing codes and requirements to enable remuneration of virtual care (via videoconference or phone) provided by physicians.

4. Procedures

4.1. Objective 1: modifications

Our approach to documenting modifications included the use of the NAVIGATE practice profile (18, 25) and the FRAME framework (11). A practice profile is a tool for describing the core components of an innovation or model of care, including the principles that underlie the model. Core components are prescribed by the innovation developer but how each core component is executed and by whom is determined by the implementing organization to guide implementation and delivery. Core components are the features of a model or intervention that must be present to ensure that it is delivered as intended to achieve expected outcomes. The profile provides a structure for documenting variations to the innovation as well as implementation outcomes. Once an innovation is described in sufficient detail, effective implementation methods can be applied to explore the organizational functions needed, develop staff competencies, monitor data for continuous improvement and sustainment, and ensure that leadership and administrative practices remain facilitative.

Prior to the pandemic, research team members developed a NAVIGATE practice profile (26). This development took place through an iterative process that included a review of key NAVIGATE manuals and other model documents, published articles from the RAISE-ETP study that developed and first implemented NAVIGATE, as well as feedback from clinicians and implementation specialists familiar with the model (21). A penultimate draft was reviewed by model originators, further revised and finalized. The final practice profile identified seven core components: Individual Resiliency Training (IRT), Supported Employment and Education (SEE), Family Education Program (FEP), Individualized Medication Management, Team Leadership, Practice Feedback and Training, and Caseload (Figure 1). We used this NAVIGATE practice profile to describe and document modifications for each core component in the current study. We adjusted the descriptions of how the components were delivered virtually and added information on mitigation strategies that were taken to facilitate the change or to reduce potential negative impacts and amplify positive impacts of the modifications. Structured reflection sessions were conducted remotely with clinicians in each NAVIGATE role (IRT, SEE, FEP, prescribers, team lead) during the re-implementation process to document modifications and impacts. At each discussion, we monitored challenges, contextual factors, and impact and tracked subsequent modifications or mitigating strategies. From these discussions we were able to identify the reasons modifications were made and at what level they occurred. This method of tracking modifications in structured reflection sessions has previously shown potential as a straightforward and low-burden approach for documenting events across a dynamic implementation setting (27). Sessions with the clinicians and the clinical manager occurred at 2–3 and then again at 12 months into the study. Interim updates by clinician representatives were provided as part of monthly meetings throughout the first year of the study and clinicians representing different NAVIGATE roles reviewed and finalized the modifications described in the practice profile. Barriers identified during the initial group sessions were reviewed by the research team to inform new adaptations for enhancing the re-implementation process.

FIGURE 1

Figure 1. NAVIGATE core components.

Modifications to the practice profile were then coded using the FRAME to document underlying process, rationale and purpose (11). For our context of re-implementation, we added an additional factor to capture the “effects” of modifications. We identified potential and realized positive, neutral and/or negative intended/unintended effects of modifications and described mitigating strategies that were undertaken to lessen negative impacts, if applicable. To document the “reasons” underlying each modification, we added the specifier “COVID-19 pandemic” as the “outer setting context” to indicate why the modification was made. Documenting modifications in response to culture was not applicable to our context, as modifications were not related to the implementation of the intervention in cultures different from where the intervention was first implemented.

4.2. Objective 2: fidelity

Implementation fidelity refers to the extent to which an intervention is delivered as intended by the program developers and in line with the program model (28). In the present study, we used the First Episode Psychosis Services Fidelity Scale (FEPS-FS) to assess fidelity to evidence-based practices for EPI delivery (29).

4.2.1. Fidelity to EPI practices

The FEPS-FS is a validated measure of fidelity to the standards of EPI care (29). Scale development was based on a review of evidence combined with an expert consensus process and is not tied to any specific model of EPI delivery. Thirty-three items are rated on a 5-point scale from “not implemented” to “fully implemented.” A rating of 4 is considered satisfactory adherence. The scale is designed such that the items measure delivery in relation to the core components of the EPI model (adherence); quality of delivery using strategies such as clinician observation is not assessed (30).

The FEPS-FS items assess team structure (integrated approach), client continuity of care (early intervention, retention), and client receipt of medical and psychosocial treatments (comprehensive care). In Ontario, a community of practice for EPI programs, the Early Psychosis Intervention Ontario Network (EPION), developed a process to assess fidelity with this scale using a site visit methodology (26, 31). Fidelity ratings are based on interviews with staff, client chart review and administrative data and are usually made after a 1-to-2-day site visit by independent assessors. In this study, COVID-19 related restrictions required us to assess fidelity remotely via phone/video staff interviews and virtually trained on-site health record abstractors (32).

Fidelity assessments were done twice; retrospectively to capture practice prior to pandemic restrictions, when care was provided in-person (January–December 2019), and after the shift to virtual care delivery (July 2020–June 2021). For each assessment period, 10 client charts were randomly selected for clients enrolled in the program for at least one year during that period. These charts were abstracted by remotely trained on-site staff. Two independent fidelity assessors conducted phone/video interviews with program informants about NAVIGATE delivery during each of these periods. Both at the beginning and throughout each interview, the assessors reminded the participant about the practice period in question. For each period, interviews were held with the team lead, prescriber and 4 clinicians in different NAVIGATE roles. The assessors then reviewed the chart, interview and program administrative data to develop preliminary ratings that were discussed in a consensus meeting with a fidelity expert and then finalized.

Individual item ratings and the total mean score were reported for each period. Item ratings were grouped into one of five domains that pertain to: team structure, access and continuity, comprehensive assessment, medical treatments and psychosocial treatments.

4.3. Objective 3: implementation facilitators and barriers

Facilitators and barriers were captured with a CFIR informed semi-structured interview. The CFIR is a determinant framework of 39 factors known to influence implementation, categorized into five major domains: intervention characteristics; outer setting; inner setting; staff characteristics; and implementation process (30). Since CAMH clinicians had previously implemented NAVIGATE, the CFIR-informed interview focused specifically on the re-implementation of virtual delivery. We included 38 CFIR constructs, omitting cost as the delivery was part of standard care. We interviewed 8 clinicians (IRT, SEE, FEP, prescriber, team lead) by videoconference. Interviews were administered and coded deductively using a variation of the Rapid Analysis (RA) method, an alternative to in-depth analysis of interview data that allows for faster analysis and dissemination of implementation findings while using fewer resources (19, 33). Coding identified facilitators and barriers as well as the direction (valence) and strength of the association between factors and implementation success. For the first analytic step of the RA method, the analysts captured interview comments on a templated summary table in real time. The summary table aligned with the CFIR interview guide (domain and factors). The second analytic step involved assigning a valence rating to each factor to denote a positive or negative influence on implementation (+, neutral, −). Strength of the association was then rated (−2, −1, 0, mixed, +1, +2) and determined by a number of factors, including level of agreement among participants, strength of conviction, and use of concrete examples. In the last analytic step, memos were written to summarize the findings for each factor.

5. Results

5.1. Objective 1: modifications

5.1.1. Cross-cutting modifications

Group meetings with clinicians revealed that three types of modifications needed for the virtual delivery of NAVIGATE were cross-cutting and independent of NAVIGATE core components, while others were unique to a core component. Cross-cutting modifications related to technology, procedures, and training.

Technological modifications included providing hardware and software to clinicians to facilitate remote work (including laptops and mobile phones), and the organization-wide roll-out of Cisco Webex, a digital platform for providing virtual care.

Procedural modifications related to privacy, safety and confidentiality guidelines for virtual care delivery which included obtaining client consent for virtual appointments, Mental Health Act certification procedures (i.e., for involuntary commitment), and changes to physician remuneration for virtual care.

Training modifications involved clinician orientation to new software applications including the digital platform used for virtual care, clinician training on building engagement with clients in the context of virtual care, provided by a youth with lived experience, risk assessment and addressing crisis management with clients in crisis, suggestions for providing trauma-informed care in a virtual setting, and considering health equity in virtual care delivery. Several of the cross-cutting modifications stemmed from decisions made at the organizational level and impacted the whole organization. For instance, changes made to the remuneration for provision of virtual care, a particularly relevant decision, was made at the provincial governmental level (Ministry of Health).

5.1.2. Core component related modifications

We documented 26 modifications related to the four NAVIGATE core clinical roles: 8 modifications for IRT, 5 for SEE, 4 for FEP and 9 for the prescriber role (Tables 1a–1f). Most of these modifications occurred during the onset of the shift to virtual care delivery. About two-thirds of the modifications were unplanned or reactive modifications. Most modifications were made by clinicians and/or the clinical manager and occurred at the clinic/unit level (59%), although one-third occurred across the organization (31%) (Table 1a). Most modifications were unrelated to the content of the intervention (69%) and were consistent to the provincial standards for EPI care (63%). Overall, modifications served to increase or maintain client engagement (34%) and to increase and maintain client retention (28%) and improve feasibility of delivery (19%). Little changes were noted for the three NAVIGATE core components that were not directly related to a clinician role. The team leadership role continued as before the shift to virtual care, though all meetings were held virtually, including supervision and training. One of the functions for the team leadership role captured in the practice profile is community outreach, which includes providing targeted education to health, social service, or community groups. There were few community outreach activities, even before the COVID-19 pandemic, and this did not increase with the switch to virtual care delivery. Regarding training and practice feedback, no significant changes were noted to the onboarding process, other than a modality switch to virtual meetings and adding training on virtual delivery of care. The team meetings continued without changes in a virtual setting, and clinicians met virtually with the clinical lead or substitute weekly. With the switch to virtual care delivery there was an increased demand for training on how to use the virtual applications. Caseload size did not change, though workload increased, and mitigating strategies for the increased workload were captured in the Practice Profile.

TABLE 1a

Table 1a. FRAME Virtual NAVIGATE - Summary of 26 modifications (11). Adapted from: Marshall et al. 2021 (34). All items with an asterisk (*) were added by the current authors.

5.1.3. Modifications for individual resiliency training

Modifications to individual resiliency training (IRT) components occurred early during the shift to virtual care delivery and were largely unplanned and reactive to the shift to virtual delivery (Table 1b). Most decisions about modifications were made by the treatment team and the clinical manager, were fidelity consistent, and served to maintain client engagement or retention. For example, clients were offered shorter but more frequent appointments, appointment reminders were sent more often, and hardcopy worksheets and handouts from the NAVIGATE modules were replaced with fillable PDF files that could be shared with clients on the video screen during appointments. Modifications were intended to maintain delivery of the IRT core components despite restrictions to in-person practice. One advantage mentioned by the IRT clinicians was that they were able to gain insights into client's living situations when they attended via videoconference from home. Disadvantages of providing IRT care via videoconference or phone were a less fulsome assessment of nonverbal cues, and client and clinician challenges with technology, connectivity, and engagement during appointments. An increased workload for IRT clinicians occurred, partly due to training demands, but also related to increased communication with clients and clinicians (e.g., via email).

TABLE 1b

Table 1b. Brief report of 8 Virtual NAVIGATE modifications pertaining to the Individual Resiliency Training (IRT) role based on FRAME elements (11). Adapted from: Marshall et al. 2021 (34).

To mitigate challenges introduced by modifications to IRT, the research team gathered web-based resources (websites, brief videos, mobile phone apps) related to the content of the IRT NAVIGATE modules to enhance client engagement in the virtual IRT sessions. These resources were selected by IRT clinicians and youth with lived experience and shared among IRT clinicians. To mitigate technological challenges, IRT clinicians connected with clients via phone to guide them on how to use the digital platform or encouraged clients to seek digital support from family members. To lessen the burden of training demands on clinicians, the team disseminated weekly, bite-sized information by email with practical tips on technology and procedures related to virtual care delivery and clinician wellness.

5.1.4. Modifications for support for education and employment

Modifications made to the Support for Education and Employment (SEE) component were all unplanned (Table 1c). Modification decisions were mostly taken by the treatment team with some input from the clinical manager and individual clinicians. Most modifications were consistent with fidelity, with the exception of a reduction in clinician visits to community-based education and employment settings. SEE modifications included changes to how SEE clinicians were introduced to clients during IRT sessions, fewer opportunities for community outreach visits due to COVID-19 related restrictions, countered by more opportunities to organize and attend virtual meetings with specialized and local supports at educational institutions (e.g., joint meetings with school counsellors). There was also a shift to focus on skills for participating in remote job interviews and learning strategies for remote schooling.

TABLE 1c

Table 1c. Brief report of 5 Virtual NAVIGATE modifications pertaining to the Supported Employment and Education (SEE) role based on FRAME elements (11). Adapted from: Marshall et al. 2021 (34).

5.1.5. Modifications for family education and support

As with the IRT component, the majority of modifications affecting provision of family education and support (FEP) were mostly unplanned (Table 1d). Modification decisions were mostly made by the clinical team. Planned adaptations included the creation of additional material (e.g., PowerPoint presentation) to support virtual delivery of psychoeducation groups. As with SEE, the process by which FEP clinicians connected with families was adjusted. Advantages of virtual FEP delivery included increased access to care meetings for caregivers and family members. Some family members and caregivers experienced barriers to using the digital platform and/or internet. Also challenging was how best to facilitate effective communication in a virtual group meeting. To address this, FEP clinicians developed and shared a structure for group meetings with all attendees and offered individual appointments as needed.

TABLE 1d

Table 1d. Brief report of 4 Virtual NAVIGATE modifications pertaining to the Family Education role based on FRAME elements (11). Adapted from: Marshall et al. 2021 (34).

5.1.6. Modifications for prescriber

Prescribers were unable to conduct certain activities in a virtual setting as compared to in-person care (Table 1e). This included physical assessments which were postponed early in the pandemic, e.g., monitoring of weight and blood pressure, and assessment of antipsychotic-related movement side effects. To mitigate these challenges and maintain adherence to clinical guidelines, prescribers leveraged community-based resources more often (using local laboratories for bloodwork and community nursing clinics for medication injections).

TABLE 1e

Table 1e. Brief report of 9 Virtual NAVIGATE modifications pertaining to the Prescriber role based on FRAME elements (11). Adapted from: Marshall et al. 2021 (34).

5.1.7. Modifications for the caseload size, team leadership, and training and practice feedback components

Few changes were noted for the three NAVIGATE core components that are not directly related to a clinician role (Table 1f). The team leadership role continued without significant changes, though all meetings were held virtually including supervision and training. Targeted community education decreased, likely related to fewer opportunities for community education as many community events were cancelled/postponed due to the COVID-19 restrictions and educational institutions were busy with the COVID-19 related practicalities including the shift to remote learning with less opportunities for psychoeducation. Training and practice feedback required several changes to the content of the training and practice feedback, e.g., training on virtual care was provided, and practice feedback focussed more on the shift to virtual care delivery and the challenges related to this new method of care delivery. Caseload size did not change, though workload increased significantly for the clinicians and clinical manager due to the added complexity introduced by technology, more frequent appointments, and, anecdotally, improved appointment attendance facilitated by virtual care.

TABLE 1f

Table 1f. Brief report of Virtual NAVIGATE modifications pertaining to the Caseload Size, Team Leadership, and Training and Practice Feedback components based on FRAME elements (11). Adapted from: Marshall et al. 2021 (34).

5.2. Objective 2: fidelity

5.2.1. Fidelity to EPI standards

Table 2 reports item, domain and total fidelity ratings based on the FEPS-FS for two time periods: during 2019, prior to the onset of COVID restrictions and the switch to virtual care delivery, and during 2021, after modifications had been implemented. Of the 33 items in the scale, 4 could not be rated due to lack of data and/or relevance to the Ontario context. For the remaining 29 items, the total mean rating exceeded 4.00 for both time periods, although there were some item level rating differences.

TABLE 2

Table 2. FEPS-FS assessment results^a.

The program structure domain mean score did not change between the traditional in-person and virtual NAVIGATE care delivery and it remained high, at 4.67. The access and continuity domain mean score declined slightly from 3.17 to 3.00. Within this domain, the early intervention item rating decreased from 3.00 to 1.00, indicating an increase in the percentage of clients that were hospitalized prior to entry in the EPI program. The targeted community education item rating also decreased from 2.00 to 1.00, indicating fewer community education sessions were being conducted. The rating for timely contact with referred individual increased with virtual delivery of NAVIGATE care, indicating more clients were seen within 2 weeks of referral. The assessment domain mean score remained high, declining slightly from 4.60 to 4.40 due to lower rating for the initial comprehensive psychosocial assessment item with virtual delivery of NAVIGATE i.e., fewer clients had all components of the comprehensive assessment documented in their consultation note. The medical treatment domain mean and item scores did not change over time and remained high, at 5.00. The psychosocial treatment domain mean score declined slightly but remained high, at 4.50.

5.3. Objective 3: facilitators and barriers

Factors (italicized) affecting virtual EPI delivery are described in Table 3 including their strength and valence. Note that factors were overwhelmingly facilitative, with 10 (27%) showing as mixed. No factors emerged as barriers to re-implementation in this setting and context. Table 3 provides ratings and summaries for each factor.

TABLE 3

Table 3. CFIR Results.

5.3.1. Intervention characteristics

Adaptability (+2) of NAVIGATE to the virtual context was most strongly associated with its re-implementation (see Table 3). Adaptations to ensure that the virtual delivery of NAVIGATE was appropriate and effective included implementing and learning how to use the Cisco Webex platform, providing clinicians with laptops and phones, and converting the intervention manual into PDF fillable forms. Clinicians felt these modifications were very effective and “working great”. One issue that remained unresolved was the transfer of client-rated side-effects completed on an iPad while waiting to see the psychiatrist.

NAVIGATE was originally implemented due to the desire for more organized and coordinated EPI care throughout Ontario (Intervention Source +1). Virtual delivery of NAVIGATE provided advantages in several ways including accessibility (clients able to meet more often), flexibility (scheduling around school and work), and cost savings (e.g., no need for transportation). Some disadvantages included not having a platform for clients to complete a questionnaire before meeting with the psychiatrist, inequity issues for clients who did not have access to virtual care, and challenges for clinicians in reading body language for assessment purposes (Relative Advantage +1).

Although a few clinicians felt the materials and supports were either not supportive enough at the beginning of the re-implementation (e.g., fillable PDF version of the manual, tip sheets) or provided too much information to absorb (lots of documents to read and videos to watch), most felt that they received helpful guidance, information and support from IT personnel as well as from the reflective practice meetings (Design Quality and Packaging +1).

Two intervention factors had mixed ratings. Clinicians felt NAVIGATE was effective for clients, largely based on their experiences and observations shared from other clinicians and clients, as well as their overall understanding of intervention (Evidence Strength and Quality, mixed). A few clinicians mentioned they were knowledgeable about the research evidence underlying the intervention. Yet, most clinicians initially felt doubtful that NAVIGATE would be as effective virtually as in-person. With time, however, they found that it worked equally well with the exception of monitoring side effects, which required face to face interaction.

With respect to Complexity (mixed), some clinicians found re-implementing NAVIGATE for virtual delivery to be difficult, particularly at the beginning, because it had to be done quickly with many details to be worked out (e.g., ensuring confidentiality, privacy). As well, the technology was challenging for some users (e.g., family members). Other clinicians reported that it was “not terribly difficult” or not much extra work to re-implement because they could rely on others “to figure it out”.

5.3.2. Outer setting factors

Provincial best practices and EPI standards were seen as providing a major incentive for the implementation of NAVIGATE (External Policies and Incentives, +2). Somewhat less facilitative was the experience of networking and collaborating with other EPI services via EPI-SET ECHO training sessions, which are intended to inform NAVIGATE practice (16). The ECHO (Extension for Community Healthcare Outcomes) model connects geographically dispersed healthcare providers in online communities of practice with the aim of increasing healthcare access (35). Affiliations with EPION and with other mental health agencies and former places of work also influenced clinicians' work (Cosmopolitanism, +1).

The extent to which NAVIGATE met Client Needs was mixed among respondent clinicians. Most perceived NAVIGATE as valuable to clients and families, based on the positive feedback they received, particularly the structured and team approach to care. However, they also noted that for some clients, the material was daunting and lengthy. Cultural and language differences, clients having comorbidities, and issues accessing the technology were also perceived to be barriers to participating in NAVIGATE. The rapid pivot to virtual delivery also meant there was no time to consult clients about the change. Peer Pressure (0) was perceived as neither a barrier nor a facilitator since no other provider organizations were delivering NAVIGATE at that time of this study.

5.3.3. Process factors

The strongest facilitator for re-implementation was the presence of Formally Appointed Implementation Leaders (+2). Although there was not a lot of pre-pandemic planning, the leaders were viewed as collaborative and helpful. The presence of Champions (+1) and Opinion Leaders (+1) was also facilitative. Clinicians felt that key people who were instrumental in pivoting to the virtual delivery of NAVIGATE worked hard and were collaborative in their approach. They noted several strategies leaders used to encourage and inform clinicians to move to virtual delivery of care including numerous emails, links to training, meetings, and providing opportunities to ask questions as well as encouraging flexibility in the delivery of NAVIGATE. Clients were informed about changes through email discussions. Clinicians further noted that there was no choice but to move to virtual care delivery but made concessions for in-person appointments when it was possible.

Clinicians held mixed opinions about the Executing of the re-implementation. They spoke of the changes as being a “tsunami”. Some clinicians mentioned they were consulted via team “huddles”, problem-solving discussions and opportunities to pose questions to the implementation leaders. Others felt that they were “told” about the changes and that execution was more instructive than collaborative.

The consensus among clinicians was that there was a lack of Planning (mixed) in the move to virtual delivery, which they recognized as unavoidable due to the sudden need to maintain service in the pandemic. Initially, the pivot to virtual delivery was overwhelming. However, clinicians felt that the implementation leaders were the appropriate people to lead the way and that they did their best to make it as easy and smooth as possible. One clinician felt that the SEE role did not receive a lot of guidance. At the time of the interview, most clinicians felt that virtual NAVIGATE had been fully re-implemented.

Opportunities for Reflecting and Evaluating were also mixed. Some clinicians spoke of receiving feedback about what was working and what was not working, as well as statistics about engagement (clients who did not attend their appointment, who they were seeing) that included discussions and reflections on the information shared. Others received informal feedback (i.e., no statistics) and others did not recall receiving any specific feedback about how virtual delivery was going.

5.3.4. Inner setting factors

Structural Characteristics (mixed) of the organization were noted as having both positive and negative influences on re-implementation. A strength was that CAMH is a large, resource-intensive setting where staff were provided with laptops, mobile phones, and rooms for private meetings with clients (virtually as well as in-person). Barriers were that clients were unable to input personal information when using the virtual modules without compromising confidentiality, and improvements are needed to the electronic health record.

CAMH as a setting was also highly facilitative for re-implementation due to its Culture (+2) and Networks and Communications (+2). Clinicians regarded the workplace culture as highly positive, collaborative, warm, healthy, supportive, client-centered, and acknowledged that it impacted positively on the transition to virtual delivery of NAVIGATE. Working together as a team and focusing on delivering the highest quality care possible were perceived as key contributors to the success of the re-implementation. The multiple and continuous channels of communication via emails, virtual team meetings, sharing links to resources, updated policies and problem-solving including communication outside of CAMH with other EPI sites via ECHO were perceived as very supportive. Although the volume of new information and communications was overwhelming, it was generally recognized as necessary to support the transition to virtual delivery within a matter of days.

Within the Implementation Climate, specifically Tension for Change (+2) and Relative Priority (+2) were the strongest facilitators in this domain. Clinicians unanimously noted a high tension for change for NAVIGATE because it provided consistency in delivering care, a holistic and standardized approach, and its multiple roles had a clear scope of practice that benefitted various client needs. The transition to virtual NAVIGATE was unambiguously perceived as the main organizational priority by all clinicians. Competing priorities fell to the wayside and all clinicians fully dedicated their time and attention to the virtual delivery, which contributed to its success.

Organizational Incentives and Reward (+1) were also facilitative with several incentives noted for both clients and clinicians. Client incentives included the convenience of accessing care which increased participation, reduced time, parking and transportation costs, and increased flexibility. Clinicians were strongly motivated by the urgent need to find a way to maintain care delivery in the face of pandemic restrictions. Many also mentioned that their efforts to re-implement were recognized by their clinical manager.

The Learning Climate (+1) at SCEIS was perceived positively and as encouraging of learning and taking on new initiatives. Clinicians valued the availability of multiple learning opportunities, both internally and externally, and the supports provided for participating in these opportunities.

Leadership Engagement (+2) was the strongest readiness facilitator. Clinicians unanimously believed there was support from leadership for the virtual delivery of NAVIGATE. Multiple discussions were held regarding what was needed, special considerations for virtual delivery of care were put in place (e.g., privacy; when in-person was essential, role-specific tasks such as who monitors side effects) and leadership were on board and engaged in the re-implementation process.

Re-implementation was supported by Available Resources (+1) including many sources of information and supports to ensure clinicians had the resources needed to perform their role successfully (e.g., Webex support, Virtual Mental Health and Outreach program, educational services, internal team, etc.). Some clinicians had not received the original NAVIGATE training in the initial implementation and perceived this as a limitation. Clinicians valued getting laptops early in the process, which was essential to the virtual transition, but noted that access to cell phones was delayed.

Experience with Access to Knowledge and Information was mixed as re-implementation did not include formal training due to the rapidity of the pivot. There were, however, multiple resources available to support the transition via links, training videos and emails. The amount of information to be accessed, absorbed and implemented in a very short period of time made the initial experience overwhelming for many clinicians but this improved with time.

5.3.5. Characteristics of clinicians

The most facilitative factor related to the clinicians was their Individual Identification with the Organization (+2). There was a general consensus among clinicians we interviewed that their commitment to CAMH strongly and positively influenced their interest in learning, taking on new initiatives, adapting to change, and providing the best care for clients. It was noted that the transition to the virtual delivery of NAVIGATE ultimately was an exercise in change management and was closely tied to how the employer was perceived.

Clinicians’ Knowledge and Beliefs about the Intervention (+1) was also supportive of re-implementation. Clinicians regarded the NAVIGATE model positively and valued the evidence base and the holistic approach. They viewed virtual delivery as advantageous but some clinicians (i.e., psychiatrists) noted the need to have in-person assessments periodically to have a more accurate sense of the clients' status.

Clinicians reported a sense of Self-Efficacy (+1) in delivering NAVIGATE virtually. For many, this confidence stemmed from feeling effective in the delivery of NAVIGATE in person, which provided a solid basis for the transition to virtual delivery. They felt prepared to deliver NAVIGATE virtually (Individual Stage of Change +1), but also slightly hesitant and overwhelmed at the start given the abrupt transition. With time, there was an increased sense of preparedness with practice and continuous refinement of the online resources to support clinicians. Participants discussed mixed thoughts and experiences related to the transition to virtual delivery of NAVIGATE and alignment with their preferred learning style. Some appreciated the convenience of accessing materials online and learning at their own pace; in contrast, others found it distracting and ineffective to be trained online. Overall, clinicians reported high levels of motivation to make virtual delivery of NAVIGATE work.

5.3.6. Client characteristics

Clinicians believed virtual navigate provided a positive experience for both clients and their families. The virtual delivery of navigate was very advantageous for continuing to access care when in-person care could not be delivered. There were fewer no-shows, increased access to care and better client engagement. A period of adjustment was needed at the start of the transition as clients and their families had to become familiar with the digital platform, as did the clinicians.

The transition to the virtual delivery of NAVIGATE was Perceived as Successful (+2), with the team being able to adapt smoothly to the new demands and to learn and work together as a team. Clinicians unanimously recommended continuing with virtual delivery of NAVIGATE while recognizing that in an ideal scenario, clients would have a choice of in-person or virtual delivery to fit their needs and preferences. Having a virtual delivery option was perceived as a way to increase access to care across the country.

5.4. Stakeholder engagement

The stakeholders, including youth and family with lived experiences, front-line clinicians, and clinical leads, participated consistently and meaningfully throughout the course of this study.

In the initial phases, all stakeholders participated in the grant application and development of the practice profile with front-line clinicians (22, 36). For objective 1, Modifications, front-line clinicians, clinical leads and youth and family with lived experiences participated in monthly meetings to explore and review modifications that occurred during the shift to virtual care delivery. Following these meetings, trainings were organized in collaboration with clinical staff, leadership and youth and family members with lived experience. Youth with lived experiences also contributed to the development of the web-based resources to enhance engagement. Regarding objective 2, Fidelity, feedback from front-line clinicians and clinical leads informed the fidelity ratings. Regarding objective 3, Implementation Facilitators and Barriers, front-line clinicians and clinical leads participated in the interviews.

Furthermore, youth and family with lived experience, front-line clinicians and clinical leads contributed to team discussions on data interpretation and development of a knowledge translation plan and products.

6. Discussion

In this mixed methods study investigating the unplanned shift to virtual delivery of EPI care, we identified several modifications required to deliver the NAVIGATE program virtually by using the NAVIGATE practice profile and the FRAME framework. We discussed the potential impact of these modifications on fidelity and outcomes during structured meetings with clinicians, revised the practice profile, and captured modifications using the FRAME. We then formally evaluated impacts on fidelity to the provincial EPI-standards with a validated assessment tool (FEPS-FS) prior to and after the modifications were made. We investigated implementation facilitators and barriers for the virtual delivery of NAVIGATE with clinicians and identified several contextual factors that were critical to re-implementation of NAVIGATE. To our knowledge, this is the first study to describe a re-implementation process this comprehensively. We summarize overall results and experiences with this re-implementation process, strengths and limitations of the approaches we used, and opportunities and needs for future research.

6.1. Modifications

Regarding the first aim of the study, the identification of modifications needed for virtual EPI care, we identified several cross-cutting and role-specific modifications. Most of these modifications were adaptable, though some challenges were identified that could not be mitigated in a virtual setting (e.g., conducting physical assessment).

Our assessment of the modifications needed to support virtual care delivery is largely similar to two recent studies that also describe the shift to virtual care in early psychosis coordinated specialty care programs (37, 38). McCormick and colleagues investigated the pandemic-driven shift to continue care delivery via videoconference and phone at 23 sites across Texas, US (37). Their results show many sites lacked training, resources, policies and procedures to shift to virtual care, and the challenges that were identified included limited capacity to deliver community-based outreach, family engagement, and vocational support, and difficulties with access and connectivity for clients. Similar to the modifications we identified in our study, organizations provided training to support staff early on in the pandemic, leveraged virtual tools e.g., e-mailing clinical worksheets, sharing mobile apps, and sharing other resources such as videos, and reimbursed virtual care - an important facilitator for virtual care delivery (37). Similarly, Meyer-Kalos and colleagues explored challenges and solutions in the shift to virtual care delivery across several EPI services in the United States, Israel, and China (38). These authors also highlighted the importance of implementing procedures to provide care virtually, adapting appointments times and duration, and adapting materials for digital use. They describe specific challenges and mitigating strategies at the clinician-level per NAVIGATE role, such as challenges for SEE clinicians associated with the COVID-19 constricted labor market and unavailability of outreach visits (38, 39). They described modifications similar to ours, such as shifting focus to practicing skills for remote learning and working and conducting job interviews remotely. Regarding the prescriber role, both our study and Meyer-Kalos’ reported challenges with follow-up for medication benefits and side-effects, and a reluctance by prescribers to make changes to medication, particularly switching to clozapine because it requires monitoring with blood tests that were challenging to obtain during the pandemic (38). Mitigating strategies also overlap across our studies, with increased frequency of appointments and involvement of family members to improve monitoring of medication (38).

There are similarities between the modifications and mitigating strategies we identified in similar studies in the child and youth health mental services sector in Ontario (9). Common strategies included provision of software and hardware, clinician training in software to provide virtual care, adapting materials, offering phone sessions and adding text message-based support to address accessibility issues, development of safety protocols, and breaking sessions into smaller segments to increase client engagement. Clinicians were encouraged to engage in self-care activities and some clinics installed flexible hours of service to accommodate clients' and clinicians' other responsibilities (9). The similarities across settings surfaced several cross-cutting modifications needed for delivery of virtual care as well as specific adjustments related to NAVIGATE role-based core components.

Coding of modifications in the FRAME (11) highlighted that modifications were mainly initiated by the COVID-19 pandemic. Modifications occurred at different levels, ranging from the SCEIS team to the CAMH organization to the provincial government. Decisions underlying the modifications were also made at these levels, by individual clinicians, clinic manager, and organizational leadership. Of note is that, the COVID-19 pandemic did not only trigger this pivot to virtual care but the COVID-19 related effects were wide-ranging, from impacts on the health systems organization, e.g., reduced access to primary care, but also impacting clients in the reduced opportunity for finding work or attending school remotely, which is reflected in several modifications.

6.2. Fidelity

Fidelity assessment with the FEPS-FS revealed that the majority of EPI items (23/29) were rated as satisfactorily or fully implemented, and that the core structure of the NAVIGATE program was strongly preserved despite modifications for virtual delivery. These positive results may be related to the extra training and support clinicians received to facilitate re-implementation from the onset of the pandemic.

Compared to the fidelity assessment of in-person NAVIGATE care, the level of program delivery was maintained for many of the assessed items and improved in several areas in the virtual context. The results for the domain access and continuity were mixed, e.g., item scores on timely contact with the referred individual improved, but more new clients had experienced inpatient psychiatric admissions prior to entering the EPI program, and delivery of targeted community education events decreased. The faster connection to a clinician after referral could reflect improved access to care virtually (fewer missed appointments), reduced clinic waitlist, and greater client flexibility to meet during the daytime (individuals were less constrained by work or school hours). On the other hand, most clients experienced an inpatient admission before their admission to NAVIGATE, and this proportion increased compared to in-person care before the COVID-19 pandemic.

Increased inpatient admission could be related to the COVID-19 pandemic. Worsening mental health symptoms and/or increased substance use during the pandemic (40, 41) could have led to more hospitalizations for psychosis (42) or a decline in visits to the primary care providers who could have otherwise referred for outpatient early intervention care (43). As well, there may have been fewer opportunities for youth to connect with their wider support system, such as teacher or coaches, who might otherwise have detected mental health issues and supported them with finding appropriate supports/early treatments.

Additionally, targeted supports for community-based education and employment also decreased. This was a challenge for the CAMH EPI program before the pandemic because of how hospital-based care is organized. The decrease in educational supports also stemmed from the cancellation or postponement of community education due to COVID-19 restrictions, and educational institutions prioritized COVID-19 related practicalities including the shift to remote learning.

Despite reservations voiced by staff about the virtual delivery of medical care in the FRAME discussions, fidelity ratings for health management in the medical care domain remained high. Fidelity feedback for health management suggested that mitigation strategies were identified such as leveraging alternatives to physical assessment (e.g., measure weight at home or blood pressure at pharmacy or primary care practice). Also, while it is possible that physicians were more cautious about medication management, prescribing remained within recommended guidelines which is what the fidelity review assesses.

Implementation remained high for delivery of psychosocial treatments, which aligns with efforts to sustain client retention by offering different options for connecting, shifting to shorter, more frequent meetings, and synchronously sharing fillable PDFs. As captured in the FRAME, most modifications were described as fidelity-consistent, which is reflected by the “fully implemented” fidelity scores. To our knowledge, there are no other published studies investigating fidelity for a virtual comprehensive EPI care program compared to in-person care. There are, however, several studies that report on treatment fidelity for virtual delivery compared to in-person delivery of a structured psychosocial intervention in other populations. In these publications, there was no evidence that virtual delivery achieved worse fidelity compared to in-person delivery (44–46).

6.3. Facilitators and barriers

CFIR interviews surfaced several factors that facilitated the re-implementation of virtual care. The most salient facilitators were adaptability of NAVIGATE, external policies and incentives, and the tension for change brought on by the COVID-19 pandemic. Implementation leaders were also highly facilitative, despite the abrupt shift and limited time for planning. Workplace culture, clinicians' identification with the organization, and the transition to virtual NAVIGATE becoming a strong relative priority in the organization.

Few barriers were mentioned, but clinicians noted that virtual delivery did not always align with client needs and resources. Some clients found the intervention related material challenging to get through, while others were challenged by cultural and language differences, co-morbidities, issues accessing technology and challenges with adequately monitoring side-effects in a virtual setting.

These results are largely in line with a recent study exploring the pandemic-related transition to virtual care across child and youth mental services in Ontario (9). Using a multi-level mixed method design and CFIR interviews, Danseco identified several facilitators including staff engagement and motivation, provision of enabling software and hardware, leadership support, and training activities (9). Clinicians also mentioned the positive impact of collaboration and having a champion or community of colleagues for learning virtual care delivery together. Barriers in the Danseco study included internet connection issues, lack of resources, and privacy concerns (9). Clinicians also noted fatigue from engaging in online sessions and a feeling of isolation from their colleagues. The authors concluded that overall, many service providers had similar experiences implementing virtual care. With the appropriate support, infrastructure, and resources, many clinicians and clients found virtual delivery of care acceptable and would like to continue using it or having it as an option (9).

Our findings also align with factors associated with implementation success across a diverse array of settings and interventions, including weight management in a large integrated U.S. healthcare system, an e-health application in Norway, and a Canadian study of a maternal and child health intervention undertaken in Mali and Ethiopia (47).

6.4. Re-implementation process

Use of the NAVIGATE practice profile and the FRAME to identify modifications facilitated a structured, explicit and comprehensive assessment of modifications in a dynamic context that could have negatively impacted care delivery (11, 48). Taking stock of modifications to core intervention components is crucial for understanding fidelity and effectiveness outcomes (18). The addition of clinician-reported barriers, mitigating strategies and impacts to our practice profile enabled us to track what strategies were used to reduce potentially negative impacts. This approach tracking and using data “along the way” to inform subsequent adaptations (e.g., updates to training, material) contrasts with more linearly designed studies that conduct fulsome impact assessments prior to refining and evaluating an adapted version of an intervention that is hypothesized to fit better (49). Rapid and iterative assessments of modifications and impacts provided a great advantage to optimizing re-implementation, especially when unplanned modifications could negatively impact outcomes (11, 48). A similar stepwise process of revising/developing policies and workflows, providing training, reflecting/evaluating, and taking steps for further improvement during the abrupt shift to virtual care in the pandemic was also observed in other health care agencies that implemented virtual delivery of care in Ontario (9). Another advantage of using the practice profile was that clarity on the intervention components made it was easy for clinicians to identify where and what modifications were needed and/or had occurred.

A disadvantage of our approach was that some of the FRAME domains overlap with the determinant domains of the CFIR, which is less efficient compared to using one instrument only. Other studies also described an overlap between the FRAME and CFIR and decided to reduce certain items of the FRAME for efficiency (50). Furthermore, several of the components of the Process domain of the FRAME were similar between the modifications and were summarized to lessen redundancy. Additionally, the original FRAME framework does not capture the impact of a modification. We added a category of impact and mitigating strategy to the FRAME constructs because systematic consideration of all potential impacts on a range of implementation and intervention outcomes is critical for further optimization of the intervention (51).

Regarding the fidelity assessment, we measured fidelity to the provincial EPI standards with a validated measure, the FEPS-FS. We intend to measure fidelity to the core components of NAVIGATE by reviewing delivery metrics from randomly selected charts, and report the results of thisin a future paper.

6.5. Strengths and limitations

To our knowledge this is the first study of the re-implementation of a comprehensive early psychosis intervention for virtual care delivery. We investigated modifications, fidelity to EPI standards, and determinant factors which are the first 3 objectives of our larger, mixed-methods study. Previous studies have investigated satisfaction, and facilitators and barriers to virtual care delivery based mostly on interviews with health care providers (52), though some included client experiences (53, 54). Our study presents a more rigorous approach to investigating re-implementation of a comprehensive intervention during an abrupt shift to virtual care initiated by the demands of the COVID-19 pandemic. Here, we report on the first objectives of the study. In a later paper we will describe client's and clinician's experiences and measures of engagement later to provide a fulsome description of the impact of virtual care delivery of NAVIGATE.

The success of our re-implementation may be unique to the COVID-19 pandemic context. COVID-19 related restrictions to social contacts likely triggered a strong motivation to continue care while adhering to these restrictions, leading to a quick pivot to virtual care delivery. Other key facilitating factors may also be unique to this context, including the support provided by CAMH and the Virtual Mental Health and Outreach program specifically, the adaptability of the NAVIGATE program, and other availability of resources such as materials and funds (30).

Furthermore, the switch to virtual care delivery may have unintentionally created disparities in the mental health care system for people with limited or no access to technology or to the private space needed to attend virtual appointments (55). Relatedly, social isolation may be an unavoidable outcome of virtual care delivery that will require further examination to address. Ongoing remuneration for virtual service delivery remains uncertain and will undoubtably be an important consideration to monitor moving forward.

6.6. Future steps

Virtual EPI care has the potential to complement traditional in-person EPI care and improve access in specific contexts, e.g., in remote geographic areas. Improving access to specialty health care is particularly relevant for individuals living in rural and remote communities as they tend to experience poorer health, greater disability, and higher mortality (56). To facilitate equitable care, it will be important to investigate client experiences with virtual care, and to address related barriers stemming from sociodemographic factors that lead to health disparities (55, 57). Following on the results from this work, more research is needed to assess the efficacy and generalizability of virtual EPI care and patient's preferences towards virtual or hybrid care, beyond the COVID-19 pandemic context.

7. Conclusion

In conclusion, re-implementation of NAVIGATE for virtual delivery during the COVID-19 pandemic was rapid, unplanned, and complex. Understanding how re-implementation transpired, involved an exploration of barriers, strategies, and impacts across levels of the organization. This study suggests that a comprehensive EPI program can be re-implemented for virtually delivery while maintaining high EPI standards with the appropriate support, infrastructure, and resources. Virtual delivery of NAVIGATE holds promise for increasing access to effective care for youth with psychosis. Going forward, it will be important to ensure future pivots to virtual delivery for NAVIGATE and other interventions maintain equitable care.

Data availability statement

The original contributions presented in the study are included in the article/Supplementary Material, further inquiries can be directed to the corresponding author.

Ethics statement

The studies involving human participants were reviewed and approved by Centre for Addiction and Mental Health's Research Ethics Board. The patients/participants provided their written informed consent to participate in this study.

Author contributions

WT, NK, AC, AV, DA, SB, LDF, AK, AP, RR, JD, and MB drafted and revised sections of the paper and approved the final version. TA, CB, SB, SJ, AM, AS, and VV made critical revisions to the paper and approved the final version. MB developed the implementation evaluation. DA consulted on the fidelity assessment plan. JD developed the fidelity assessment plan. AC consulted on virtual health care delivery. AV consulted on the development of the project. TA, AM, and VV consulted on the evaluation of engagement of people with lived experience. NK was the principal investigator. MB and JD contributed equally to the paper. All authors contributed to the article and approved the submitted version.

Funding

This study received funding from the Canadian Institutes of Health Research, the Ontario Ministry of Health and University of Toronto Miner's Lamp Innovation Fund - COVID-19 Response as of September 2020 (CIHR Grant # MS2-173082).

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

1. Correll CU, Galling B, Pawar A, Krivko A, Bonetto C, Ruggeri M, et al. Comparison of early intervention services vs. treatment as usual for early-phase psychosis: a systematic review, meta-analysis, and meta-regression. JAMA Psychiatry. (2018) 75(6):555–65. doi: 10.1001/jamapsychiatry.2018.0623

PubMed Abstract | CrossRef Full Text | Google Scholar

2. Kane JM, Robinson DG, Schooler NR, Mueser KT, Penn DL, Rosenheck RA, et al. Comprehensive versus usual community care for first-episode psychosis: 2-year outcomes from the NIMH RAISE early treatment program. Am J Psychiatry. (2016) 173(4):362–72. doi: 10.1176/appi.ajp.2015.15050632

PubMed Abstract | CrossRef Full Text | Google Scholar

3. Anderson KK, Norman R, MacDougall A, Edwards J, Palaniyappan L, Lau C, et al. Effectiveness of early psychosis intervention: comparison of service users and nonusers in population-based health administrative data. Am J Psychiatry. (2018) 175(5):443–52. doi: 10.1176/appi.ajp.2017.17050480

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Bond GR, Drake RE, Luciano A. Employment and educational outcomes in early intervention programmes for early psychosis: a systematic review. Epidemiol Psychiatr Sci. (2015) 24(5):446–57. doi: 10.1017/S2045796014000419

PubMed Abstract | CrossRef Full Text | Google Scholar

5. Craig TKJ, Garety P, Power P, Rahaman N, Colbert S, Fornells-Ambrojo M, et al. The lambeth early onset (LEO) team: randomised controlled trial of the effectiveness of specialised care for early psychosis. Br Med J. (2004) 329(7474):1067. doi: 10.1136/bmj.38246.594873.7C

CrossRef Full Text | Google Scholar

6. Malla A, Schmitz N, Norman R, Archie S, Windell D, Roy P, et al. A multisite Canadian study of outcome of first-episode psychosis treated in publicly funded early intervention services. Can J Psychiatry. (2007) 52(9):563–71. doi: 10.1177/070674370705200904

PubMed Abstract | CrossRef Full Text | Google Scholar

7. Petersen L, Jeppesen P, Thorup A, Abel MB, Øhlenschlaeger J, Christensen TØ, et al. A randomised multicentre trial of integrated versus standard treatment for patients with a first episode of psychotic illness. Br Med J. (2005) 331(7517):602. doi: 10.1136/bmj.38565.415000.E01

CrossRef Full Text | Google Scholar

8. Brown E, Gray R, Lo Monaco S, O’Donoghue B, Nelson B, Thompson A, et al. The potential impact of COVID-19 on psychosis: a rapid review of contemporary epidemic and pandemic research. Schizophr Res. (2020) 222:79–87. doi: 10.1016/j.schres.2020.05.005

PubMed Abstract | CrossRef Full Text | Google Scholar

9. Danseco E, Kurzawa J, Sundar P, Brown J, Huang C. Evaluating the sector-wide implementation of virtual child and youth mental health services in response to the COVID-19 pandemic: perspectives from service providers, agency leaders and clients. Implement Res Pract. (2021) 2:263348952110456. doi: 10.1177/26334895211045690

CrossRef Full Text | Google Scholar

10. Inchausti F, MacBeth A, Hasson-Ohayon I, Dimaggio G. Telepsychotherapy in the age of COVID-19: A commentary. J Psychother Integr. (2020) 30(2):394–405. doi: 10.1037/int0000222

11. Wiltsey Stirman S, Baumann AA, Miller CJ. The FRAME: an expanded framework for reporting adaptations and modifications to evidence-based interventions. Implement Sci. (2019) 14(1):58. doi: 10.1186/s13012-019-0898-y

PubMed Abstract | CrossRef Full Text | Google Scholar

12. Mackie TI, Ramella L, Schaefer AJ, Sridhar M, Carter AS, Eisenhower A, et al. Multi-method process maps: an interdisciplinary approach to investigate ad hoc modifications in protocol-driven interventions. J Clin Transl Sci. (2020) 4(3):260–9. doi: 10.1017/cts.2020.14

PubMed Abstract | CrossRef Full Text | Google Scholar

13. Stirman SW, Miller CJ, Toder K, Calloway A. Development of a framework and coding system for modifications and adaptations of evidence-based interventions. Implement Sci. (2013) 8:65–65. doi: 10.1186/1748-5908-8-65

PubMed Abstract | CrossRef Full Text | Google Scholar

14. Quimby KR, Murphy MM, Harewood H, Howitt C, Hambleton I, Jeyaseelan SM, et al. Adaptation of a community-based type-2 diabetes mellitus remission intervention during COVID-19: empowering persons living with diabetes to take control. Implement Sci Commun. (2022) 3(1):5. doi: 10.1186/s43058-022-00255-9

PubMed Abstract | CrossRef Full Text | Google Scholar

15. Wong S, Hassett L, Koorts H, Grunseit A, Tong A, Tiedemann A, et al. Planning implementation and scale-up of physical activity interventions for people with walking difficulties: study protocol for the process evaluation of the ComeBACK trial. Trials. (2022) 23(1):40. doi: 10.1186/s13063-021-05990-3

PubMed Abstract | CrossRef Full Text | Google Scholar

16. Kozloff N, Foussias G, Durbin J, Sockalingam S, Addington J, Addington D, et al. Early psychosis intervention-spreading evidence-based treatment (EPI-SET): protocol for an effectiveness-implementation study of a structured model of care for psychosis in youth and emerging adults. BMJ Open. (2020) 10(6):e034280. doi: 10.1136/bmjopen-2019-034280

PubMed Abstract | CrossRef Full Text | Google Scholar

17. Fetters MD, Curry LA, Creswell JW. Achieving integration in mixed methods designs—principles and practices. Health Serv Res. (2013) 48(6 Pt 2):2134–56. doi: 10.1111/1475-6773.12117

PubMed Abstract | CrossRef Full Text | Google Scholar

18. Metz A. Practice Profiles: A Process for Capturing Evidence and Operationalizing Innovations. White Paper. National Implementation Research Network. (2016). Available at: https://files.eric.ed.gov/fulltext/ED606401.pdf

19. Nevedal AL, Reardon CM, Opra Widerquist MA, Jackson GL, Cutrona SL, White BS, et al. Rapid versus traditional qualitative analysis using the consolidated framework for implementation research (CFIR). Implement Sci. (2021) 16(1):67. doi: 10.1186/s13012-021-01111-5

PubMed Abstract | CrossRef Full Text | Google Scholar

20. Tempelaar W, Barwick M, Crawford A, Voineskos A, Addington D, Addington J, et al. Adapting evidence-based early psychosis intervention services for virtual delivery: protocol for a pragmatic mixed methods implementation and evaluation study. JMIR Res Protoc. (2021) 10(12):e34591. doi: 10.2196/34591

PubMed Abstract | CrossRef Full Text | Google Scholar

21. Mueser KT, Penn DL, Addington J, Brunette MF, Gingerich S, Glynn SM, et al. The NAVIGATE program for first-episode psychosis: rationale, overview, and description of psychosocial components. Psychiatr Serv. (2015) 66(7):680–90. doi: 10.1176/appi.ps.201400413

PubMed Abstract | CrossRef Full Text | Google Scholar

22. Hawke LD, Relihan J, Miller J, McCann E, Rong J, Darnay K, et al. Engaging youth in research planning, design and execution: practical recommendations for researchers. Health Expect. (2018) 21(6):944–9. doi: 10.1111/hex.12795

PubMed Abstract | CrossRef Full Text | Google Scholar

23. https://www.cisco.com/c/en/us/products/collateral/unified-communications/camh-case-study.html

24. https://digital.camhx.ca/

25. Blase K, Fixsen D. Core intervention components: identifying and operationalizing what makes programs work. ASPE research brief. US Department of Health and Human Services. Washington, DC: US Department of Health and Human Services (2013).

26. Durbin J, Selick A, Langill G, Cheng C, Archie S, Butt S, et al. Using fidelity measurement to assess quality of early psychosis intervention services in Ontario. Psychiatr Serv. (2019) 70(9):840–4. doi: 10.1176/appi.ps.201800581

PubMed Abstract | CrossRef Full Text | Google Scholar

27. Finley EP, Huynh AK, Farmer MM, Bean-Mayberry B, Moin T, Oishi SM, et al. Periodic reflections: a method of guided discussions for documenting implementation phenomena. BMC Med Res Methodol. (2018) 18(1):153. doi: 10.1186/s12874-018-0610-y

PubMed Abstract | CrossRef Full Text | Google Scholar

28. Breitenstein SM, Gross D, Garvey CA, Hill C, Fogg L, Resnick B. Implementation fidelity in community-based interventions. Res Nurs Health. (2010) 33(2):164–73. doi: 10.1002/nur.20373

PubMed Abstract | CrossRef Full Text | Google Scholar

29. Addington DE, Norman R, Bond GR, Sale T, Melton R, McKenzie E, et al. Development and testing of the first-episode psychosis services fidelity scale. Psychiatr Serv. (2016) 67(9):1023–5. doi: 10.1176/appi.ps.201500398

PubMed Abstract | CrossRef Full Text | Google Scholar

30. Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. (2009) 4(50). doi: 10.1186/1748-5908-4-50

PubMed Abstract | CrossRef Full Text | Google Scholar

31. Selick A, Langill G, Cheng C, Addington D, Archie S, Butt S, et al. Feasibility and acceptability of a volunteer peer fidelity assessment model in early psychosis intervention programmes in Ontario: results from a pilot study. Early Interv Psychiatry. (2021) 15(3):480–5. doi: 10.1111/eip.12961

PubMed Abstract | CrossRef Full Text | Google Scholar

32. Addington D, Noel V, Landers M, Bond GR. Reliability and feasibility of the first-episode psychosis services fidelity scale-revised for remote assessment. Psychiatr Serv. (2020) 71(12):1245–51. doi: 10.1176/appi.ps.202000072

PubMed Abstract | CrossRef Full Text | Google Scholar

33. Gale RC, Wu J, Erhardt T, Bounthavong M, Reardon CM, Damschroder LJ, et al. Comparison of rapid vs in-depth qualitative analytic methods from a process evaluation of academic detailing in the veterans health administration. Implement Sci. (2019) 14(1):11. doi: 10.1186/s13012-019-0853-y

PubMed Abstract | CrossRef Full Text | Google Scholar

34. Marshall S, Taki S, Love P, Laird Y, Kearney M, Tam N, et al. The process of culturally adapting the healthy beginnings early obesity prevention program for arabic and Chinese mothers in Australia. BMC Public Health. (2021) 21(1):284. doi: 10.1186/s12889-021-10270-5

PubMed Abstract | CrossRef Full Text | Google Scholar

35. Arora S, Thornton K, Murata G, Deming P, Kalishman S, Dion D, et al. Outcomes of treatment for hepatitis C virus infection by primary care providers. N Engl J Med. (2011) 364(23):2199–207. doi: 10.1056/NEJMoa1009370

PubMed Abstract | CrossRef Full Text | Google Scholar

36. Henderson J, Sword W, Niccols A, Dobbins M. The connections research team. Implementing stakeholder-informed research in the substance abuse treatment sector: strategies used by connections, a Canadian knowledge translation and exchange project. Subst Abuse Treat Prev Policy. (2014) 9(1):21. doi: 10.1186/1747-597X-9-21

PubMed Abstract | CrossRef Full Text | Google Scholar

37. McCormick KA, Sevillano L, Klodnick VV, Lopez MA, Cohen DA. Virtual technology’s critical role in sustaining coordinated specialty care in Texas during the COVID-19 pandemic. Psychiatr Serv. (2022):appips202100429 p. doi: 10.1176/appi.ps.202100429

CrossRef Full Text | Google Scholar

38. Meyer-Kalos PS, Roe D, Gingerich S, Hardy K, Bello I, Hrouda D, et al. The impact of COVID-19 on coordinated specialty care (CSC) for people with first episode psychosis (FEP): preliminary observations, and recommendations, from the United States. Israel and China Null. (2021) 34(3–4):387–410.

Google Scholar

39. Szmulewicz A, Öngür D, Shinn AK, Carol EE, Dow J, Yilmaz N, et al. Impact of the COVID-19 pandemic on the employment and educational outcomes of individuals in a first-episode psychosis clinic. Psychiatr Serv. (2022) 73(2):165–71. doi: 10.1176/appi.ps.202100114

PubMed Abstract | CrossRef Full Text | Google Scholar

40. Horigian VE, Schmidt RD, Feaster DJ. Loneliness, mental health, and substance use among US young adults during COVID-19. J Psychoactive Drugs. (2021) 53(1):1–9. doi: 10.1080/02791072.2020.1836435

PubMed Abstract | CrossRef Full Text | Google Scholar

41. Imtiaz S, Wells S, Rehm J, Hamilton HA, Nigatu YT, Wickens CM, et al. Cannabis use during the COVID-19 pandemic in Canada: a repeated cross-sectional study. J Addict Med. (2021) 15(6):484–90. doi: 10.1097/ADM.0000000000000798

PubMed Abstract | CrossRef Full Text | Google Scholar

42. Chong WWY, Acar ZI, West ML, Wong F. A scoping review on the medical and recreational use of Cannabis during the COVID-19 pandemic. Cannabis Cannabinoid Res. (2022) 7(5):591–602. doi: 10.1089/can.2021.0054

PubMed Abstract | CrossRef Full Text | Google Scholar

43. Stephenson E, O’Neill B, Gronsbell J, Butt DA, Crampton N, Ji C, et al. Changes in family medicine visits across sociodemographic groups after the onset of the COVID-19 pandemic in Ontario: a retrospective cohort study. CMAJ Open. (2021) 9(2):E651–8. doi: 10.9778/cmajo.20210005

PubMed Abstract | CrossRef Full Text | Google Scholar

44. Corona LL, Stainbrook JA, Simcoe K, Wagner L, Fowler B, Weitlauf AS, et al. Utilization of telemedicine to support caregivers of young children with ASD and their part C service providers: a comparison of intervention outcomes across three models of service delivery. J Neurodev Disord. (2021) 13(1):38. doi: 10.1186/s11689-021-09387-w

PubMed Abstract | CrossRef Full Text | Google Scholar

45. Ruble LA, McGrew JH, Toland MD, Dalrymple NJ, Jung LA. A randomized controlled trial of COMPASS web-based and face-to-face teacher coaching in autism. J Consult Clin Psychol. (2013) 81(3):566–72. doi: 10.1037/a0032003

PubMed Abstract | CrossRef Full Text | Google Scholar

46. Woolf C, Caute A, Haigh Z, Galliers J, Wilson S, Kessie A, et al. A comparison of remote therapy, face to face therapy and an attention control intervention for people with aphasia: a quasi-randomised controlled feasibility study. Clin Rehabil. (2016) 30(4):359–73. doi: 10.1177/0269215515582074

PubMed Abstract | CrossRef Full Text | Google Scholar

47. Barwick M, Barac R, Damschroder L. Factors associated with effective implementation: research and practical implications. Implementation science 3.0. Switzerland: Springer (2020). 81–100. Available at: https://www.springer.com/gp/book/9783030038731

48. Moore JE, Bumbarger BK, Cooper BR. Examining adaptations of evidence-based programs in natural contexts. J Prim Prev. (2013) 34(3):147–61. doi: 10.1007/s10935-013-0303-6

PubMed Abstract | CrossRef Full Text | Google Scholar

49. Chambers DA, Norton WE. The adaptome: advancing the science of intervention adaptation. Am J Prev Med. (2016) 51(4 Suppl 2):S124–131. doi: 10.1016/j.amepre.2016.05.011

PubMed Abstract | CrossRef Full Text | Google Scholar

50. Haley AD, Powell BJ, Walsh-Bailey C, Krancari M, Gruß I, Shea CM, et al. Strengthening methods for tracking adaptations and modifications to implementation strategies. BMC Med Res Methodol. (2021) 21(1):133. doi: 10.1186/s12874-021-01326-6

PubMed Abstract | CrossRef Full Text | Google Scholar

51. Kirk MA, Moore JE, Wiltsey Stirman S, Birken SA. Towards a comprehensive model for understanding adaptations’ impact: the model for adaptation design and impact (MADI). Implement Sci. (2020) 15(1):56. doi: 10.1186/s13012-020-01021-y

PubMed Abstract | CrossRef Full Text | Google Scholar

52. Mohammed HT, Hyseni L, Bui V, Gerritsen B, Fuller K, Sung J, et al. Exploring the use and challenges of implementing virtual visits during COVID-19 in primary care and lessons for sustained use. PLoS One. (2021) 16(6):e0253665. doi: 10.1371/journal.pone.0253665

PubMed Abstract | CrossRef Full Text | Google Scholar

53. Donelan K, Barreto EA, Sossong S, Michael C, Estrada JJ, Cohen AB, et al. Patient and clinician experiences with telehealth for patient follow-up care. Am J Manag Care. (2019) 25(1):40–4. PMID: 30667610.30667610

PubMed Abstract | Google Scholar

54. Nanda M, Sharma R. A review of patient satisfaction and experience with telemedicine: a virtual solution during and beyond COVID-19 pandemic. Telemed J E Health. (2021) 27(12):1325–31. doi: 10.1089/tmj.2020.0570

PubMed Abstract | CrossRef Full Text | Google Scholar

55. Crawford A, Serhal E. Digital health equity and COVID-19: the innovation curve cannot reinforce the social gradient of health. J Med Internet Res. (2020) 22(6):e19361. doi: 10.2196/19361

PubMed Abstract | CrossRef Full Text | Google Scholar

56. Leider JP, Meit M, McCullough JM, Resnick B, Dekker D, Alfonso YN, et al. The state of rural public health: enduring needs in a new decade. Am J Public Health. (2020) 110(9):1283–90. doi: 10.2105/AJPH.2020.305728

PubMed Abstract | CrossRef Full Text | Google Scholar

57. De Vera K, Challa P, Liu RH, Fuller K, Feroz AS, Gamble A, et al. Virtual primary care implementation during COVID-19 in high-income countries: a scoping review. Telemed J E Health. (2022) 28(7):920–31. doi: 10.1089/tmj.2021.0377

PubMed Abstract | CrossRef Full Text | Google Scholar