Widespread clinical implementation of the teen online problem-solving program: Progress, barriers, and lessons learned

Objective: We describe the clinical implementation in North America of Teen Online Problem Solving (TOPS), a 10+ session, evidence-based telehealth intervention providing training in problem-solving, emotion regulation, and communication skills.

Methods: Twelve children's hospitals and three rehabilitation hospitals participated, agreeing to train a minimum of five therapists to deliver the program and to enroll two patients with traumatic brain injuries (TBI) per month. Barriers to reach and adoption were addressed during monthly calls, resulting in expansion of the program to other neurological conditions and extending training to speech therapists.

Results: Over 26 months, 381 patients were enrolled (199 TBI, 182 other brain conditions), and 101 completed the program. A total of 307 therapists were trained, and 58 went on to deliver the program. Institutional, provider, and patient barriers and strategies to address them are discussed.

Conclusions: The TOPS implementation process highlights the challenges of implementing complex pediatric neurorehabilitation programs while underscoring potential avenues for improving reach and adoption.

1. Introduction

The United States and other countries invest considerable resources annually developing an evidence base to improve the quality and consistency of health care. However, a gap exists between the identification of promising evidence-based practices (EBP) validated through randomized controlled clinical trials and their introduction into clinical care. Evidence suggests that the vast majority of EBPs never move beyond peer-reviewed publications into everyday clinical practice (1). This so-called “valley of death” has generated increasing interest in implementation science and support for efforts to systematically move EBPs into standard clinical care (2).

Pediatric neurorehabilitation suffers from both a dearth of EBPs and unique challenges around efforts to clinically implement them. Most children with behavioral health care needs following traumatic (TBI) or other acquired brain injuries (ABI), such as stroke or brain tumors, fail to receive behavioral health treatments of any type (3, 4). Factors contributing to this gap include difficulty identifying the role of the TBI or ABI in problems, a paucity of providers who have an appreciation of the complex interplay of brain-based impairments and everyday functioning (5, 6) a lack of liaising between medical specialists and community-based personnel (7, 8), and limited access to medical and mental health providers due to family resources issues such as transportation and insurance coverage. Moreover, adolescents and parents may not want behavioral health care services due to perceived stigma around “mental health” and alternative, non-evidence based “medical” approaches, such as brain-training games (9, 10).

Recent evidence-based guidelines for pediatric ABI (10) have identified Teen Online Problem Solving [TOPS], along with Online Family Problem-Solving Treatment [F-PST] and Counselor Assisted Problem Solving [CAPS], developed by Wade and colleagues and hereafter collectively referred to as TOPS, as a practice standard for treating behavior problems, executive function deficits, and family difficulties for adolescents with brain injuries. More than 20 years of research including five randomized controlled trials, a comparative effectiveness trial, and an individual participant meta-analysis provide substantial evidence of the program's efficacy. Outcomes have included improvements in child behavior, executive functioning, and participation in school, home, and community settings, as well as reduced parent-teen conflict and parent depression and distress (11–13). Much of the support for the program's efficacy is based on studies of self-guided completion of online learning modules coupled with videoconference sessions with a therapist. A comparative effectiveness study investigating this delivery model against face-to-face delivery with no website and self-guided delivery through the website alone, however, suggest that all three modes of delivery could be effective in improving executive dysfunction and quality of life (14, 15).

In 2019, an implementation award from the Patient Centered Outcomes Research Institute (PCORI) sought to examine the clinical implementation of the TOPS program content in these three modalities, considering patient choice, insurance coverage for behavioral health care, and the availability of therapists and broad-band internet access. The RE-AIM (Reach Effectiveness Adoption Implementation Maintenance) framework was utilized to examine the numerous facets of implementation. Specifically for the purposes of this investigation, the focus was on characterizing reach, defined as the number of patients screened, referred, and treated, and adoption, defined as the number of therapists trained and delivering the program (16). Our Reach target was to enroll two patients per site with TBI each month. This goal was extended to four patients per month (two with TBI and two with other conditions) when we expanded to non-TBI diagnoses. For Adoption, our target was to train five therapists per site to deliver the program. Ten sites, including eight children's hospitals and two pediatric rehabilitation hospitals, began implementation in February 2020 and an additional five sites were added in fall 2020.

Our objectives here are to: (1) describe the process of implementation over the initial 26 months including reach and adoption data, considering the unique context of COVID-19; (2) characterize barriers to reach and adoption, as well as strategies for overcoming those barriers; and (3) report changes in reach and adoption over time. In doing so, we hope to promote and inform further efforts to implement EBPs in pediatric rehabilitation.

2. Materials and methods

2.1. Program description and development

The TOPS program includes ten core sessions addressing cognitive reframing, metacognitive strategies, self-regulation, and communication skills in a collaborative problem-solving framework. Adolescents and caregivers were encouraged to complete the program together. However, therapists had the option of working with adolescents individually, depending on family circumstances and parental availability (17, 18). For adolescents with greater impairment, emphasis is placed on helping parents understand how they could better support the child through environmental modifications. Supplemental session content focused on concerns faced by some, but not all, adolescents. See prior publications for a more detailed description of the program and session content (17). The TOPS website included learning modules covering the core and supplemental session content along with a problem-solving component that supported families in working through the five-step problem-solving process to create a step-by-step plan for achieving their aim.

In 2019, ten sites, including eight large children's hospitals and two rehabilitation hospitals were identified as initial implementation sites based on their patient volume and interest in providing active neurorehabilitation to their patients. Investigators at these sites actively solicited therapists at their sites who were interested in receiving training and delivering the program. In February 2020, therapists across sites began training, and in May 2020 began using the TOPS program with patients and families (details about the training process are provided separately below).

To expand reach, we created a parallel transdiagnostic version of the website that could be used with other acquired brain injuries (stroke, pediatric brain tumor, infectious and anoxic insults) and brain conditions (e.g., epilepsy) that were associated with substantial executive dysfunction). We used a multistep interactive process for the transdiagnostic website. In August of 2020, therapists began using the existing TOPS program for TBI with adolescents with diagnoses other than TBI and provided feedback regarding the appropriateness of various content and needed modifications. Between July 2020 and January 2021, we conducted key informant interviews with patients and families with a range of neurological conditions (e.g., brain tumors, epilepsy, stroke) who suggested changes to the website. Between January 2021 and March 2021, we also conducted a survey of medical providers regarding the frequency of neurocognitive and socioemotional concerns across diagnoses. This 31 item survey contained 7 items pertaining to physical symptoms (e.g., fatigue, pain); 7 items pertaining to cognitive-communication symptoms (e.g., concentration/attention, processing speed); 8 items pertaining to behavioral or emotional symptoms (e.g., impulsivity, depression); and 7 items pertaining to social and family functioning (e.g., social rejection, family impact/functioning). Ninety-seven providers began the survey and 34 completed it. Responses indicated substantial overlap in concerns across conditions, including high levels of attention problems, executive dysfunction, and school difficulties, with some differences in the profiles of most problematic concerns for each diagnosis (e.g., greater behavioral dysregulation among those with anoxic injuries).

Together, feedback from therapists delivering the existing program, data from the key informant interviews, and provider surveys informed modifications to the website and program delivery. Improvements to the transdiagnostic website included referring to “brain conditions” rather than “brain injuries”, removing references to “before your injury”, incorporating videos of adolescents and young adults with non-TBI neurological diagnoses, adding an alternate core session on managing fears and worries to address anxiety (as an alternative to controlling your anger), and creating additional supplemental modules. New supplemental content focused on making choices, self-advocacy, initiation, managing medications, and coping with visible and invisible disabilities.

Clinicians also proposed modifications to patient-eligibility criteria to further support successful program use in their setting (19). Thus, although the program was validated with adolescents with TBI between 13 and 18 years of age, we determined that clinicians could enroll both younger (i.e. 12), and older (i.e. 20–22), patients who they thought would benefit. Clinicians could also offer the program to adolescents with intellectual disabilities and comorbid neurological diagnoses if they viewed the family as capable of scaffolding the teen's involvement.

2.2. Therapist training

Therapists were trained during a 7-hour training delivered either in a single, day-long session or two half days. Training was supported by a detailed therapist's manual and therapist's training website. We sought to train a minimum of five therapists at each site to deliver the program. Initial therapists included pediatric psychologists, neuropsychologists, and psychology interns and fellows. Three therapist trainings were held in person and the remaining trainings were conducted virtually due to the pandemic. Training at all ten original implementation sites was completed in spring 2020. Five additional sites were added and trained in fall 2020, with 15 sites enrolling patients beginning in 2021.

To further expand reach, we developed a multiprong strategy for increasing therapist capacity that was tailored to the site. At sites with active psychology or neuropsychology training programs, we developed processes for integrating TOPS into existing clinical rotations for predoctoral interns and postdoctoral fellows, or creating a distinct TOPS rotation. When successful, this process has provided a pipeline of available therapists for TOPS referrals and allowed closer integration of neuropsychological assessment and treatment.

Secondly, we began training speech language pathologists (SLPs) to deliver the program in the Fall of 2020. As described elsewhere (Lundine, Chitwood, & Wade, under review), prior to training SLPs, the PI worked with a team of SLPs to revise the manual and training materials to reframe emotion regulation issues in the context of cognitive communication challenges and develop resources for knowing when substantial mental health or family concerns might necessitate consultation or referral to a mental health provider.

Finally, in areas with few psychologists/neuropsychologists, training was extended to master's level clinicians in social work and other disciplines. Although this approach expanded therapist availability at these sites, it raised additional challenges about finding time in their schedules to receive consultation/supervision around the neurological underpinnings of the child's behavioral health concerns and how to tailor treatment accordingly.

2.3. Plan and metrics

As noted above, we initiated implementation at eight tertiary care children's hospitals and two rehabilitation hospitals. Additional funding from PCORI supported expansion to four additional children's hospitals and one additional pediatric rehabilitation hospital in fall 2020. Initial meetings with each site mapped processes for identifying, screening, and referring patients to the program. We conducted monthly cross-site meetings to discuss barriers and highlight successful implementation strategies beginning in March 2020. These monthly “All Teach All Learn” meetings (20), with the goal of fostering a network of shared knowledge (e.g., templates for chart notes, tip sheets for families for using the website, use of QR codes to link families to the website login), included Site PIs and research support staff (where applicable).

Given findings of comparable efficacy among face-to-face, therapist-guided online, and self-guided only versions of the program (14, 15), implementation sites were encouraged to consider both clinician recommendations and patient and family choice regarding the mode of treatment delivery (15). At the inception of the implementation program, 8 of 10 sites were not offering telehealth services, and consequently patients at those sites could only opt for in-person treatment or self-guided completion of the program online. With the onset of the global pandemic in winter 2020, telehealth practices rapidly evolved (21); by summer 2020, all participating sites were offering telehealth visits. The reintroduction of in-person visits for behavioral health care varied by site and region; however, all sites were again offering in-person visits by spring 2021.

Implementation teams also used the monthly “All Teach All Learn” calls and monthly active therapist calls to identify barriers to implementation and engage in problem solving to develop and review strategies for overcoming barriers. Project PIs also maintained quarterly one-on-one calls with each site beginning in fall 2020 as a strategy to better understand and address site-specific barriers. We also conducted qualitative interviews with a total of three patients and three therapists regarding their experiences completing and delivering the program and barriers that they encountered to further inform our approach to improving patient engagement. Open-ended questions probed difficulties with accessing the website/logging on, understanding and engaging with the online learning module content, and suggestions for improving the program. These data were compiled and analyzed thematically. Strategies to address barriers were identified in a similar fashion. Details, the range of strategies that were identified, and trials are described below. Given the short implementation timeline and considerable heterogeneity across sites and concomitant barriers, we diverged in some respects from the traditional plan-do-study-act cycles associated with quality improvement activities. Specifically, we trialed multiple interventions to improve uptake simultaneously. Additionally, the team at each site determined which strategies to deploy, since not all actions were possible at every site.

To assess initial implementation success, we focused on Reach and Adoption. Reach was defined by number of eligible patients referred and enrolled in the program (target four patients enrolled per site per month). Adoption was defined as the number of therapists trained and the number who went on to deliver the program (target five therapists per site trained to deliver the program). We report data regarding program enrollment (TBI or transdiagnostic), treatment condition (self-guided vs. therapist-supported), and therapist metrics (number trained, number actively treating, trainee involvement).

3. Results

3.1. Implementation reach and adoption

Figure 1 depicts program enrollment from May 2020, when enrollment began, until June 2022. In September 2020, we began enrolling patients with diagnoses other than TBI and in January 2021, we began enrolling at the five additional sites.

FIGURE 1

Figure 1. TOPS program enrollments from May 2020 through June 2022.

3.2. Patient enrollment

We projected enrolling two patients per site per month with TBI, and two patients per site per month with other diagnoses, following expansion to other diagnoses in Fall 2020. New sites joined the TOPS program in January of 2021 and began enrolling patients. As depicted in Figure 1, a total of 381 patients were enrolled in the program over 26 months. Enrollment rates varied substantially over time and across sites, with the total enrolled at the original sites ranging from 9 to 63 and numbers for the additional sites ranging from 9 to 29. Among patients enrolled, 199 (52.23%) had sustained a TBI and 182 (47.77%) had other ABIs or neurological diagnoses including brain tumors, stroke, spina bifida, epilepsy, and brain infection/encephalitis.

3.3. Program adherence and completion

Adherence varied between the self-guided and therapist-guided programs. Of the 205 families enrolled in the self-guided treatment through June 2022, 10 (∼5%) completed the program by finishing at least five sessions. 101 (∼49%) completed no sessions over a six-month period. The remainder (n = 94; 46%) were actively enrolled. Of the 176 families enrolled in the therapist-guided treatment through June 2022, 94 (∼53%) completed the program by finishing at least five sessions. 52 (29.5%) ended treatment before completing five sessions, and 30 (17%) were still actively involved in the treatment.

3.4. Therapist adoption

We planned to train a minimum of five therapists per site across 15 sites (75 total) to deliver the program. From February 2020 to spring 2022, 307 providers were trained to deliver the TOPS program at implementation sites. Provider backgrounds included clinical psychologists, neuropsychologists, and varying levels of psychology and neuropsychology trainees. After extending training to SLPs who worked with this population, 38 SLPs completed training through Spring 2022. Figure 2 depicts the number of therapists and SLPs trained and delivering the program over time.

FIGURE 2

Figure 2. TOPS therapists trained and delivering the program from spring 2020 through spring 2022.

3.5. Therapists delivering TOPS

Of the 307 individuals trained, 66 therapists, including 25 clinical psychologists, 14 neuropsychologists, 13 SLPs, 3 social workers, 2 master's level counselors, and 9 psychology trainees, delivered the program to patients. Reasons for not delivering the program after completing training included: (1) completed the training for other purposes (i.e., trained as support team staff, to provide supervision, for educational purposes, to provide appropriate referrals), (2) moved to a different role or position before having an opportunity to deliver the program, and (3) plan to deliver the program but have not yet had the opportunity. Of note, therapists at non-supported sites often experienced considerable lags gaining institutional approval and support that would allow them to deliver the program.

3.6. Barriers and strategies

3.6.1. Institutional barriers

Quarterly meetings with implementation teams at each site identified a range of potential barriers including: (1) lack of a medical home or established follow-up clinics for many of the patients with conditions of interest, (2) lack of behavioral health care providers for these populations, (3) lack of communication among subspecialties treating these children about neurobehavioral concerns and associated failure to generate referrals, and (4) lack of broader institutional support/reliance on a single champion. Fragmented follow-up care made patient identification challenging, and this was especially true for rural patients who may be less likely to receive follow-up care at all. Additional potential barriers arose from the age range and severity of patients treated; this was particularly evident at the participating rehabilitation hospitals where substantial numbers of patients were extremely low functioning and unable to engage with the program. Catchment areas that crossed state boundaries posed additional barriers to offering the therapist-guided option of the program, because therapists were most often not licensed in multiple states.

Developing a workflow to integrate TOPS into existing clinical care and getting buy-in from administrative staff to schedule appointments that were seen as different and required different steps (e.g., setting up screen sharing and using a different visit type) from typical psychotherapy appointments posed challenges at some sites. At some institutions, the videoconferencing platform that they used for telehealth did not support screen sharing, requiring additional problem-solving around program delivery. Staff/administrative infrastructure to determine insurance coverage, schedule appointments, and process billing was necessary to support families working with TOPS therapists, either face to face or via telehealth. Sites without existing infrastructure for these services had to establish procedures and identify personnel who could perform those tasks.

Awareness of the program was a key issue identified across sites, and this was a particular issue when there was not an established follow-up clinic for these patients, but rather follow-up across various subspeciality clinics without embedded TOPS-trained providers. Potential referral sources also had difficulty recognizing who was eligible and determining who was appropriate to refer, particularly if there were significant comorbid mental health concerns. In the latter situation, anecdotal feedback indicated that they were more likely to use traditional mental health referrals, rather than refer to a novel program with which their institution lacked experience.

Given these Institutional barriers, the following strategies were identified:

• Establish strategies for routinely screening clinic schedules and flagging potentially eligible patients at sites with existing follow-up clinics, including use of EMR search features.

• Promote outreach (including TOPS training) for behavioral health providers embedded in these clinics so that they could support enrollment.

• Identify a coordinator who could screen clinic schedules and flag potentially eligible patients based on age and diagnoses supported this process, although this took time to implement and was not possible at all sites.

• Identify site-specific procedures for streamlining the process of delivering the program and minimizing additional administrative burdens.

• Engage leadership to support broader marketing initiatives to reach out to patients who may not be receiving follow-up care.

3.6.2. Provider barriers

Although substantial numbers of potential providers received the TOPS training at each site, most did not plan to deliver the program because it was not part of their current workflow or role at the site (e.g., functioned in supervisory rather than direct care capacity). Since most of the patients with TBI/ABI were not currently receiving behavioral health services, offering care to them would require additional providers, which was not planned at most sites. COVID-19, and the mental health challenges arising from it, further exacerbated therapist shortages due to increased demand for services and fewer providers due to illness, resignation, or furlough.

Lack of provider buy-in or reluctance to refer patients to something new was also noted. This reluctance was characterized by one provider as follows: “To use a shopping analogy, our goal is to get great bread. We have multiple options we know and trust in our region. A newer brand may come on the market, and it might be tasty indeed, but if the barriers to getting it are basically the same, why switch?” Additionally, some providers expressed reluctance to refer patients to a self-guided program, and wanted to limit referrals to the therapist-involved options. Associated factors included perceptions that the program duplicated existing services or that the program wouldn't work in their setting because it was developed elsewhere. Remembering how to refer appropriate patients and finding time during a relatively brief outpatient visit to describe the program and make the referral constituted additional barriers. Specifically, although families expressed greater interest in the program when the provider had the opportunity to log on and demonstrate it to them, this was often precluded by tightly scheduled visits. Finally, providers often lacked the ability to follow-up and determine whether the family engaged with the program and to understand reasons for nonengagement.

Given these Provider barriers, the following strategies were identified:

• Develop brief presentations and informative fliers that site champions shared with brain injury and other subspecialty clinics, as well as other providers who were likely to come into contact with eligible patients.

• Tailor treatment delivery modes to further address barriers to care such as insurance coverage, therapist availability, and out-of-state residence.

3.6.3. Patient/family barriers

Reasons families cited for declining treatment included the adolescent not wanting to identify as having issues related to their TBI/brain condition or the desire to prioritize another area (e.g., schoolwork, participating in sport). In some cases, these reasons may reflect the ongoing stigma associated with mental health treatment. For other families, general stressors (e.g., parent with cancer) took precedence over the brain condition or the child was already receiving treatment from community providers, and the family was reluctant to start an additional program, even if it was self-directed. The family focus of the treatment also led some families to decline due to high levels of family conflict, lack of parental time, or the desire to focus the treatment on the child. Some families expressed discomfort with the technical aspects of navigating the website or lacked broadband access; whereas others preferred a treatment that did not involve screens or that was more customized to their needs. Although traditional, face-to-face treatment was an option at all sites once pandemic restrictions relaxed, distance and available therapists continued to pose barriers. Families who wanted to work with a therapist were often deterred by therapist unavailability and long wait times for a first appointment. Limitations of insurance coverage, and out-of-state residence posed further barriers to obtaining either of the therapist options. For patients with a specific concern (e.g., anger management), there was reluctance to engage in a more comprehensive program that addressed issues in addition to their concern. Finally, although there was a Spanish language version of the TOPS website, adolescents with Spanish-speaking parents preferred the English version and primary languages other than English or Spanish were not supported.

Approximately one quarter of families who expressed interest in the program failed to follow through by creating a username and password, or received a username and password, but never accessed the program. Barriers identified by referral sources included lost log in information and difficulties updating their password, fatigue with screens/videoconferencing, and being overwhelmed by multiple stressors.

Given these Patient/Family barriers, the following strategies were identified:

• Increase patient awareness and linkages to the program through presentations to stakeholder organizations, such as brain injury and brain tumor support groups. Some sites worked to more closely integrate TOPS delivery into existing programs, making it a step-down program following initial outpatient rehabilitation.

• Link TOPS referrals to neuropsychological evaluations and feedback sessions to establish TOPS as a complement, rather than competitor, to existing services through increased education and collaboration with providers across subspecialties.

• Establish procedures for directly enrolling families during the child's clinic visit, thereby eliminating the gap between agreeing to enroll and creating a login/password. When time permitted, enrolling providers also previewed the sessions and showed parents and adolescents some of the skills they would learn.

• Tailor program delivery to address patient concerns. In practice, this meant shifting the order of session content to maximize relevance. For example, therapists could address emotion regulation or communication before working on executive functioning skills, for example, if that corresponded to patient and family concerns.

4. Discussion

Our implementation of TOPS led to the development of solutions to overcome challenges at the institutional, provider, and patient/family level; such solutions may be relevant to others seeking to implement EBPs for patients with low-incidence conditions and for those interested in providing a complex eHealth treatment program into clinical practice. To our knowledge, this is the first pediatric rehabilitation intervention to be broadly implemented, although efforts have been made to implement other evidence-based pediatric interventions such as Bright IDEAS (22). Our choice to implement through large children's hospitals allowed us reach children with a range of complex neurological conditions who might not otherwise receive evidence-based follow-up care. However, it also introduced complex institutional barriers that were further exacerbated by the COVID pandemic. Follow-up medical care for TBI and other complex neurological conditions can be siloed and focused on presenting concerns, making it difficult to connect patients who could benefit from the program. For example, headaches may be addressed by neurologists, while school issues are evaluated by neuropsychologists. An imbalance between therapist availability and clinical demand limited many sites' ability to offer the therapist-guided option. Patient and family barriers, including ambivalence about receiving behavioral health treatment for their concerns, have posed additional challenges. Anecdotally, the self-guided program may afford youth and families who are not ready to work with a psychologist an opportunity to begin to understand and address some of their concerns. Our collaborative model involving monthly cross-site meetings has been integral in identifying and exploring tailored solutions. Additionally, these meetings created an informal, national network of pediatric brain injury providers, allowing a broader sharing of experiences and resources. Given heterogeneity among our sites, solutions that are vetted at one site can be extended to other sites with similar clinical structures. We offer our experiences to inform future implementation efforts in the field of pediatric rehabilitation.

The COVID pandemic, which coincided with our implementation efforts, has offered unprecedented barriers and unique opportunities. Outpatient clinics were cancelled for a period of time and at some sites, the individuals who were trained to deliver the program were temporarily furloughed due to low patient volume. Institutions, providers, and patients faced demands and financial strains that made the introduction of a new program challenging and often unwelcome. However, the expansion of telehealth and Medicaid waivers to support provision of telehealth across state boundaries in some regions served as a catalyst for the provision of telehealth services at institutions that were previously resistant to such initiatives. Patient and family enthusiasm and engagement with telehealth waxed and waned over the course of the first two years of the pandemic. Although families initially embraced telehealth visits, therapists reported greater family fatigue with videoconferencing over time, particularly when both school and outpatient medical visits/therapies were virtual. As a consequence, some patients embraced the opportunity to resume in-person therapy sessions when restrictions were relaxed. Hybrid delivery models that included some virtual and some in-person visits also improved uptake. As challenges arising from the pandemic continued to evolve, we sought to adapt our approach to address these challenges. For example, we retrained therapists who were trained early in the pandemic and were subsequently furloughed before they could deliver the program and continued to explore program tailoring options to improve patient engagement (e.g., automatic reminders to complete online modules). Ultimately, it is not possible to fully disentangle the positive (e.g., great telehealth access) and negative impacts (e.g., reduced therapist availability, patient disengagement) of the COVID pandemic or the ultimate effectiveness of our responses.

During our initial two years of implementation, our reach was substantially lower than the two patients per site per month with TBI that was initially anticipated. Although COVID resulted in a number of unanticipated challenges and delays in ramping up, implementation within large, complex medical systems required considerable time to establish pathways and procedures before successfully enrolling patients. Inclusion of other neurological conditions with similar neurobehavioral consequences allowed us to substantially expand our reach, nearly doubling the numbers enrolled, and extend evidence-based practices to historically under-served patient populations. Consistent with other multisite implementations there continues to be considerable variability across sites suggesting the possibility to further extend reach as we more broadly implement the outreach strategies that we have identified (23).

Efforts to expand therapist adoption of the program (training and program delivery) have been more successful. As sites have established procedures for screening patients for appropriateness and referring them to the program, newly trained therapists can be more readily connected with patients on the waiting list. Thus, we are less likely to lose potential therapists to moves/completion of training before they have an opportunity to deliver the program or lose patients due to a lack of available TOPS-trained therapists. Positive preliminary feedback from SLPs who have successfully delivered the TOPS program suggests that this may be another avenue for extending program adoption. At another level, the first cohort of fellows who were trained in TOPS have recently begun jobs, potentially supporting further program spread. Although promising, it will take a few years to determine whether those who delivered the program during fellowship continue to use it in their new roles.

Implementation through large children's hospitals that are part of broader academic networks has afforded unique opportunities to extend the program to adolescents with a range of low-incidence conditions. However, it also posed challenges due to the complex and varied systems for tracking and managing children with ABI and other neurological conditions and the multiple subspecialties involved in their care. Heterogeneity across sites necessitated tailoring solutions to the institution, rather than relying on a one-size-fits-all approach. Thus, although successful solutions could be shared through our monthly All Teach All Learn format, adaptation to the local context was frequently required. Specifically, larger hospital systems often had more complex procedures for implementing new programs that required vetting and legal approval at a variety of levels; whereas, smaller systems were more nimble and responsive. Moreover, the pandemic exacerbated institutional challenges arising from understaffing and stretched resources.

At the patient level, we identified program-factors, such as lost logins/passwords and lack of reminders, that can be addressed to support engagement. However, factors such as negative attitudes toward behavioral health treatments or overall fatigue and burnout have proved harder to address. Additionally, complex comorbidities have raised issues among both parents and referral sources regarding how to sequence treatments and where TOPS fits in that sequence.

Our experiences to date should be considered in the context of a number of important limitations. As with any multisite implementation, there was inconsistent fidelity to critical elements and cross-site variability in some procedures. This included variable patient screening procedures, which made it difficult to accurately evaluate patient uptake and reasons for nonparticipation. Tracking of implementation metrics across sites has been variable based on site resources (e.g., a coordinator who can help collect and enter these data). Additionally, as with any multisite quality improvement project, sites varied in the exact timing of introduction of different improvement strategies across sites (e.g., one site trialed something and then when successful others followed). This made it challenging to evaluate the impact of each specific intervention on reach.

In summary, the initial two years of TOPS implementation have illustrated the complexity and challenges of moving an EBP of complex eHealth program into clinical care. They have also underscored the importance of a team-based approach and ongoing trouble-shooting/problem-solving. Future directions include creation of a sustainable online platform for training and extending the program to additional sites. Additionally, development of subsequent pediatric rehabilitation interventions could benefit from thinking about clinical implementation during the intervention development phase, thereby promoting longer-term institutional buy-in.

Data availability statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics statement

The studies involving human participants were reviewed and approved by Cincinnati Children's Hospital Medical Center. Written informed consent to participate in this study was provided by the participants’ legal guardian/next of kin.

Author contributions

All authors assisted in data collection, manuscript conceptualization and preparation and reviewed the final draft. SLW and KW designed the study and obtained funding. All authors contributed to the article and approved the submitted version.

Funding

This research was partially funded through a Patient-Centered Outcomes Research Institute (PCORI) Dissemination and Implementation Award (DI-2018C3-14566). Additional funding came from the State of Ohio, Ohio Development Services Agency, Ohio Third Frontier, Grant Control No. TECG2019-0159 and Cincinnati Children's Hospital Medical Center's Innovation Fund.The statements in this article are solely the responsibility of the authors and do not necessarily represent the views of the Patient-Centered Outcomes Research Institute (PCORI), its Board of Governors or Methodology Committee.

Acknowledgments

We would like to acknowledge the following coordinators who have contributed time and efforts to this study: TA, JR, ED, ES, BL, CD, KC, KH, and ME-S.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

1. Glasgow R, Lichtenstein E, Marcus A. Why don’t we see more translation of health promotion research to practice? Rethinking the efficacy-to-effectiveness transition. Am J Public Health. (2003) 93(8):1261–7. doi: 10.2105/ajph.93.8.1261

PubMed Abstract | CrossRef Full Text | Google Scholar

2. Mensah G, Engelgau M, Stoney C, Mishoe H, Kaufmann P, Freemer M, et al. News from NIH: a center for translation research and implementation science. Transl Behav Med. (2015) 5(2):127–30. doi: 10.1007/s13142-015-0310-7

PubMed Abstract | CrossRef Full Text | Google Scholar

3. Fisher A, Aguilar M, Zhang N, Yeates K, Taylor H, Kurowski B, et al. Caregiver and child behavioral health service utilization following pediatric traumatic brain injury. Res Child Adolesc Psychopathol. (2021) 49(4):491–501. doi: 10.1007/s10802-020-00737-1

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Weinstein R, Krouse R, Holcomb M, Payne C, Erps K, Krupinski E, et al. Telehealth dissemination and implementation (D&I) research: analysis of the PCORI telehealth-related research portfolio. In: Latifi R, Doarn CR, Merrell RC, editors. Telemedicine, telehealth and telepresence: Principles, strategies, applications, and new directions. Switzerland: Springer International Publishing (2021). p. 77–96.

5. Gianinazzi M, Rueegg C, von der Weid N, Niggli F, Kuehni C, Michel G. Mental health-care utilization in survivors of childhood cancer and siblings: the Swiss childhood cancer survivor study. Support Care Cancer. (2014) 22(2):339–49. doi: 10.1007/s00520-013-1976-3

PubMed Abstract | CrossRef Full Text | Google Scholar

6. Fuentes M, Thompson L, Quistberg D, Haaland W, Rhodes K, Kartin D, et al. Auditing access to outpatient rehabilitation services for children with traumatic brain injury and public insurance in Washington State. Arch Phys Med Rehabil. (2017) 98(9):1763–1770.e1767. doi: 10.1016/j.apmr.2016.12.013

PubMed Abstract | CrossRef Full Text | Google Scholar

7. Fuentes M, Wang J, Haarbauer-Krupa J, Yeates K, Durbin D, Zonfrillo M, et al. Unmet rehabilitation needs after hospitalization for traumatic brain injury. Pediatrics. (2018) 141(5):e20172859. doi: 10.1542/peds.2017-2859

PubMed Abstract | CrossRef Full Text | Google Scholar

8. Lever K, Peng J, Lundine J, Caupp S, Wheeler K, Sribnick E, et al. Attending follow-up appointments after pediatric traumatic brain injury: caregiver perceived barriers and facilitators. J Head Trauma Rehabil. (2019) 34(1):e21–34. doi: 10.1097/HTR.0000000000000433

PubMed Abstract | CrossRef Full Text | Google Scholar

9. Miley A, Elleman C, Chiu R, Moscato E, Fisher A, Slomine B, et al. Professional stakeholders’ perceptions of barriers to behavioral health care following pediatric traumatic brain injury. Brain Inj. (2022) 16(4):1–8. doi: 10.1080/02699052.2022.2034956

CrossRef Full Text | Google Scholar

10. Laatsch L, Dodd J, Brown T, Ciccia A, Connor F, Davis K, et al. Evidence-based systematic review of cognitive rehabilitation, emotional, and family treatment studies for children with acquired brain injury literature: from 2006 to 2017. Neuropsychol Rehabil. (2020) 30(1):130–61. doi: 10.1080/09602011.2019.1678490

PubMed Abstract | CrossRef Full Text | Google Scholar

11. Wade S, Cassedy A, Taylor H, McNally K, Kirkwood M, Stancin T, et al. Adolescent quality of life following family problem-solving treatment for brain injury. J Consul Clin Psychol. (2019) 87(11):1043–55. doi: 10.1037/ccp0000440

CrossRef Full Text | Google Scholar

12. Wade S, Carey J, Wolfe C. The efficacy of an online cognitive-behavioral, family intervention in improving child behavior and social competence following pediatric brain injury. Rehabil Psychol. (2006) 51(3):179–89. doi: 10.1037/0090-5550.51.3.179

CrossRef Full Text | Google Scholar

13. Zhang N, Kaizar E, Narad M, Kurowski B, Yeates K, Taylor H, et al. Examination of injury, host, and social-environmental moderators of online family problem solving treatment efficacy for pediatric traumatic brain injury using an individual participant data meta-analytic approach. J Neurotrauma. (2019) 36(7):1147–55. doi: 10.1089/neu.2018.5885

PubMed Abstract | CrossRef Full Text | Google Scholar

14. Kurowski B, Taylor H, McNally K, Kirkwood M, Cassedy A, Horn P, et al. Online family problem-solving therapy (F-PST) for executive and behavioral dysfunction after traumatic brain injury in adolescents: a randomized, multicenter, comparative effectiveness clinical trial. J Head Trauma Rehabil. (2019) 35:165–74. doi: 10.1097/htr.0000000000000545

CrossRef Full Text | Google Scholar

15. Wade S, Kurowski B, Cassedy A, Sklut M, Taylor H, McNally K, et al. The relationship of adolescent and parent preferences for treatment modality with satisfaction, attrition, adherence, and efficacy: the coping with head injury through problem-solving (CHIPS) study. J Pediat Psychol. (2019) 44(3):388–401. doi: 10.1093/jpepsy/jsy087

CrossRef Full Text | Google Scholar

16. Glasgow R, Harden S, Gaglio B, Rabin B, Smith M, Porter G, et al. RE-AIM planning and evalution framework: adapting to new science and practice with a 20-year review. Front Public Health. (2019)7:1–9. doi: 10.3389/fpubh.2019.00064

PubMed Abstract | CrossRef Full Text | Google Scholar

17. Wade S, Kingery K, Narad M, Taylor H, Stancin T, Kirkwood M, et al. Teen online problem solving for teens with traumatic brain injury: rationale, methods, and preliminary feasibility of a teen only intervention. Rehabil Psychol. (2017) 62(3):290–9. doi: 10.1097/DBP.0000000000000489

PubMed Abstract | CrossRef Full Text | Google Scholar

18. Wade S, Taylor H, Yeates K, Kirkwood M, Zang H, McNally K, et al. Online problem solving for adolescent brain injury: a randomized trial of 2 approaches. J Dev Behav Pediatr. (2018) 39(2):154–62. doi: 10.1017/s1355617701766118

PubMed Abstract | CrossRef Full Text | Google Scholar

19. Harvey G, Kitson A. PARIHS Revisited: from heuristic to integrated framework for the successful implementation of knowledge into practice. Implement Sci. (2016) 11(33):1–13. doi: 10.1186/s13012-016-0398-2

PubMed Abstract | CrossRef Full Text | Google Scholar

20. Nembhard I. All teach, all learn, all improve?: the role of interorganizational learning in quality improvement collaboratives. Health Care Manage Rev. (2012) 37(2):154–64. doi: 10.1097/HMR.0b013e31822af831

PubMed Abstract | CrossRef Full Text | Google Scholar

21. Uscher-Pines L, McCullough C, Dworsky M, Sousa J, Predmore Z, Ray K, et al. Use of telehealth across pediatric subspecialties before and during the COVID-19 pandemic. JAMA Network Open. (2022) 5(3):e224759–e224759. doi: 10.1001/jamanetworkopen.2022.4759

PubMed Abstract | CrossRef Full Text | Google Scholar

22. Voll M, Fairclough D, Morrato E, McNeal D, Embry L, Pelletier W, et al. Dissemination of an evidence-based behavioral intervention to alleviate distress in caregivers of children recently diagnosed with cancer: bright IDEAS. Pediatr Blood Cancer. (2022) 69:e29904. doi: 10.1002/pbc.29904

PubMed Abstract | CrossRef Full Text | Google Scholar

23. Goldstein S, Dahale D, Kirkendall E, Mottes T, Kaplan H, Muething S, et al. A prospective multi-center quality improvement initiative (NINJA) indicates a reduction in nephrotoxic acute kidney injury in hospitalized children. Kidney Int. (2020) 97(3):580–8. doi: 10.1016/j.kint.2019.10.015

PubMed Abstract | CrossRef Full Text | Google Scholar