The VA National TeleNeurology Program implementation: a mixed-methods evaluation guided by RE-AIM framework

Introduction: The Veteran Affairs (VA) Office of Rural Health (ORH) funded the Veterans Health Administration (VHA) National TeleNeurology Program (NTNP) as an Enterprise-Wide Initiative (EWI). NTNP is an innovative healthcare delivery model designed to fill the patient access gap for outpatient neurological care especially for Veterans residing in rural communities. The specific aim was to apply the RE-AIM framework in a pragmatic evaluation of NTNP services.

Materials and methods: We conducted a prospective implementation evaluation. Guided by the pragmatic application of the RE-AIM framework, we conceptualized a mixed-methods evaluation for key metrics: (1) reach into the Veteran patient population assessed as total NTNP new patient consult volume and total NTNP clinical encounters (new and return); (2) effectiveness through configurational analysis of conditions leading to high Veteran satisfaction and referring providers perceived effectiveness; (3) adoption and implementation by VA sites through site staff and NTNP interviews; (4) implementation success through perceived management, implementation barriers, facilitators, and adaptations and through rapid qualitative analysis of multiple stakeholders’ assessments; and (5) maintenance of NTNP through monitoring quarterly TeleNeurology consultation volume.

Results: NTNP was successfully implemented in 13 VA Medical Centers over 2 years. The total NTNP new patient consult volume in fiscal year 2021 (FY21) was 836 (58% rurally residing); this increased to 1,706 in fiscal year 2022 (FY22) (55% rurally residing). Total (new and follow-up) NTNP clinical encounters were 1,306 in FY21 and 3,730 in FY22. Overall, the sites reported positive experiences with program implementation and perceived that the program was serving Veterans with little access to neurological care. Veterans also reported high satisfaction with the NTNP program. We identified the patient level of perceived excellent teleneurologist–patient communications, reduced need to drive to get care, and that NTNP provided care that the Veteran otherwise could not access as key factors related to high Veteran satisfaction.

Conclusions: The VA NTNP demonstrated substantial reach, adoption, effectiveness, implementation success, and maintenance over the first 2 years of the program. The NTNP was highly acceptable to both the clinical providers making the referrals and the Veterans receiving the referred video care. The pragmatic application of the RE-AIM framework to guide implementation evaluations is appropriate, comprehensive, and recommended for future applications.

Introduction

There is a national shortage of general neurology physician staff in the Veterans Health Administration (VHA). The fiscal year (FY) 2018 VHA Shortage Occupations Report cites physician staff (medical officer) as the first ranked clinical occupation, and many Veteran Affairs (VA) healthcare systems list neurology among their shortage occupation selections. When VHA access is limited, community referrals are an alternative option if accessible. Since 2014 with the passing of the Veterans Access, Choice and Accountability Act (Choice Act) and the VA MISSION Act (MISSION Act, 2018), Veterans had increased opportunities to receive care from community providers in the private healthcare sector (1). The expenditures for neurological care in the community are staggering, totaling over $27 million in FY18.

With more care options available, VHA developed the Referral Coordination Initiative (RCI) to help Veterans make informed decisions about where to receive their healthcare (1). The RCI teams review Veteran referrals and consults and review VA care options from traditional face-to-face clinic visits at the VA or in the community, VA telehealth visits, or electronic consults (eConsults) by VA specialists who review the patient chart and provide recommendations without the need for the Veteran to drive to a VA location for care (1).

There are 2.8 million rurally residing Veterans enrolled in VA care (2); many of these areas have little to no neurology coverage (both within VA and the community), and access to care is difficult. In a survey of VA facility leadership, over 20% of facilities reported experiencing difficulty with access to community neurology care (3). While VA community referral costs reflect some aspect of this resource need, it is likely that there are missed opportunities and Veterans who simply do not receive this care at all.

The VA Clinical Video Telehealth (CVT) for specialty care, including neurology, has successfully increased Veteran access to specialists. The encounters of all neurology CVT general neurology, as well as specialty clinics [e.g., Parkinson's, amyotropic lateral sclerosis (ALS), epilepsy], have steadily increased over the last few years. However, most of this growth has represented outreach from large VA Medical Centers (VAMCs) to Veterans within that healthcare system, with no nationwide TeleNeurology program providing access to neurology care at more rural systems.

The VHA National TeleNeurology Program (NTNP), funded in 2020 by the VA Office of Rural Health (ORH) as an Enterprise-Wide Initiative (EWI), is an innovative healthcare delivery model designed to fill the patient access gap for outpatient general and specialty neurological care. It harnesses the VHA neurology expertise from urban (VAMCs) to provide access to outpatient neurology care via teleheath for Veterans residing in rural communities in the United States.

VA NTNP components

• NTNP hub facility: Philadelphia VAMC where the medical director and program administrator are located and key administrative functions occur.

• Target spoke facilities: VAMCs that have no or insufficient neurology services and serve a large proportion (>50%) of rurally residing Veterans (as per the rurality calculator).

• Target site services include video telehealth new patient and follow-up consultations using CVT or directly to the Veterans’ homes [VA Video Connect (VVC)].

To understand the implementation process of the NTNP across the VHA over a 2-year period (FY 2021–2022), we evaluated the NTNP implementation using a pragmatic application of the RE-AIM framework (4): reach, effectiveness, adoption, implementation, and maintenance. According to RE-AIM, implementation goals stride toward widespread adoption of the innovation, reach a large number of people as intended, sustain implementation at affordable costs, and produce replicable, effective outcomes.

Recently, the NTNP reported program effectiveness in improved Veteran patient access to outpatient neurology services with reduced wait times and lower monthly volume of post-implementation Community Care Neurology (CCN) consults in NTNP vs. control sites (5, 6). In this evaluation, we used additional mixed-methods data to further evaluate the reach, effectiveness, adoption, implementation, and maintenance of the NTNP across participating VAMCs in the first 2 years of program implementation.

Materials and methods

Design

We conducted a prospective evaluation of the VA NTNP which was implemented in VHA facilities in rural US communities. We used data collected in the first 2 years of NTNP activity, FY2021–2022. Guided by the RE-AIM framework (7, 8), we utilized a mixed-methods evaluation to collect and analyze data in the five RE-AIM domains (see Table 1).

TABLE 1

Table 1. RE-AIM evaluation measures.

Setting VA National TeleNeurology Program

The Department of Veteran Affairs NTNP was funded by the ORH with start-up activities in FY2020 and the first clinical implementation at a VA facility in October 2021. In FY2021, the NTNP was implemented at 12 VAMCs, and it is currently active in 13 sites with one more joining in FY2022. Thus, these analyses included data from sites active in FY2021 and FY2022 with a site sample of 13 VAMCs from across VHA. See Table 2 for site-level contextual data.

TABLE 2

Table 2. The National TeleNeurology Program clinical site adoption.

National TeleNeurology Program

The NTNP provides outpatient neurological evaluation and management through telehealth delivery by video to home (VVC) and video in an outpatient clinic (CVT). Veterans can choose between NTNP and other neurology services for which they may be eligible, including care in the community (CCN) that is paid for by VA. All data analyzed in this project were collected for operational and quality improvement purposes as part of the NTNP ORH evaluation; this project was approved as operational (non-research) by the VA and the Indiana University Institutional Review Board (see signed memo of understanding).

Candidate NTNP sites were identified by examining site neurology FTE, neurology wait times, and patient rurality data. NTNP leadership would then reach out to site leadership to initiate a conversation about interest in NTNP services. Once the process of site exploratory conversations had begun, other sites sometimes self-identified due to difficulties with neurology access.

Participants

We included multiple key stakeholders involved in the management, delivery, and utilization of the NTNP healthcare services for our evaluation. Table 3 displays the participants’ survey disposition across the 13 VAMCs. Nested within the 13 VAMCs were key stakeholders: primary care clinical providers and staff who referred and utilized NTNP services, Veteran patients who sought outpatient care at the sites and were the recipients of the NTNP services for neurological services, and VAMC site leadership and management who facilitated local NTNP implementation. For Veterans with an NTNP or CCN consult placed, we captured demographics (age, gender, race) and rurality (defined by VHA as urban, rural, or highly rural) from Corporate Data Warehouse (CDW) data.

TABLE 3

Table 3. NTNP Veteran patient survey disposition by site.

Measurement

We operationalized the RE-AIM domains and outcomes as follows in Table 1 based upon Glasgow and colleagues’ pragmatic applications of RE-AIM for healthcare (7).

Reach metrics and data

We conceptualized Reach into the Veteran patient population as the total NTNP new patient completed consult volume, extracted from the VA CDW and as the total (new and follow-up) NTNP clinical encounters extracted from the VHA Support Service Center Capital Assets (VSSC) database. We excluded consultations for neurology procedures from the CCN consult counts, since NTNP does not provide procedural consultation (e.g., electromyography and nerve conduction studies, electroencephalography studies). Some of these procedures were classified as consultations rather than procedures and therefore were eliminated via Structured Query Language (SQL) text string searching and manual review of consultation names and descriptions at all sites.

Adoption metrics and data

We measured Adoption by VA sites during pre-implementation and early implementation phases using notes from weekly pre-implementation meetings between NTNP and site leaders and interviews with key site personnel 3–6 months after initial implementation of the program.

Effectiveness metrics and data

We used mixed methods to measure NTNP Effectiveness from several perspectives.

Access

We measured access to care by calculating time in days from the consultation being placed to being scheduled (for all NTNP and CCN consults with a scheduled date) and from consultation placement to completion (for all NTNP and CCN completed consults). We reviewed any consults that remained in pending status from the study time period to determine if they could be classified as either cancelled/discontinued or completed.

Satisfaction

Veteran perspectives on satisfaction and experience with NTNP were collected via telephone interviews conducted within 2 weeks of a completed consult. Veterans who completed an NTNP consult in the first 6 months of NTNP activity at their site were eligible for a patient satisfaction interview. We attempted three calls with each Veteran seen in the first 3 months of program implementation at that NTNP site and a random 50% of those seen in 4–6 months. Questions about satisfaction, similarity of the visit to an in-person visit, and likelihood of recommending a TeleNeurology visit were asked. These questions were individually answered using a 7-point Likert scale where a higher score indicated greater satisfaction. The survey included other questions taken from VA telehealth and other telehealth surveys regarding prior telemedicine experiences, technological difficulties, communication with the neurologist providing care, and the ways in which their TeleNeurology experience did or did not improve access to care (9, 10). These telehealth experience questions were scored on a 5-point Likert scale indicating a degree of agreement with the statement, and higher scores indicated a stronger agreement.

Provider satisfaction was assessed with an emailed survey sent within 1 week of completion of the NTNP consult; if the provider had answered an NTNP survey in the preceding 30 days, the current consult was excluded from the provider survey list. Provider surveys were also completed for the first 6 months after program implementation at each site. Overall satisfaction, whether the consult answered their question, and clarity of the consult were rated on a 1–10 scale, and higher scores indicated greater satisfaction.

Staff perceptions of program effectiveness were assessed during early implementation telephone interviews conducted 3–6 months after the initial implementation began. These interviews are described more fully in the Implementation section that follows.

Implementation

Similar to our assessment of effectiveness, we used mixed methods to assess implementation in a variety of ways, from multiple stakeholders, and at different time points. For all interviews, we obtained verbal consent, audio-recorded, and transcribed the interviews. For email surveys and interviews, participation was voluntary. The VA REDCap platform was utilized to email and receive survey responses from clinical providers who had placed a NTNP consult.

Pre-implementation activities

We participated in all weekly meetings between NTNP leadership and site leadership once a site had determined to implement NTNP. We took meeting notes and reviewed materials shared by the NTNP team including an implementation checklist with updated notes and comments.

Early implementation

We assessed each site's initial experiences with program implementation with staff interviews 3–6 months after implementation. We used the RE-AIM framework to guide development of questions for each type of interview participant (8) as well as a core set of questions across participants. We identified key staff at each site, focusing on site clinical leadership (chief of staff, chief of primary care), operational leadership (primary care general practice manager, telehealth manager), schedulers, and primary care providers. Interviews focused on the site's experiences in NTNP adoption and early implementation, any barriers or facilitators to implementation success, and plans for continued use and maintenance of NTNP.

NTNP leadership

We interviewed all of the NTNP leadership and the participating teleneurologists at the end of FY2021 to understand their perspectives on the barriers and facilitators of NTNP implementation and program goals.

Maintenance

We operationalized maintenance as the continued use of NTNP consultations as assessed by monitoring quarterly TeleNeurology consultation volume over time using CDW data. We also used early implementation site interviews and NTNP leadership interviews to assess perspectives on likelihood of continuing participation (sites) and strategies to support program growth and maintenance (NTNP leadership).

Evaluation analyses

Reach

We obtained quarterly data, including new consultation volume, status, and timing for both NTNP and CCN consults and reported these data back to NTNP as part of our ongoing evaluation. We used Wilcoxon rank-sum tests to compare demographic characteristics of patients with NTNP and CCN consults, excluding patients that had both an NTNP and a CCN consult.

Effectiveness

Accessing metrics of the time in days to schedule a consult (date scheduled minus date placed) and completing a consult (date completed minus date placed) between NTNP and CCN consults were compared using the Wilcoxon rank-sum test, excluding patients that had both an NTNP and a CCN consult. We developed Veteran satisfaction scores as the sum of three key questions (overall satisfaction, recommendation to another Veteran, and likeness to a face-to-face visit), the total score ranged from 3 to 21, and higher scores indicated greater satisfaction. We compared satisfaction mean scores between Veterans with an in-home (VVC) vs. an in-clinic (CVT) NTNP visit using the Wilcoxon rank-sum test with a significance of p < 0.05. Provider satisfaction was similarly measured by using overall satisfaction ratings (1–10, higher score indicating greater satisfaction).

We used configurational analysis to look across all individual cases and identify the crucial conditions that distinguished Veterans who rated “highest” satisfaction, a score of 7 on a 1–7 scale, from those who did not. Configurational analysis is a mathematical, cross-case approach that applies Boolean algebra, set theory, and logic to identify a “minimal theory,” i.e., the key difference-making conditions that consistently and uniquely explain an outcome of interest (11, 12). Configurational analysis searches for necessary and sufficient conditions for an outcome to appear and can detect causal complexity (when several conditions must appear together for an outcome to occur) as well as equifinality (when multiple pathways lead to the same outcome) (13, 14).

To aid with factor selection, we began by substantively reviewing the complete set of factors and identifying 45 factors with a theoretically plausible connection to the “high Veteran satisfaction” outcome. Next, using this analytic dataset of 45 factors, we applied the “minimally sufficient conditions” (“msc”) function within the R package “cna” (39) to inductively analyze the dataset and identify configurations of conditions with particularly strong connections to the outcome (i.e., high clinic participation). This configurational approach to data reduction has been described in detail earlier (15, 16). To briefly summarize, we considered all one-, two-, and three-condition configurations that met pre-designated thresholds for consistency and coverage. Consistency is a parameter related to how reliably a model yields an outcome and is calculated as the number of cases covered by the model and have the outcome present divided by the total number of cases covered by the model. Coverage is a parameter indicating explanatory power and is calculated as the number of cases covered by the configuration and have the outcome present divided by the total number of cases that have the outcome. We began with a consistency threshold of 100% and a coverage threshold of at least 15% to avoid overfitting. If no configurations met this dual threshold, we iteratively lowered the specified consistency level by 5 points (e.g., from 100% to 95%, etc.) and repeated the process. We continued relaxing the consistency threshold until at least two potential configurations that met the specified consistency and coverage thresholds were noted. We then assessed all configurations that satisfied those thresholds.

Using this approach, we identified a smaller subset of factors to use in the model-developing phase of the analysis. We developed models by iteratively using model-building functions within the “cna” software package in R (39). We assessed models based on their overall consistency and coverage, as well as potential model ambiguity (when competing models explain the outcome equally well based on consistency and coverage scores). After a preliminary model was identified, we optimized coverage by reviewing the condition table to consider additional configurations that met consistency and coverage thresholds for facilities with a higher impact that were not explained by models developed thus far. We repeated this method for satisfaction item “Would you recommend TeleNeurology to other Veterans like yourself?” Our final models met an overall consistency threshold of ≥75%, a coverage threshold of ≥60%, and no model ambiguity. The coincidence analysis package (“cna”) in R (39), R (version 3.5.0), RStudio (version 1.1.383), and Microsoft Excel were used in analyses.

Rapid qualitative analyses of implementation data

We used rapid qualitative analysis methods to code pre-implementation meeting notes and early implementation interviews, focusing on elements within domains of the RE-AIM framework (17). We synthesized each site's early implementation interviews, combined with Veteran and provider satisfaction data, into a site report to share with NTNP leadership. We used these site-level early implementation reports for comparisons of RE-AIM elements across sites.

Results

Reach

The total NTNP new patient consult volume was 836 in fiscal year 2021 (FY21) of which 58.1% were rurally residing Veterans and 1,706 in fiscal year 2022 (FY22) of which 55% were rurally residing Veterans. Thus, new patient consults doubled from FY21 to FY22. Total (new and follow-up) NTNP clinical encounters increased almost threefold from 1,306 in FY21 (see Figure 1) to 3,730 in FY22 (see Figure 2).

FIGURE 1

Figure 1. Total NTNP clinical encounters (new and follow-up) in FY 2021. Approximately 58.5% of total NTNP clinical encounters were provided to rural Veteran patients in FY21.

FIGURE 2

Figure 2. Total NTNP clinical encounters (new and follow-up) in FY 2022. Approximately 59% of total NTNP clinical encounters were provided to rural Veteran patients in FY22.

Adoption

In FY2021, the NTNP was successfully implemented in 12 VAMCs; six sites had limited local VA outpatient neurology care, and six had none. The time from initial site meeting to program launch averaged 92.3 days, with a range from 21 to 197 days. In FY2022, an additional VAMC (without any local VA outpatient neurology) adopted the NTNP for a total of 13 VAMCs over 2 years in the VHA across the United States with one of those sites halted in the second year since they no longer had neurology access challenges (see Table 3).

Effectiveness

We conducted Veteran surveys among those who had completed a NTNP virtual visit to determine effectiveness. Of the 648 Veterans eligible, 301 (46.5%) completed a phone interview (see Table 3). Approximately two-thirds of the Veterans surveyed had prior experience with a telehealth visit (see Table 4); approximately half preferred to get their healthcare from the VA, and half preferred to get their healthcare from a mixture of the VA and the community (see Table 5). Almost no Veteran reported that they exclusively preferred community care (0%–8%). Prior telehealth experience varied by site from 38% to 92%. In response to the question, “Why did you choose TeleNeurology service?”, the most frequent responses across the sites were as follows: due to its shorter wait time (35.1%), other reason (35.1%), shorter drive time (22.4%), prefer the VA (4.6%), and perceive VA care to be of higher quality (2.9%) (see Table 6). Overall, sites reported positive experiences with program implementation and that the program was serving Veterans who otherwise would find it difficult to get neurologic care. Veterans also reported high satisfaction with the NTNP program. Mean satisfaction scores (range 1–7) were high across all sites. Combining sites, the overall satisfaction mean score of the program was 6.3 (SD 1.2).

TABLE 4

Table 4. Patient survey: telehealth experience.

TABLE 5

Table 5. Healthcare preferences.

TABLE 6

Table 6. Patient survey: main reason.

Comparison of patient satisfaction by VVC with CVT TeleNeurology visits

In Table 7, we compare the patient satisfaction between the two models of TeleNeurology visits on telepresence (e.g., hear, see, and understand the virtual provider). Overall, Veteran patients attending both NTNP models were satisfied with the telepresence of the visit. However, the Veterans’ ratings of the level of agreement significantly differed on two aspects of the visit. First, Veterans completing a CVT visit reported a higher level of agreement [mean = 4.72 (0.72)] with the statement, “I was able to hear the provider clearly by video,” compared with those completing a VVC visit [mean = 4.45 (0.92)], p < 0.02. Second, Veterans who had completed a VVC visit agreed more with the statement that telehealth reduced the need to travel long distances to meet with the provider [mean = 4.69 (0.84)] compared with those who had completed a CVT TeleNeurology visit [mean = 4.36 (1.06)], p < 0.02. Veterans who received either a VVC or CVT TeleNeurology visit highly agreed that they could clearly see the provider, could ask the teleneurologist questions, got an appointment on day and time that worked for them, and believed that they received neurological care that they would not have been able to access otherwise.

TABLE 7

Table 7. Comparisons between VVC and CVT TN visits.

Qualitative responses

The majority (67%) of the Veterans’ qualitative comments about why they would or would not recommend NTNP were categorized as positive. Most Veterans appreciated the access to care and reduced need to travel provided by telehealth visits:

A great way (Direct telehealth Video to Veteran patient) to communicate; don’t have to travel and can do it on your own time; was a great way to figure out some things and to get some answers; it worked out great and was a good experience. [Veteran patient 521—VVC]

It [TeleNeurology] saved me from driving 100 miles to the next VA …. [Veteran patient 126]

She [Teleneurologist] is the best I’ve ever had. I’ve had other doctors over telehealth, but this was the best. I already recommended her specifically. I would recommend the program because of her [Teleneurologist]. The way she conducted herself, and talked to me and asked me questions, she was not like, ‘Hey, this is your only option,’ she gave me 47 options. And then went through all of them and described whether one was better than the other, etc. She’s very good at what she does. [Veteran patient 651]

Doctor was very caring and interested. Didn’t rush. Wanted to do something for me and wanted to help. [Veteran patient 637]

Negative themes (9%) included a general preference for in-person healthcare and some mistrust around the completeness of the physical assessment via telehealth:

I prefer to be seen in person; I am more of a “hands-on, face-to-face person. [Veteran patient 381]

The doctor had to observe my gait and tremors that I suffer from, and he was doing it over a telephone [video camera]. I think it worked out, I just don’t know how well he was able to capture that from the phone. Definitely would have been able to witness that more in person. [Veteran patient 413]

An additional 24% of Veteran responses about their satisfaction ratings were deemed as neutral. That is, the Veterans expressed both positive and negative comments concurrently often stating the specific encounter was positive, but nonetheless, they prefer in-person visits:

…They (Teleneurologist) treated me with respect, and it was not a troublesome visit at all. I like person to person better, but that’s just me. I didn’t feel like anything was missing from this appointment that would have happened in person, I just think in person it would have been more comfortable for me. But I’m not against telehealth. [Veteran patient 566]

Configurational analyses

To further understand Veteran factors related to their high satisfaction with NTNP, we conducted configurational analyses. The first configurational model was to identify factors related to Veterans’ rating of overall satisfaction as maximal (score of 7). In this dataset, there were 168 Veterans who reported a 7 vs. 104 Veterans who did not.

The identified model was a single pathway composed of three conditions, all of which had to be present the following: Veterans agreed at the maximum level (five out of five) with the statements that they could ask questions of the teleneurologist, that the teleneurologist listened to them during the video visit, and that telehealth reduced the need to travel long distances for care. This model had 77% (130/168) consistency and 80% (130/162) coverage.

The second model was designed to identify factors related to Veterans’ rating of a maximum score of 7 for the item “Would you recommend TeleNeurology to other Veterans like yourself?” In this dataset, 201 Veterans who reported a 7 for Recommend vs. 85 Veterans for all else were identified.

This model had two pathways

Veterans agreed at the maximum level (five out of five) that telehealth reduced the need to travel long distances for care, or they agreed at the maximum level (five out of five) with the statement that their provider explained things in a way that was easy to understand and agreed at the second-highest level (four out of five) with the statement that telehealth reduced the need to travel long distances for care. This model had 83% (171/205) consistency and 90% (171/189) coverage.

Site providers

Clinical providers from the spoke sites who utilized NTNP for their patients had completed surveys about their experiences. The majority of providers (95.8%) reported they would recommend TeleNeurology care for other Veteran patients. Moreover, the clinical providers reported that the NTNP consult addressed their neurologic questions (mean = 9.0, SD = 1.7), the neurologic plan was clear (mean = 9.0, SD = 1.6), and the referring providers were overall satisfied with NTNP services (mean = 8.9, SD = 1.7) (see Table 8).

TABLE 8

Table 8. Provider consult completion survey.

Implementation

NTNP leadership interviews helped identify program aspects that were innovative and viewed as critical to NTNP success, including:

• Establishing the buy-in from the spoke leadership and collaborating with general practice managers.

• Virtually embedding teleneurologists within sites to alleviate inefficiencies of interfacility consults.

• Establishing NTNP as a clinic within the spoke site so teleneurologists directly chart into the spoke site Computerized Patient Record System (CPRS) environment using WebVRAM to improve communication and continuity of care.

• Creating virtual Teams channels for each site and their assigned teleneurologist(s) as a direct and secure communications platform to help “…bring to light problems as well as how to solve [those problems” [NTNP 1].

• Holding NTNP teleneurologists’ team activities like consistent meetings and a journal club, and a clearly shared purpose seemed to positively contribute to a sense of teaming among clinical members. Most expressed strong satisfaction working in the program; one sample quote from a program teleneurologist: “It [NTNP experience] has been wonderful. The whole team is very well connected, everyone is really helpful. The ancillary support as far as connectivity and technology, and anything else we need is there” [NTNP 2].

Challenges noted by NTNP leadership included:

• Attaining site readiness to use the scheduling software, Telehealth Management Platform (TMP), resulting in a shift in initial NTNP implementation focus to providing technological support (e.g., a staff dedicated to training spoke site staff, writing SOPs, and providing feedback to the TMP national team so they could deploy patches as needed).

• Tailoring NTNP to needs at sites where some neurology resources were already available, especially integrating NTNP consult workflow and scheduling into existing neurology consult review and scheduling practices which considerably varied between sites, thus requiring ongoing communication and adaptations at the site level.

• Enhancing site awareness of the availability of the NTNP tools.

The success of the program has helped spur program enhancement in several directions to address identified neurological care resource needs (e.g., number 1 referral for NTNP is for Veterans with headaches). This includes a patient education nurse clinic for headache pilot and neurology brownbag series provided by teleneurologists based on topics chosen by spoke sites. Developments have been pursued due to the enthusiasm of NTNP members; however, moving forward, a caution is to not grow too rapidly and/or overtax staff.

Challenges to implementation included scheduling software utilization, schedulers’ turnover onsite, and variation in the site interpretation of the VA RCI. Facilitators for site implementation included adoption of a uniform scheduling process, coordination of care in real time using Microsoft Teams chat across NTNP, and creation of Teams channels for each site and their assigned teleneurologist(s), establishing NTNP as a clinic within the spoke site so teleneurologists directly chart into the spoke site CPRS environment to improve communication and continuity of care. We also found that NTNP activities including consistent meetings with time allotted for attendance and a journal club and a clearly shared purpose among TN staff seemed to positively contribute to a sense of teaming among clinical members.

Several site adaptations were reported: establishment of a specific RCI process to facilitate NTNP initial implementation and use of “single pathway” neurology consultation to simplify the provider process and ensure that the Veterans are aware of all possible choices for neurology care.

Maintenance

The NTNP maintenance was evident over time as new visits doubled and total encounters increased threefold from FY21 to FY22. Moreover, FY22 growth was significant both in the clinical and the administrative teams with hiring of over 17 Full Time Equivalent Employee (FTEE) to meet the increased demand in services. The NTNP team now includes 20 active neurologists, seven nurses, one social worker, one clinical pharmacy practitioner, four advanced telehealth technicians, a program manager, an administrative officer, a chief technology officer, a senior teleneurologist–consultant, a lead/supervising neurologist, and a medical director.

All sites in FY22 expressed their intention to continue in NTNP. Facility leadership at the four sites surveyed in FY22 thus far all (100%) indefinitely expect to need NTNP services and are open to some form of cost-sharing with the program in the future. Leadership mentioned Veteran satisfaction, continuing patient/facility demand, and workload credit as being key factors when considering payment for the program.

Discussion

Our implementation evaluation demonstrated that the VA spoke sites adopted, implemented, and sustained the NTNP video consult services for outpatient neurology care across the VHA targeting VA facilities which serviced Veterans residing in rural communities with limited or no access to local neurology specialty care services either in the VA or out in the community. NTNP had successfully reached into the targeted rural Veteran communities and the VA rural facilities as new patient consults doubled and total clinical encounters tripled over a period of 2 years.

Establishing the local site leadership buy-in during pre-implementation and embedding the teleneurologists’ clinics within the local clinical flow along with establishing virtual platform communications for spoke site clinicians and staff to have direct access to the teleneurologists facilitated the outpatient TeleNeurology implementation. Site readiness to use specific scheduling software and tailoring NTNP workflow to sites with local neurology presented challenges. The innovation of embedding NTNP by setting up virtual local clinics within each site facilitated the implementation and providing specialty care access to rural patients’ enhanced sustainment. Over a 2-year period, NTNP was adopted, implemented, and sustained across rural facilities in VHA. We found executive leadership support and spoke site communications as important facilitators to successful implementation. Indeed, some leadership expressed their willingness to cost-share with NTNP.

The NTNP addresses the VA facility directors’ gaps in specialty care access reported in 2021 (16) by efficiently harnessing the VA neurologists located at urban VA facilities and coordinating their VA virtual outpatient TeleNeurology clinics in rural VA facilities or among sites with limited or no neurology services through telehealthcare delivery. Telehealthcare delivery models redistribute urban resources to the rural patients and communities. Similar to what the virtual cohesive team of teleneurologists built to deliver acute stroke care to VA rural facilities in the VA TeleStroke program (5, 18), NTNP has developed a virtual team of teleneurologists who report high satisfaction with their NTNP organization and with providing VA virtual outpatient neurology services.

Both the spoke sites’ referring providers and the Veteran patients who received NTNP services reported high satisfaction with NTNP. Veterans who received NTNP services at a VA Community-Based Outpatient Clinic (CBOC) through CVT reported similar satisfaction with NTNP compared with Veterans who received direct video telehealth (VVC) except on two areas: Veterans who received CVT reported hearing the providers significantly better than those receiving VVC, and Veterans who received VVC reported that they did not have to drive for miles to see their provider more so than those who received CVT. Our results are in concert with previous research on direct video telehealth visits for outpatient neurology in a single healthcare institution (19) and with CVT for pharmacy outpatient care at VA CBOCs where patients were highly satisfied with CVT visits in comparison with face-to-face visits (20).

The significance of the impacts of the successfully implemented NTNP is enormous. This NTNP telehealth delivery model provided Veteran patients access to acceptable VA specialty care where neurology specialty care both in VA and out in the rural communities (non-VA) were extremely limited or otherwise not available (3). This is important as the Veteran patients who received NTNP services reported that at least 50% preferred to receive VA care and another 50% preferred to receive a mixture of VA and community care. Telehealth virtual delivered care enables the Veteran to receive multidisciplinary guideline concordant care in rural communities. Furthermore, the NTNP services provided the VA RCI located at each VA facility with an additional VA healthcare option for sourcing VA outpatient neurology consults placed by VA primary care clinicians to provide efficient and timely access to high-quality specialty care. Thus, NTNP enhanced the VA RCI coordinator's ability to provide VA care to the Veteran residing in rural communities. This additional VA option for specialty care may be cost-effective for the VHA organization and the Veteran patients. Future research will need to evaluate the cost-savings to determine the cost impacts.

We further examined sets of patient conditions related to high patient NTNP satisfaction using advanced configurational analyses (21). Veterans perceived strong doctor–patient communication skills by the teleneurologists, the reduction of the need to drive long distances for healthcare, and the perceptions that NTNP provided access to neurology care that the patient would not otherwise have as factors related to the highest level of Veteran patient satisfaction with NTNP services.

Similar to other large-scale telehealth program implementation evaluations (22), we applied the RE-AIM framework to guide our NTNP evaluation of the multiple components, reach, effectiveness, adoption, implementation, and maintenance, across the national VHA system. To extend the evaluation beyond the implementation proportions and rates, we intentionally chose to apply the expanded pragmatic application of RE-AIM for healthcare by Glasgow and Estabrooks (7). The pragmatic application includes the local context in which adoption, implementation, and maintenance occur, an in-depth evaluation for whom the program reached, which organizations/users implemented the program, implementation barriers and facilitators, and how the program was adapted and maintained. Moreover, our team prospectively extracted deductively RE-AIM elements from qualitative interviews and observations throughout the longitudinal evaluation. We recommend this prospective planning and data extraction to comprehensively capture the framework elements. The results demonstrated the robustness of the pragmatic application of the RE-AIM framework for implementation evaluations, and future comprehensive implementation evaluations should include the expanded, pragmatic RE-AIM application.

Limitations

Our evaluation of the NTNP had several limitations. First, we limited our sample of Veteran patients who received NTNP services to a site's first 6 months of implementation. Moreover, we reduced the sample by 50% during months 4–6. It is possible that the Veterans’ satisfaction may have changed over time. Nonetheless, we randomized the 50% sample to reduce bias. Furthermore, according to the qualitative analysis, the 9% of Veterans who reported negative perceptions admit that their NTNP experience was positive; however, the negative perceptions were due to the patients’ beliefs that it was necessary to see the clinician face to face for a neurologic visit. Second, our sample of spoke site clinicians was less than 100% of those utilizing NTNP. It is possible that the experiences of the clinician who did not complete a survey or interview differed from those expressed in this evaluation. However, the site clinicians’ evaluations of NTNP were consistently positive across the 13 NTNP sites located across VHA.

Conclusions

The VA NTNP demonstrated substantial reach, effectiveness, adoption, implementation success, and maintenance over the first 2 years of the program. The NTNP was highly acceptable to both the clinical providers making the referrals and the Veterans receiving the referred video care. Satisfaction with NTNP was related to strong TeleNeurology provider–patient engagement and perceptions that the NTNP provided access to specialty care that Veterans could not otherwise access. The pragmatic application of the RE-AIM framework guided an expanded robust evaluation; therefore, we recommend the pragmatic RE-AIM framework for future applications of implementation evaluations.

Data availability statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics statement

The evaluation was deemed as non-research by the NTNP and the VA.

Author contributions

LW, TD, JW, and HM guided the development of the evaluation. TD developed and edited the manuscript. JW, SS, and RI developed and directed the NTNP and edited the manuscript. HM and GB managed the qualitative data collection and analyses. EM conducted the configurational analysis and edited the manuscript. LP edited the manuscript. QT, ST, JD, and LM abstracted the administrative and clinical data and analyzed the data. RI and AN managed NTNP adoption and the teleneurologists and edited the manuscript. All authors contributed to the article and approved the submitted version.

Funding

Funding was provided to the NTNP (Wilkinson, PI) by the VA Office of Rural Health. The EXTEND QUERI Center is funded by the VA HSR&D QUERI program (QUE 20-010, TD/LW), and TD is supported by a VA HSR&D Research Career Scientist Award (RCS-19-002).

Conflict of interest

HM was employed by the Regenstrief Institute, Inc. TM, EM, LM and LW were affiliated scientists with the Regenstrief Institute, Inc. but did not receive any salary or funding for this evaluation.

The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Author disclaimer

This work was prepared while LP was employed by the South Texas Veterans Health Care System. The opinions expressed in this article are the author’s own and do not reflect the view of the National Institutes of Health, Department of Health and Human Services, or the US government.

References

1. Mattocks KM, Cunningham KJ, Greenstone C, Atkins D, Rosen AK, Upton M. Innovations in community care programs. Policies and research. Med Care. (2021) 59(Suppl 3):s229–31. doi: 10.1097/MLR.0000000000001550

PubMed Abstract | CrossRef Full Text | Google Scholar

2. U.S. Department of Veteran Affairs, Office of Rural Health (2022). Available at: https://www.ruralhealth.va.gov (Accessed December 29, 2022).

3. Mattocks KM, Kroll-Desrosiers A, Kinney R, Elway Anashua R, Cunningham KJ, Mengeling MA. Understanding VA’s use of and relationships with community care providers under the mission act. Med Care. (2021) 59(Suppl 3):S252–S258. doi: 10.1097/MLR.0000000000001545

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Glasgow RE, Harden SM, Gaglio B, Rabin B, Smith ML, Porter GC, et al. (2019). RE-AIM planning and evaluation framework: adapting to new science and practice with a 20-year review. Front Public Health (2019) 29(7):64. doi: 10.3389/fpubh.2019.00064

CrossRef Full Text | Google Scholar

5. Williams LS. The VA TeleNeurology Program. Presentation at the Bi-annual VA Health Services Research and Development Conference in Baltimore, MD (2023).

6. Wilkinson JR, Myers LJ, Daggy J, Martin HA, Bastin G, Yang Z, et al. The VA National TeleNeurology Program (NTNP): implementing TeleNeurology to improve equitable access to outpatient neurology care. JGIM. (2023) 38(Suppl 3):887–93. doi: 10.1007/s11606-023-08121-7

PubMed Abstract | CrossRef Full Text | Google Scholar

7. Glasgow RE, Estabrooks PE. Pragmatic applications of RE-AIM for health care initiatives in community and clinical settings. Prev Chronic Dis. (2018) 15:E02. doi: 10.5888/pcd15.170271

PubMed Abstract | CrossRef Full Text | Google Scholar

8. Glasgow RE. RE-AIMing research for application: ways to improve evidence for family medicine. J Am Board Fam Med. (2006) 19(1):11–9. doi: 10.3122/jabfm.19.1.11

PubMed Abstract | CrossRef Full Text | Google Scholar

9. Wilkinson JR, Spindler M, Wood SM, Marcus SC, Weintraub D, Morley JF, et al. High patient satisfaction with telehealth in Parkinson disease: a randomized controlled study. Neurol Clin Pract. (2016) 6:241–51. doi: 10.1212/CPJ.0000000000000252

PubMed Abstract | CrossRef Full Text | Google Scholar

10. Nowak KL, Biocca F. The effect of the agency and anthropomorphism on users’ sense of telepresence, copresence, and social presence in virtual environments. Presence (2003) 12(5):481–94. doi: 10.1162/105474603322761289

CrossRef Full Text | Google Scholar

11. Whitaker RG, Sperber N, Baumgartner M, Thiem A, Cragun D, Damschroder L, et al. Coincidence analysis: a new method for causal inference in implementation science. Implement Sci. (2020) 15(1):108. doi: 10.1186/s13012-020-01070-3

PubMed Abstract | CrossRef Full Text | Google Scholar

12. Miech EJ, Freitag MB, Evans RR, Burns JA, Wiitala WL, Annis A, et al. Facility-level conditions leading to higher reach: a configurational analysis of national VA weight management programming. BMC Health Serv Res. (2021) 21(1):797. doi: 10.1186/s12913-021-06774-w

PubMed Abstract | CrossRef Full Text | Google Scholar

13. Cohen DJ, Sweeney SM, Miller WL, Hall JD, Miech EJ, Springer RJ, et al. Improving smoking and blood pressure outcomes: the interplay between operational changes and local context. Ann Fam Med. (2021) 19(3):240–8. doi: 10.1370/afm.2668

PubMed Abstract | CrossRef Full Text | Google Scholar

14. Yakovchenko V, Miech EJ, Chinman MJ, Chartier M, Gonzalez R, Kirchner JE, et al. Strategy configurations directly linked to higher hepatitis C virus treatment starts: an applied use of configurational comparative methods. Med Care. (2020) 58(5):e31–8. doi: 10.1097/MLR.0000000000001319

PubMed Abstract | CrossRef Full Text | Google Scholar

15. Hickman SE, Miech EJ, Stump TE, Fowler NR, Unroe KT. Identifying the implementation conditions associated with positive outcomes in a successful nursing facility demonstration project. Gerontologist. (2020) 60(8):1566–74. doi: 10.1093/geront/gnaa041

PubMed Abstract | CrossRef Full Text | Google Scholar

16. Mattocks KM, Kroll-Desrosiers A, Kinney R, Elway Anashua R, Cunningham KJ, Mengeling MA. Understanding VA’s use of and relationships with community care providers under the mission act. Med Care. (2021) 59(Suppl 3):S252–8. doi: 10.1097/MLR.0000000000001545

PubMed Abstract | CrossRef Full Text | Google Scholar

17. Gale RC, Wu J, Erhardt T, Boundthavong M, Reardon CM, Damschroder LJ, et al. (2019). Comparison of rapid vs. in-depth qualitative analytic methods from a process evaluation of academic detailing in the Veterans Health Administration. Implement Sci (2019) 14(1):11. doi: 10.1186/s13012-019-0853-y

PubMed Abstract | CrossRef Full Text | Google Scholar

18. Patel H, Damush TM, Miech EJ, Rattray NA, Martin HA, Savoy A, et al. Building cohesion in distributed telemedicine teams: findings from the Department of Veterans Affairs National Telestroke Program. BMC Health Serv Res. (2021) 21(1):124. doi: 10.1186/s12913-021-06123-x

PubMed Abstract | CrossRef Full Text | Google Scholar

19. Ross L, Bena J, Bermel R, McCarter L, Ahmed Z, Goforth H, et al. Implementation and patient experience of outpatient TeleNeurology. Telemed J E Health. (2021) 27(3):323–9. doi: 10.1089/tmj.2020.0032

PubMed Abstract | CrossRef Full Text | Google Scholar

20. Hatton J, Chandra R, Lucius D, Ciuchta E. Patient satisfaction of pharmacist-provided care via clinical video teleconferencing. J Pharm Pract. (2018) 31(5):429–33. doi: 10.1177/0897190017715561

PubMed Abstract | CrossRef Full Text | Google Scholar

21. Wang Y, Kung L, Gupta S. Leveraging big data analytics to improve quality of care in healthcare organizations: a configurational perspective. Br J Manag. (2019) 30(2):362–88. doi: 10.1111/1467-8551.12332

CrossRef Full Text | Google Scholar

22. Peracca SB, Jackson GL, Lamkin RP, Mohr DC, Zhao M, Lachica O, et al. Implementing teledermatology for rural veterans: an evaluation using the RE-AIM framework. Telemed J E Health. (2021) 27(2):218–26. doi: 10.1089/tmj.2020.0013

PubMed Abstract | CrossRef Full Text | Google Scholar