VALIDATE—Utilization of the Viz.ai mobile stroke care coordination platform to limit delays in LVO stroke diagnosis and endovascular treatment

Background: Thousands of hospitals worldwide have adopted mobile artificial intelligence (AI)-based stroke care coordination platforms. Studies exploring the benefit of these platforms have been scrutinized due to small sample size, serial cohort design, and measurement of metrics with multiple determinants. In this large multi-center study, we evaluated the ability of an AI-based stroke care coordination platform to expedite contact with the interventionalist (NIR) for potential thrombectomy.

Methods: Acute stroke consultations seen by TeleSpecialists, LLC physicians at 166 facilities (17 states) utilizing Viz.ai software (AI) vs. no AI software (non-AI) were extracted from the TeleCare by TeleSpecialists™ database from December 1, 2021, through March 31, 2022. The primary outcome was time from patient arrival to first contact with the interventionalist to discuss need for potential thrombectomy (Arrival-to-NIR notification).

Results: A total of 14,116 cases were analyzed. Compared to the non-AI cohort, Arrival-to-NIR notification in the AI cohort was: (1) 39.5 min faster (44.13% reduction, p < 0.001) in the overall analysis; (2) 33.0 min faster (34.0% reduction, p < 0.001) in the non-thrombectomy (non-TC) facility subgroup analysis; and (3) 34.0 min faster (43.59% reduction, p < 0.001) in the thrombectomy capable (TC) facility subgroup analysis. IQR range comparison demonstrated a significant improvement in uniformity of stroke workflow across all AI subgroups. Significant, albeit small, confounding biases were revealed in the data. The presence of AI within the non-TC subgroup correlated with a lower acceptance rate for thrombectomy by the NIR (delta = −10.79% absolute and 23.17% relative reduction, p < 0.0001).

Conclusions: While this study was limited by our inability to capture detailed neuroimaging timelines and patient outcomes, it suggests a potential significant benefit of AI-based stroke care coordination platforms and underscores the critical need to development robust “big data” systems to study the effects of AI, and other emerging technologies, on stroke systems of care.

Introduction

The rapid treatment of large vessel occlusion (LVO) stroke has been validated in short- and long- treatment window clinical trials and meta-analyses as a critical determinant of mechanical thrombectomy eligibility and good patient outcomes (Sun et al., 2013; Goyal et al., 2015, 2016; Jovin et al., 2015; Saver et al., 2015; Albers et al., 2018; Mulder et al., 2018; Nogueira et al., 2018; Carrion-Penagos et al., 2022). Previous reports indicate that factors delaying mechanical thrombectomy (MT) are complex and often driven by delay in LVO-stroke diagnosis and contact with neurointerventionalists (NIR; Sun et al., 2013; McTaggart et al., 2017; Lachkhem et al., 2018; Danziger et al., 2021; Aroor et al., 2022; Carrion-Penagos et al., 2022). In hopes of minimizing such delays, artificial intelligence (AI)-based mobile stroke platforms are now being promulgated worldwide. By combining AI-based automated LVO detection, real-time mobile high-resolution neuroimage file sharing, and multi-user communication capabilities, these platforms have been touted to accelerate stroke workflow (Murray et al., 2020; Lotan, 2021). The most utilized of these platforms are Viz.ai (VIZ) and RapidAI. Several hospitals have reported improvement in stroke workflow after implementation of such systems (Table 1; Devlin et al., 2020; Morey et al., 2021; Elijovich et al., 2022; Hassan et al., 2022a,b; Figurelle et al., 2023). Conclusions based on these past reports must be approached cautiously for three reasons. First, these studies have been primarily smaller single center studies raising concern for reproducibility. Secondly, the metrics analyzed in past studies have been complex and heavily influenced by a multitude of variables not directly under the control of the AI platform (e.g., spoke hospital door-in door-out times, time to groin puncture, length of stay, and patient outcome). Lastly, most studies have been serial cohort studies in which metrics were assessed before and after AI-based stroke platform implementation (often with years between cohorts) during which time numerous other changes in stroke workflow likely transpired. A large-scale multicenter parallel cohort study focused on stroke workflow is now warranted to assess the potential benefits of a mobile AI-based care coordination platform, better understand potential biases in the data, and assist in the design of future “big data” studies.

Table 1

Table 1. Utilization of an AI-based system to improve acute stroke workflow summary of previous studies.^*

Methods

Acute stroke consultations seen by TeleSpecialists, LLC physicians in the emergency departments of 166 facilities (17 states) that utilized the Viz.ai platform (AI) and non-AI facilities (non-AI) were extracted from the TeleCare by TeleSpecialists™ database from December 1, 2021, through March 31, 2022. Facilities using another non-VIZ AI-based platform or with protocols in which the teleneurologist did not initiate direct contact with the NIR were excluded. The same group of teleneurologists, all working remotely, evaluated patients at AI and non-AI facilities. Per the teleneurology company protocol, a 100% data entry into the web-based telemedicine database is required by all teleneurologists, including first confirmed contact time with the NIR either by phone or confirmed messaging in app, and all data must be entered before closing a case and all cases must be closed by end of shift. The two standardized workflows within both the AI and non-AI facilities are detailed in Figure 1. Figure 1A depicts the standard workflow at AI and non-AI facilities in which the stroke code involved EMS prenotification (pre-alert) and Figure 1B depicts the workflow in which no EMS prenotification occurred. The workflow at all facilities involved a series of highly standardized steps, the order of which being dependent on whether an EMS pre-alert occurred. These steps involved patient arrival and triage team evaluation in the emergency department (ED), teleneurology stroke alert activation (SAA) by the ED, immediate web-based text activation of the teleneurologist, emergent video camera login by the teleneurologist, emergent neuro imaging acquisition in radiology and viewed by the teleneurologist, and first call to the NIR to discuss possible MT in the event of suspicion for LVO on the part of the teleneurologist. In the case of EMS prenotification to the hospital, early teleneurology stroke alert activation often occurred such that the teleneurologist would arrive on camera prior to patient ED arrival. In the case of no prenotification, EMS arrival patients would be assessed by ED facility team just prior to teleneurology stroke alert activation. A similar workflow pertained to walk in patients presenting to triage. In all cases, the focus of the teleneurologist was to facilitate administration of IV thrombolysis and ensure rapid execution of advanced neuroimaging (CTA+CTP) if indicated. The primary difference in workflow between the AI and non-AI cohorts occurred with the teleneurologist and NIR viewing of the CT/CTA/CTP (see Figure 1). In the non-AI cohort, images were manually reconfigured by the CT technician and pushed to their hospital-based image reviewing platform (typically PACS) for review by the teleneurologist. In the AI cohort, neuroimaging was automatically pushed to the AI cloud-based server and made available for viewing on the teleneurologist's phone-based mobile AI platform typically within 4 min of image acquisition. By protocol, the teleneurologist reviewed all neuroimaging emergently and had the option to review imaging with a radiologist prior to emergent contact with the NIR to discuss possible MT in patients where LVO was suspected.

Figure 1

Figure 1. The two different stroke patient work flows within both AI and non-AI facilities. (A) Depicts workflow for patients in which EMS prenotification did occur and (B) depicts workflow for patients in which EMS prenotification did not occur.

Our primary outcome compared the AI vs. non-AI cohorts' median value and data dispersion (IQR) for patient arrival time to teleneurologist first contact with NIR to discuss potential MT (Arrival-to-NIR notification). “NIR notification” was defined as either: (1) first direct phone contact between the teleneurologist and the NIR; or (2) the time of confirmed messaging response by the NIR and discussion on mobile AI platform. All first contract NIR notification times were entered into the telemedicine web-based database directly by the teleneurologist. We hypothesized that the metric of Arrival-to-NIR notification was less likely to be confounded by the myriad of other non-AI-platform related variables, such as referral hospital door in-door out times, thrombectomy start time or duration, length of stay, or neurological outcome. The time of first contact with NIR is typically not collected at most centers, however, an accurate timestamp was mandated by our teleneurologists per company quality protocol. Furthermore, while the retrospective analysis of this large dataset allowed us to compare certain metrics across a vast number of hospitals, the limitations of the database precluded our ability to analyze other metrics of potential interest including CT/CTA/CTP start times, thrombectomy start time, length of stay or patient outcome. We performed two prespecified subgroup analyses to control for potential influence of other variables on the results. We subgroup analyzed metrics for thrombectomy-capable vs. non-thrombectomy-capable facilities and for EMS pre-alerted vs. no EMS pre-alert cohorts.

Descriptive data analysis and inferential tests were conducted in this study. Continuous variables were summarized using means, medians, and Interquartile ranges [IQR], while categorical variables were presented as frequencies and percentages. The normality of the data was assessed using the Shapiro test. To compare continuous variables between groups, the student's t-test was used for normally distributed data, and the Mann–Whitney test was used for non-normally distributed data. Pearson's Chi-square test was employed for comparisons involving categorical variables. Data analytics were performed on multiple cohorts of data. Statistical significance was determined at a p-value of < 0.05, and all statistical analyses were conducted using the statistical package R 4.0.2. Trial design oversight and complete statistical analysis were performed according to rigorous scientific standards and best practices by faculty from the University of Tennessee Chattanooga (LG) and the University of Tennessee Health Science Center (GH) funded independently by the NeuroScience Innovation Foundation.

Results

Data were extracted retrospectively from a total of 14,116 stroke alerts across 17 states including 8,557 alerts performed at 76 different AI facilities and 5,559 alerts performed at 90 different non-AI facilities.

Overall analysis—AI vs. non-AI cohorts

Table 2A shows a detailed comparison between AI and non-AI patient populations. The primary finding of this study was a 39.5 min (44.13%) faster time from patient Arrival-to-NIR notification at AI [AI: 50.0 min (40.00, 82.00) vs. non-AI: 89.50 min (59.25, 122.00), p < 0.001]. Importantly, a statistically significant reduction was seen in the IQR in the Arrival-to-NIR in AI vs. non-AI (p = 0.0116; Table 2, footnote line 3). When assessing for potential biases in the Arrival-to-NIR notification times for AI vs. non-AI, a significantly faster time (2.24 min) was observed at AI facilities for patient arrival to teleneurology stroke alert activation [“Bias1,” AI: 10.32 min (4.75, 20.94) vs. non-AI: 12.56 min (6.15, 26.28), p < 0.001]. A small but significantly faster time (1.00 min) was also observed at AI facilities for teleneurologist stroke alert activation to first teleneurologist video login [“Bias2,” AI: 2 min (1.00, 4.00) vs. non-AI: 3 min (2.00, 5.00), p < 0.001]. When Bias1 and Bias2 were totaled (3.24 min), they accounted for 8.20% of the overall 39.50 min reduction in Arrival-to-NIR notification. Table 2A shows that the percentage of patients arriving by EMS vs. private vehicle was higher in AI group (55.74 vs. 51.09%, p < 0.001). A higher percentage of EMS pre-alerts occurred in AI cohort (AI: 24.94 vs. non-AI: 17.93%, p < 0.001). Of the pre-alerts, no significant difference was seen in the median time from EMS pre-alert to facility arrival between the two cohorts. While the overall percentage of patients receiving IV thrombolytics in the two cohorts was not significantly different (AI: 6.37 vs. non-AI: 6.46%, p = 0.86), a 4 min faster door to needle time was observed at AI facilities [AI: 40.0 min (30.00, 52.00) vs. non-AI: 44.0 min (32.00, 57.50), p = 0.0184].

Table 2

Table 2. Overall analyses: AI vs. non-AI comparison for various cohorts.

Subgroup analysis: AI vs. non-AI by thrombectomy capability

Table 3 summarizes the results of the subgroup analyses. The predominant finding was a shorter Arrival-to-NIR notification duration in AI group regardless of whether the patient presented to a non-thrombectomy capable facility (non-TC) or a thrombectomy capable facility (TC; Table 3A). Within the non-TC subgroup, the presence of AI platform was associated with a 33 min (34.02%) median faster time in Arrival-to-NIR notification [AI: 64 min (46.00, 91.00) vs. non-AI: 97 min (62.00, 126.50), p < 0.001]. Within the TC subgroup the presence of AI platform was associated with a 34.0 min (43.59%) median faster time in Arrival-to-NIR notification [AI: 44.0 min (33.75, 59.00) vs. non-AI: 78 min (55.00, 95.00), p < 0.001]. As seen in the overall analysis, a statistically significant reduction was seen in the IQR in Arrival-to-NIR in AI vs. non-AI in both the non-TC and TC subgroups (non-TC: p = 0.0181, TC: p = 0.0148; Table 3 footnotes, line 3 and 4). A statistically faster time for Bias1 was seen for AI in both non-TC and TC subgroups; (non-TC: median delta = 1.41 min, p < 0.001; TC: median delta = 6.09 min, p < 0.001). Similarly, a statistically faster time for Bias2 was seen for AI in both non-TC and TC subgroups; (non-TC: median delta = 1 min, p < 0.001; TC: median delta = 1 min, p < 0.001). Bias1 + Bias2 accounted for 7.30% (non-TC) and 20.85% (TC) of the overall faster Arrival-to-NIR notification times in the AI cohorts (Table 3, footnotes lines 5 and 6).

Table 3

Table 3. Subgroup analyses: effect of AI in thrombectomy capable vs. non-thrombectomy capable facilities.

There was no difference in the percentage of patients treated with IV thrombolytics between AI vs. non-AI in either the TC or non-TC subgroups. While there was no difference between AI vs. non-AI median door to needle time (DTN) in the non-TC subgroup (Table 3A), a significantly faster DTN (delta = −6.5 min) was observed in AI group within the TC subgroup [37.50 min (27.00, 48.75) vs. 44.00 min (33.00, 54.00), p = 0.0125].

Effect of EMS pre-alert stroke notification

Due to 7.01% more EMS pre-alert patients in the AI group (which could theoretically lead to shorter Arrival-to-NIR notification times by having the teleneurologist on camera faster thereby expediting CTA acquisition), we next performed additional sub analyses first with all pre-alert patients excluded, then with only pre-alert patients included.

With pre-alerts excluded (Table 2B), there was a 30.53 min (31.83%) reduction in Arrival-to-NIR notification at AI facilities [AI: 65.40 min (45.72, 92.15) vs. non-AI: 95.93 min (66.97, 124.77), p < 0.0001]. The pre-alert excluded analysis also demonstrated only a small, albeit significant, difference for AI vs. non-AI for Bias1 and Bias2 (Table 2B). The pre-alert excluded subgroup analysis (Table 3B) showed a significant, 34.52 min (32.98%) faster Arrival-to-NIR notification in the non-TC AI subgroup [AI: 70.15 min (47.31, 94.13) vs. non-AI: 104.67 min (74.24, 133.52), p < 0.0001] and a 27.08 min (33.50%) faster Arrival-to-NIR notification in the TC AI subgroup [AI: 53.77 min (43.25, 86.95) vs. non-AI: 80.85 min (60.62, 101.17), p = 0.0144]. Importantly, in the pre-alert excluded analysis exact times from teleneurologist first video login attempt to first NIR notification could be calculated (“Login-to-NIR notification”), effectively removing Bias1 and Bias2 entirely (calculation of this metric for the all-patients-included analysis was not possible due to imbalance in the percent of pre-alerts in AI vs. non-AI with teleneurologists typically being on camera prior to patient arrival in the pre-alerted patients thus skewing the data). This analysis demonstrated a 28.5 min (38.38%) faster time in the teleneurologist Login-to-NIR notification for AI [AI: 45.75 min (32.64, 69.45) vs. non-AI: 74.25 min (51.60, 101.22), p < 0.0001]. Similarly, the subgroup analysis (Table 3B) demonstrated the non-TC Login-to-NIR notification time to be 35.77 min (43.56%) faster at AI [AI: 46.34 min (32.76, 71.83) vs. non-AI: 82.11 min (51.76, 110.44), p < 0.0001] and for TC, the Login-to-NIR notification time was 23.59 min (37.18%) faster at AI [AI: 39.86 min (32.26, 60.49) vs. non-AI: 63.45 min (42.44, 76.47), p = 0.0187].

Table 2C shows the results of the overall analysis with EMS pre-alert patients included only. This analysis revealed a large 60.14 min (57.42%) faster time in Arrival-to-NIR notification (p = 0.0002) and a 58.3 min (57.44%) faster time in Login-to-NIR notification (p = 0.0011) in favor of the AI cohort. In the non-TC subgroup analysis (Table 3C) highly significant faster times in the AI cohort were maintained for both time metrics, however, in the TC subgroup a significant difference was not seen in the setting of low subject number.

Rate of LVO detection and NIR acceptance for MR

A statistically higher rate of advanced imaging was performed at AI vs. non-AI in the overall analysis (42.33 vs. 37.00%, p < 0.001, Table 2A). This difference was largely driven by the higher rate of advanced imaging performed in the TC subgroup (Table 3A). In the overall analysis (Table 2A), a slightly higher rate of LVO detection occurred in AI (AI: 5.36% vs. non-AI: 4.14%; p < 0.001). When limited to those that underwent advanced imaging (CTA or CTA/CTP), no difference in overall LVO detection was found between AI vs. non-AI. The subgroup analysis, however, did demonstrate a significantly higher rate of LVO detection among patients undergoing advanced imaging in the AI cohort within the TC subgroup (7.10 vs. 0.71%, p = 0.0001) but not the non-TC subgroup (NS).

Lastly, we assessed the effect of AI platform on the rate of NIR acceptance for MT. In the overall analysis (Table 2A), the presence of AI platform had an inverse effect on the rate of NIR acceptance for thrombectomy on patients with advanced imaging performed (AI: 5.4% vs. non-AI: 5.6%; p < 0.001) and a more robust reduction in NIR acceptance for thrombectomy in the suspected LVO population for which teleneurologists initiated contact with NIR (AI: 42.27% vs. non-AI: 50.00%; p < 0.0001). This critical finding was driven by a 10.79% absolute reduction (23.17% reduction) in the acceptance rate by the NIR for suspected LVO patients at non-TC (AI: 35.77% vs. non-AI: 46.56%, p < 0.0001, Table 3A).

Discussion

We compared stroke workflow between a cohort of hospitals that utilized an AI-based stroke care coordination system vs. a cohort of hospitals that did not. We analyzed 14,116 stroke alerts performed over 4 months across 166 facilities in 17 states. We chose our primary outcome to be the metric of patient arrival to first contact with NIR to discuss need for thrombectomy (Arrival-to-NIR notification) as it was this measurable variable, we hypothesized, that would be most directly under the control of the AI-based care coordination platform. Ideally, measurement of CTA start time to NIR contact would have been preferable however neuro imaging start times were not available within our telemedicine database. In addition to reduced Arrival-to-NIR times, we also hypothesized that the presence of AI may lead to improved consistency of stroke care as represented by smaller values for IQR across multiple metrics. We analyzed potential imbalances in AI vs. non-AI cohorts which could skew our primary outcome measure including cohort differences in patient arrival time to teleneurology stroke alert activation (Bias1) and teleneurology stroke alert activation to first teleneurologist video login (Bias2). Lastly, we addressed the potential confounding factor of the significant 7.01% higher percentage of EMS pre-alert activations in the AI cohort (by analyzing EMS pre-alert and non-alert patients separately. This was performed to mitigate the concern that a higher percentage of EMS pre-alert patients in AI could lead to faster times to teleneurologist login, faster time to CTA acquisition, and thereby faster Arrival-to-NIR notification in AI. Significant differences did exist between the AI and non-AI cohorts: (1) 60.62% of patients overall were treated utilizing AI; (2) 100% of comprehensive centers utilized AI; (3) while an equal percentage of patients were treated with IV thrombolysis at AI and non-AI, DTN times were slightly faster at AI (4 min); and (4) a higher percentage of EMS pre-alerts occurred at AI than non-AI (58 vs. 41%).

The primary finding of this study was a 39.5 min faster median time at AI vs. non-AI for Arrival-to-NIR notification to discuss potential thrombectomy (44.13%, p < 0.001). In evaluating for potential biases contributing to the faster Arrival-to-NIR notification results at AI facilities, we did identify a faster arrival to stroke alert activation time (Bias1) and teleneurology stroke alert activation to teleneurologist first login attempt time (Bias2). When combined, these biases together represented 8.20% of the overall 39.50 min (44.13%) faster time in Arrival-to-NIR notification in the AI cohort. The exact magnitude of the combined biases for individual centers was not calculated but, in some cases, may have been larger given the broad IQR for Bias1 and Bias2. The relatively small median size of these biases is consistent with our conclusion that presence of AI was associated with a significantly faster time from patient Arrival-to-NIR notification at AI facilities. Importantly, the potential for a combined Bias1 and Bias2 effect significantly skewing our results was mitigated by performing a subgroup analysis in which we excluded the 22.18% of total patients that underwent EMS pre-alert (non-AI: 17.93% and AI: 24.93%). Analysis of the data with pre-alert patients removed allowed us to minimize the potential bias introduced by a higher percentage of pre-alert patients at AI facilities and permitted calculation of teleneurologist first Login-to-NIR notification thus effectively removing the Bias1 and Bias2 effects. With pre-alert patients removed, we again found a highly significant [30.53 min (31.83%), p < 0.0001] faster overall Arrival-to-NIR notification time in favor of AI and, importantly, a statistically significant 28.50 min (38.38%, p < 0.0001) faster teleneurologist Login-to-NIR notification favoring AI (Table 2B). The second major finding of this study was the fact that significantly faster Arrival-to-NIR notification time at AI facilities occurred regardless of whether the patient presented to a non-TC or TC facility [non-TC delta: −33.0 min (−34.02%) and TC delta: −34 min (43.59%), p < 0.001 for both]. The magnitude of the combined Bias1 and Bias2 effect in the subgroup analyses was again found to be relatively small (albeit subject to the same caveat regarding potential larger bias effect at some centers due to the wide Bias1 and Bias2 IQR values). A similar analysis wherein pre-alert patients were removed to mitigate effect of bias again revealed significantly faster Arrival-to-NIR notification for both the non-TC and TC subgroups [non-TC: 34.52 min (32.98%), p < 0.0001; TC: 27.08 min (33.50%), p = 0.0144] and a significantly reduced time in teleneurologist Login-to-NIR notification for AI at both the non-TC [35.77 min (43.56%), p < 0.0001] and TC subgroups [23.59 min (37.18%), p = 0.0187%, Table 3B].

The association between AI and consistency of stroke workflow appeared to be a particularly important finding. We found a highly significant reduction in the IQR spread at AI facilities in the global and subgroup analyses for both metrics of patient Arrival-to-NIR notification and teleneurologist Login-to-NIR notification (see Table 2 footnotes line 3 and Table 3 lines 3, 4, 7, 8). This significant reduction in the IQR spread in the AI vs. non-AI group further suggests that AI may improve uniformity of stroke care delivery. As expected, the presence of AI was not associated with an increase in the rate of thrombolysis over non-AI facilities and only a 4 min faster DTN needle time. As the decision to initiate thrombolysis is typically based on plain CT and is typically administered intravenously before CTA/CTP is performed, we would not expect the presence of AI to affect this metric. The overall 4 min faster DTN time (9.1%, p = 0.0184) in AI may be related to a 20.3% higher volume of comprehensive centers in the AI cohort which may operate with increased efficiency.

This study investigated potential effects of AI on critical stroke workflow metrics rather than clinical outcomes due to the multifaceted determinants of outcome and the challenges of collecting outcome data on such a large dataset. Nonetheless, the limitations of this study highlight the critical importance of developing large scale database systems connecting multi-healthcare networks in which a wide range of stroke workflow metrics, full neuroimaging time points, full patient transfer details, thrombectomy procedure metrics, and both short term and long-term outcome measures are analyzed. The development of such big data systems is essential if we are to fully vet the effects of any new technology on patient outcomes in real world settings. A significant limitation of this study is our inability to capture CTA times. This raises the potential bias in which a shorter time from patient arrival to time of CTA completion (potentially driven by the higher EMS pre-alert population in the AI cohort) may have been a driver to faster Arrival-to-NIR notification. It is however important to consider that each stroke alert at AI and non-AI facilities was executed under the watchful eye of the same groups of teleneurologist specifically charged to orchestrate stroke codes to prevent unnecessary delays in the performance of CTA. We therefore think it is unlikely that a significant delay in the performance of the CTA (thereby delaying appearance of neuroimaging on the teleneurologists' mobile app) would account for the large difference in the Arrival-to-NIR times between AI and non-AI groups. Our results did show that while the median duration of EMS prenotification prior to arrival for AI vs. non-AI was not significantly different, a significant 4.65% (p < 0.001) higher percentage of AI patients arrived by EMS and of all EMS patients a 7.01% (p < 0.001) higher percentage of AI cases experienced prehospital alert. The presence of EMS pre-hospital alert has been reported by multiple investigators to be associated with faster times for door-to-CT, door-to-needle, and door-to-mechanical thrombectomy along with improved patient outcome (Patel et al., 2011; Lin et al., 2012; Hsieh et al., 2016; Sheng et al., 2018; Fujiwara et al., 2022; Oostema et al., 2023). Therefore, we cannot exclude these imbalances in EMS metrics having some effect on our overall faster Arrival-to-NIR time for AI vs. non-AI. However, a significantly faster Arrival-to-NIR for AI vs. non-AI in the non-TC subgroup analysis (33.0 min, 34.02%, p < 0.001) was demonstrated despite the fact that this subgroup exhibited no difference in the percentage of patients arriving by EMS or the percent of EMS patients undergoing pre-alert. This finding would tend to argue against a large confounding effect in our results due to EMS metric imbalance. Interestingly, we also found that the largest improvement in Arrival-to NIR times correlated with those patients that exhibited both EMS pre-alert and the presence of AI (Arrival-to-NIR overall analysis improvement 57.42% (p = 0.0002) AI vs. non-AI; and Arrival-to-NIR improvement in non-TC subgroup 52.51% (p = 0.0127) AI vs. non-AI). The exact cause of this apparent interaction remains unclear. Future development of big data systems will be critical to allowing us to calculate CTA time-to-NIR-notification time, and similar metrics, that will be more recalcitrant to bias effects.

Given the extreme challenges that hospitals are facing in stroke care today, the effect that AI demonstrated in our study on the rate of patient acceptance by the NIR appears to be particularly relevant. In the overall analysis, the presence of an AI platform had an inverse effect on the rate of NIR acceptance for MT on the group of patients in which the teleneurologist contacted NIR for a suspected LVO. This negative effect of AI on NIR acceptance of LVO-identified patients appeared to be driven by patients presenting to non-TC centers [10.79% absolute reduction (−23.17%), p < 0.001]. While the exact reason for this phenomenon is uncertain, we hypothesize that the AI platform may impact NIR decision making by providing a unique high-quality mobile CT, CTA, and CTP review station to review outside facility neuroimaging. We hypothesize that the AI mobile platform, utilized by NIRs in this study, which allowed them to scrutinize a CTA with full MIP and 3-D rotational ability around any point in the brain may have played a significant role in improved selection of appropriate endovascular candidates. With neuroimaging confidently reviewed, the NIRs may have declined to accept for MT what would otherwise have been a “futile transfer” upon patient arrival at the TC center. The reduction in futile transfers has never been more important than now for both hub and spoke hospitals (Mullen et al.). With hub hospitals across the United States experiencing critical staffing shortages, the ability to limit transfers to those patients most in need of higher level of care is of paramount importance. Similarly, the need to shut down futile transfers, thereby driving spoke hospital revenue, is now critically important for countless referral hospitals teetering on financial collapse (American Hospital Association, 2022; Davenport, 2023; Thompson, 2023; Mullen et al.). The positive results of the “large core” interventional trials enrolling patients with ASPECTS scores 3–5, will likely lead to a change in guidelines such that more patients with large infarcts are transferred and undergo thrombectomy (Yoshimura et al., 2022; Huo et al., 2023; Sarraj et al., 2023). Despite this expected paradigm shift, the goal of getting every patient admitted expediently to a center able to maximize their outcome will not change. Therefore, AI will likely remain a critical tool in our armamentarium for LVO detection, provider communication, and treatment facilitation.

Our results beg the question: are these findings specific to the practice of telemedicine or are they more widely generalizable? Stroke workflow at telemedicine-based facilities is directly mirrored after highly efficient in-person neurologist-based facilities with only minor variations. As such, when assessing the potential effect of AI on stroke workflow, our findings are less likely to represent a “telemedicine-specific” effect. The presence of AI platform did not increase the LVO detection rate across the total population of patients that underwent advanced imaging (Table 2A). A small, but significant, increase was seen in the rate of LVO detection among those patients undergoing advanced imaging in the TC AI vs. non-AI subgroups (7.10 vs. 0.715%, p = 0.0001) but not the non-TC subgroup (Table 3A). The lack of difference in the overall LVO detection rate in our study between AI and non-AI may have been attributable to both a treating neurologist and a radiologist reviewing all neuroimaging emergently. At other hospital systems without this level of intense emergent neuroimaging scrutiny a greater LVO detection rate with AI may be more likely. Previous investigators have reported higher levels of LVO detection rates commensurate with the level of neuro-specific training of their reviewing physicians (Karamchandani et al., 2022).

Conclusion

In conclusion, despite our modern-day deeper understanding of “time is brain” (now incorporating the concept of “slow progressors and fast progressors”), we still cannot predict the specific infarct rate for any individual LVO patient. As such, all potential MT patients should be treated as fast progressors with laser focus by the ED stroke team on rapid LVO detection and getting appropriate neuroimaging into the hands of the treating NIR physicians. Numerous studies have been published giving us great insight into the highly time dependent nature of MT eligibility and outcomes (Sun et al., 2013; Goyal et al., 2015, 2016; Jovin et al., 2015; Saver et al., 2015; Albers et al., 2018; Mulder et al., 2018; Nogueira et al., 2018; Carrion-Penagos et al., 2022). While it was not the purpose of this study to prove “time is brain,” the degree of potential time savings reported in this study would be considered by most clinicians to be in the highly relevant range. In fact, delays in contacting the neurointerventionalist have been estimated to comprise ~57% of the entire patient door in/door out time at spoke hospitals (Sun et al., 2013). The results of this large multicenter study, when combined with the results of previous studies, support the adoption of advanced AI-based care coordination platforms with automated LVO detection, neuroimaging review, and communication systems into the armamentarium of acute stroke care hospitals. Most importantly, this study underscores the critical need to develop “big data” systems linking multi-hospital networks to capture detailed data on stroke workflow, neuro imaging times, specifics of treatments, and patient outcomes. Only in this way will we be able to fully vet the potential benefit of AI (and other emerging technologies), expedite adoption, and maximize the outcomes of our patients.

Data availability statement

The original contributions presented in the study are included in the article/supplementary material, further inquiries can be directed to the corresponding author.

Ethics statement

The studies involving humans were approved by TeleSpecialists IRB, Fort Myers, FL, USA. The studies were conducted in accordance with the local legislation and institutional requirements. The Ethics Committee/Institutional Review Board waived the requirement of written informed consent for participation from the participants or the participants' legal guardians/next of kin because, this was purely a retrospective analysis of data acquired from the TeleSpecialist's database.

Author contributions

TD: Conceptualization, Funding acquisition, Methodology, Writing – original draft, Writing – review & editing. LG: Conceptualization, Data curation, Formal analysis, Methodology, Project administration, Supervision, Visualization, Writing – original draft, Writing – review & editing. OC: Data curation, Formal analysis, Visualization, Writing – review & editing. GH: Data curation, Formal analysis, Supervision, Writing – review & editing, Conceptualization, Investigation, Methodology, Validation, Visualization. MF: Conceptualization, Data curation, Investigation, Supervision, Writing – review & editing, Resources. AA: Data curation, Writing – review & editing, Supervision, Conceptualization, Formal analysis, Methodology, Project administration, Validation. CB: Data curation, Supervision, Writing – review & editing, Resources. HA: Data curation, Supervision, Writing – review & editing, Formal analysis. TS: Conceptualization, Data curation, Formal analysis, Funding acquisition, Investigation, Methodology, Resources, Supervision, Validation, Visualization, Writing – original draft, Writing – review & editing.

Funding

The author(s) declare that financial support was received for the research, authorship, and/or publication of this article. Trial design oversight and complete statistical analysis were performed by faculty from the University of Tennessee Chattanooga (LG) and the University of Tennessee Health Science Center (GH) funded independently by the NeuroScience Innovation Foundation.

Acknowledgments

We would like to thank Peter Catalano at the NeuroScience Innovation Foundation for assistance in early conceptualization of this project.

Conflict of interest

TD receives consulting support from and has equity interest in VIZ.ai. TD, MF, AA, CB, HA, and TS are employees of the commercial telestroke provider from whom the data are provided in this study.

The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Abbreviations

AI, Artificial intelligence-enabled hospital cohort; Arrival-to-NIR notification, Patient arrival time to first call by the teleneurologist to the NIR to discuss potential MT in patient with suspected LVO; CT, Computerized tomography; CTA, Computerized tomography angiography; CTP, Computerized tomography perfusion; ED, Emergency department; Login-to-NIR contact, Teleneurologist first camera login time to time to first contact with NIR to discuss need for possible thrombectomy; MIP, Maximum intensity projection images on CT; MT, Mechanical thrombectomy; NIR, Neurointerventionalist; Non-AI, Non-artificial intelligence-enabled enabled hospital cohort; Non-TC, Non-thrombectomy capable hospital subgroup; NS, Not statistically significant; PACS, Picture archiving and communication system; TC, Thrombectomy capable hospital subgroup; VIZ, Viz. ai automated LVO detection platform.

References

Albers, G., Marks, M., Kemp, S., Christensen, S., Tsai, J. P., Ortega-Gutierrez, S., et al. (2018). Thrombectomy for stroke at 6 to 16 hours with selection by perfusion imaging. N. Engl. J. Med. 378, 708–718. doi: 10.1056/NEJMoa1713973

PubMed Abstract | Crossref Full Text | Google Scholar

American Hospital Association (2022). AHA Report: Rural Hospital Closures Threaten Access. Solutions to Preserve Care in Local Communities. American Hospital Association. Available online at: https://www.aha.org/system/files/media/file/2022/09/rural-hospital-closures-threaten-access-report.pdf (accessed June 1, 2023).

Google Scholar

Aroor, S. R., Asif, K. S., Potter-Vig, J., Sharma, A., Menon, B. K., Inoa, V., et al. (2022). Mechanical thrombectomy access for all? Challenges in increasing endovascular treatment for acute ischemic stroke in the United States. J. Stroke 24, 41–48. doi: 10.5853/jos.2021.03909

PubMed Abstract | Crossref Full Text | Google Scholar

Carrion-Penagos, J., Thind, S., Coleman, E., Brorson, J., McKoy, C., Mendelson, S., et al. (2022). Endovascular therapy delay for acute large vessel occlusion (LVO) is associated with worse functional outcome and increased mortality. AAN Abstract 2022:13.004. doi: 10.1212/WNL.98.18_supplement.309

PubMed Abstract | Crossref Full Text | Google Scholar

Danziger, R., Tan, C., Churilov, L., Mitchell, P., Dowling, R., Bush, S., et al. (2021). Intrinsic hospital factors: overlooked cause for variations in delay to transfer for endovascular thrombectomy. J. NeuroIntervent. Surg. 13, 968–973. doi: 10.1136/neurintsurg-2020-016836

PubMed Abstract | Crossref Full Text | Google Scholar

Davenport, K. (2023). New Medicare Designation Could Prevent Closure of Struggling Rural Hospitals. National Conference of State Legislatures. Available online at: https://www.ncsl.org/state-legislatures-news/details/new-medicare-designation-could-prevent-closure-of-struggling-rural-hospitals (accessed June 1, 2023).

Google Scholar

Devlin, T., Shah, R., Patterson, J., Owen, J., Paradis, O., and Heath, G. (2020). Distinction: Utilization of Applied Artificial Intelligence to Facilitate LVO Detection and Synchronizing Workflow to Improve Time to Treatment in High-Volume Hub & Stroke Networks. World Stroke Organization.

Google Scholar

Elijovich, L., Iii, D. D., Nickele, C., Alexandrov, A., Inoa-Acosta, V., Arthur, A. S., et al. (2022). Automated emergent large vessel occlusion detection by artificial intelligence improves stroke workflow in a hub and spoke stroke system of care. J. Neurointerv. Surg. 14, 704–708. doi: 10.1136/neurintsurg-2021-017714

PubMed Abstract | Crossref Full Text | Google Scholar

Figurelle, M., Meyer, D., Perrinez, E., Santiago-Dieppa, D. R., Khalessi, A. A., Bolar, D. S., et al. (2023). Viz.ai implementation of stroke augmented intelligence and communications platform to improve indicators and outcomes for a comprehensive stroke center and network. Am. J. Neuroradiol. 44, 47–53. doi: 10.3174/ajnr.A7716

PubMed Abstract | Crossref Full Text | Google Scholar

Fujiwara, S., Kuroda, T., Matsuoka, Y., Ohara, N., Imamura, H., Yamamoto, Y., et al. (2022). Prehospital stroke notification and endovascular therapy for large vessel occlusion: a retrospective cohort study. Sci. Rep. 12:10107. doi: 10.1038/s41598-022-14399-0

PubMed Abstract | Crossref Full Text | Google Scholar

Goyal, M., Demchuk, A., Bijoy, K., Eesa, M., Rempel, J. L., Thornton, J., et al. (2015). Randomized assessment of rapid endovascular treatment of ischemic stroke. N. Engl. J. Med. 372, 1019–1030. doi: 10.1056/NEJMoa1414905

PubMed Abstract | Crossref Full Text | Google Scholar

Goyal, M., Menon, B. K., van Zwam, W. H., Dippel, D. W. J., Mitchell, P. J., Demchuk, A. M., et al. (2016). Endovascular thrombectomy after large-vessel ischaemic stroke: a meta-analysis of individual patient data from five randomized trials. Lancet 387, 1723–1731. doi: 10.1016/S0140-6736(16)00163-X

PubMed Abstract | Crossref Full Text | Google Scholar

Hassan, A., Ringheanu, V., Preston, L., and Tekle, W. G. (2022b). Artificial intelligence-parallel stroke workflow tool improves reperfusion rates and door-in to puncture interval. Stroke Vasc. Interv. Neurol. 2:e000224. doi: 10.1161/SVIN.121.000224

Crossref Full Text | Google Scholar

Hassan, A., Ringheanu, V., and Tekle, W. (2022a). The implementation of artificial intelligence significantly reduces door-in-door-out times in a primary care center prior to transfer. Interv. Neuroradiol. 2022:15910199221122848. doi: 10.1177/15910199221122848

PubMed Abstract | Crossref Full Text | Google Scholar

Hsieh, M.-J., Tang, S. C., Chiang, W. C., Tsai, L. K., Jeng, J. S., Ma, M. H. M., et al. (2016). Effect of prehospital notification on acute stroke care: a multicenter study. SJTREM 24:57. doi: 10.1186/s13049-016-0251-2

PubMed Abstract | Crossref Full Text | Google Scholar

Huo, X., Ma, G., Tong, X., Zhang, X., Pan, Y., Nguyen, T. N., et al. (2023). Trial of endovascular therapy for acute ischemic stroke with large infarct. N. Engl. J. Med. 388, 1272–1283. doi: 10.1056/NEJMoa2213379

PubMed Abstract | Crossref Full Text | Google Scholar

Jovin, T., Chamorro, A., Cobo, E., de Miquel, M. A., Molina, C. A., Rovira, A., et al. (2015). Thrombectomy within 8 hours after symptom onset in ischemic stroke. N. Engl. J. Med. 372, 2296–2306. doi: 10.1056/NEJMoa1503780

PubMed Abstract | Crossref Full Text | Google Scholar

Karamchandani, R., Helms, A., Satyanarayana, S., Yang, H., Clemente, J. D., Defilipp, G., et al. (2022). Automated detection of intracranial large vessel occlusions using Viz.ai software: experience in a large, integrated stroke network. Brain Behav. 13:e2808. doi: 10.1002/brb3.2808

PubMed Abstract | Crossref Full Text | Google Scholar

Lachkhem, L., Rican, S., and Minvielle, E. (2018). Understanding delays in acute stroke care: a systematic review of reviews. Eur. J. Publ. Health 28, 426–433. doi: 10.1093/eurpub/cky066

PubMed Abstract | Crossref Full Text | Google Scholar

Lin, C., Peterson, E., Smith, E., Saver, J. L., Liang, L., Xian, Y., et al. (2012). Emergency medical service hospital prenotification is associated with improved evaluation and treatment of acute ischemic stroke. Circ. Cardiovasc. Qual. Outcomes 5, 514–522. doi: 10.1161/CIRCOUTCOMES.112.965210

PubMed Abstract | Crossref Full Text | Google Scholar

Lotan, E. (2021). Emerging Artificial Intelligence Imaging Applications for Stroke Interventions. Am. J. Neuroradiol. 42, 255–256. doi: 10.3174/ajnr.A6902

PubMed Abstract | Crossref Full Text | Google Scholar

McTaggart, R. A., Yaghi, S., Cutting, S. M., Hemendinger, M., Baird, G. L., Haas, R. A., et al. (2017). Association of a primary stroke center protocol for suspected stroke by large-vessel occlusion with efficiency of care and patient outcomes. J. Am. Med. Assoc. Neurol. 74, 793–800. doi: 10.1001/jamaneurol.2017.0477

PubMed Abstract | Crossref Full Text | Google Scholar

Morey, J. R., Zhang, X., Yaeger, K. A., Fiano, E., Marayati, N. F., Kellner, C. P., et al. (2021). Real-world experience with artificial intelligence-based triage in transferred large vessel occlusion stroke patients. Cerebrovasc. Dis. 50, 450–455. doi: 10.1159/000515320

PubMed Abstract | Crossref Full Text | Google Scholar

Mulder, M., Jansen, I., and Goldhoornb, R. (2018). Time to endovascular treatment and outcome in acute ischemic stroke MR CLEAN registry results. Circulation 138, 232–240. doi: 10.1161/CIRCULATIONAHA.117.032600

PubMed Abstract | Crossref Full Text | Google Scholar

Mullen D. Avila A. Mowzoon N. Optimization of acute stroke-related hospital finances utilizing artificial intelligence. J. Healthc. Manag. .

Google Scholar

Murray, N., Unberath, M., Hager, G., and Hui, F. K. (2020). Artificial intelligence to diagnose ischemic stroke and identify large vessel occlusions: a systematic review. J. NeuroIntervent. Surg. 12, 156–164. doi: 10.1136/neurintsurg-2019-015135

PubMed Abstract | Crossref Full Text | Google Scholar

Nogueira, R., Jadhav, A., Haussen, D., Bonafe, A., Budzik, R. F., Bhuva, P., et al. (2018). Thrombectomy 6 to 24 hours after stroke with a mismatch between deficit and infarct. N. Engl. J. Med. 378, 11–21. doi: 10.1056/NEJMoa1706442

PubMed Abstract | Crossref Full Text | Google Scholar

Oostema, J. A., Nickles, A., Allen, J., Ibrahim, G., Luo, Z., and Reeves, M. J. (2023). Emergency medical services compliance with prehospital stroke quality metrics is associated with faster stroke evaluation and treatment. Stroke 55, 101–109. doi: 10.1161/STROKEAHA.123.043846

PubMed Abstract | Crossref Full Text | Google Scholar

Patel, M., Rose, K., O'Brien, E., and Rosamond, W. D. (2011). Prehospital notification by emergency medical services reduces delays in stroke evaluation. Stroke 42, 2263–2268. doi: 10.1161/STROKEAHA.110.605857

PubMed Abstract | Crossref Full Text | Google Scholar

Sarraj, A., Hassan, A., Abraham, M., Ortega-Gutierrez, S., Kasner, S. E., Hussain, M. S., et al. (2023). Trial of endovascular thrombectomy for large ischmic stroke. N. Engl. J. Med. 388, 1259–1271. doi: 10.1056/NEJMoa2214403

PubMed Abstract | Crossref Full Text | Google Scholar

Saver, J. L., Goyal, M., Bonafe, A., Diener, H. C., Levy, E. I., Pereira, V. M., et al. (2015). Stent-retriever thrombectomy after intravenous t-PA vs. t-PA alone in stroke. N. Engl. J. Med. 372, 2285–2295. doi: 10.1056/NEJMoa1415061

PubMed Abstract | Crossref Full Text | Google Scholar

Sheng, Z., Zhang, J., Shang, M., Zhong, G., Chen, Z., Lin, L., et al. (2018). Prehospital notification procedure improves stroke outcome by shortening onset to needle time in Chinese urban area. Aging Dis. 3, 426–434. doi: 10.14336/AD.2017.0601

PubMed Abstract | Crossref Full Text | Google Scholar

Sun, C. H. J., Nogueira, R. G., Glenn, B. A., Connelly, K., Zimmermann, S., Anda, K., et al. (2013). “Picture to puncture” a novel time metric to enhance outcomes in patients transferred for endovascular reperfusion in acute ischemic stroke. Stroke 127, 1139–1148. doi: 10.1161/CIRCULATIONAHA.112.000506

PubMed Abstract | Crossref Full Text | Google Scholar

Thompson, D. (2023). Hundreds of Hospitals Could Close Across Rural America. US News and World Report. Available online at: https://www.usnews.com/news/health-news/articles/2023-01-16/hundreds-of-hospitals-could-close-across-rural-america (accessed June 1, 2023).

Google Scholar

Yoshimura, S., Sakai, N., Yamagami, H., Uchida, K., Beppu, M., Toyoda, K., et al. (2022). Endovascular therapy for acute stroke with a large ischemic region. N. Engl. J. Med. 386, 1303–1313. doi: 10.1056/NEJMoa2118191

PubMed Abstract | Crossref Full Text | Google Scholar