Comparison of virtual reality platforms to enhance medical education for procedures

Background: Historically, medical education relied on apprentice-based experiences requiring direct observation in patient cases. Simulation-based education has been shown to improve resident confidence but can be time intensive and difficult to coordinate. The COVID-19 pandemic demonstrated the need to develop distributed educational tools. Virtual reality (VR) platform has been shown to improve resident confidence and proficiencies. This pilot study compared educational and cost effectiveness of low-cost cardboard viewer VR (CVVR) and commercially available integrated headset VR (IHVR).

Methods and Materials: We created a 2D, 360-degree VR video of an intracavitary brachytherapy case for treatment of cervical cancer. Radiation oncology residents from a single ACGME-accredited training program were recruited and randomized to IHVR or CVVR. Both groups were given unlimited access to their randomized technology. Each resident performed a timed intracavitary procedure on a simulator while five implant quality metrics were recorded. A pre- and post-simulation questionnaire assessed self-confidence, procedural knowledge, and perceived usefulness of VR technology.

Results: There were 13 residents, including four post-graduate year (PGY)-2, three PGY-3, two PGY-4, and four PGY-5, in the study. Both VR technologies improved self-perceived overall confidence. Average time required for implant (mean: CVVR - 200 s vs IHVR - 235 s, p = 0.38) and median objective proficiencies of implant quality (5/5 in both group, p = 0.56) were similar. There was no difference between CVVR and IHVR as useful, enjoyable and engaging educational tool. Both groups would recommend the technology to another trainee. IHVR-based program would cost ∼33x more than CVVR-based program based on an assessment of US-based programs.

Conclusion: CVVR is a cost-effective alternative to a IHVR as a virtual video-based education tool.

Introduction

Classical medical education depends on patient experiences supplemented with 2D images, slideshow presentation or non-immersive video-based education. However, exposure to different patient and procedural experiences can be heterogeneous among trainees. Although this training can be effective, it may lack resident engagement and immersion (Grassini et al., 2020; Monaghan 2020; Portelli et al., 2020). Simulation-based education has been shown to improve satisfaction, confidence and skill in multiple medical subspecialities (Lynch et al., 2005; Lorello et al., 2014; Gupta et al., 2017; Zhao et al., 2018; Donnelly et al., 2020; Mesko et al., 2020). One challenge to simulation-based education can be time-intensive, requires considerable equipment and coordination, which limits a resident’s ability to repeat the simulation for sustained learning. Similar to other procedural fields, resident education was further impacted by the COVID-19 pandemic due to decreased case volume, reduced resident staffing, and limitation to conduct simulation-based education (SBE) due to social distancing (Bambakidis and Tomei, 2020; Monaghan 2020). These limitations reveal a clear need to develop virtual educational tools to complement any patient-based experiences.

In radiation oncology, brachytherapy is a procedural intervention critical in treatment of cervical cancer (Dimopoulos et al., 2009a; Dimopoulos et al., 2009b; National Cancer Institute, 2016). Adequate procedural training is necessary to improve tumor control and reduce complication rates (Irvin et al., 2003; Viswanathan et al., 2012). Traditional brachytherapy procedural training involves direct instruction during real patient encounters. However, with decreasing utilization of brachytherapy in cervical cancer over time, there are fewer traditional teaching opportunities for resident physicians (Han et al., 2013). Even prior to the COVID-19 pandemic, only about half of the residents graduating from Accreditation Council for Medical Education (ACGME)-accredited programs expressed confidence in developing a brachytherapy practice. The majority of the residents expressed that low case volume was the greatest barrier to brachytherapy skill (Marcrom et al., 2019). This also reveals a need to develop virtual educational systems to supplement resident training.

Virtual reality (VR) offers a unique solution to educational challenges by offering a socially distanced, on-demand, immersive experience that has been shown to develop resident confidence and technical proficiency (Ahlberg et al., 2007; Pulijala et al., 2018; Khan et al., 2019; Baniasadi et al., 2020). In the past, VR SBE has required expensive equipment including commercially available headset such as Oculus^© Go (Meta Platforms Inc, Menlo Park, CA) stand-alone headset (Taunk et al., 2021). However, these high costs may present logistical challenges and limit widespread access, which ultimately could hinder the implementation of VR SBE into training programs. Thus, more cost-effective technologies should be investigated. This pilot study compared educational and cost effectiveness of low-cost cardboard viewer VR (CVVR) and commercially available integrated headset VR (IHVR). In particular, this study focuses on the 2D-360° immersive video aspect of VR systems available to both IHVR and CVVR systems. This study excludes 3D-video and 180-degree videos, and additional features afforded by IHVR including hand tracking.

Methods

Participants

All 18 Radiation Oncology residents from a single ACGME-accredited training program were invited to participate in the study. 13 radiation oncology resident physicians with or without prior brachytherapy experience volunteered and were ultimately recruited to participate in the VR SBE. Residents were randomized 1:1 to two different virtual reality delivery platforms: commercially available Oculus Go stand-alone VR goggle or Google cardboard VR viewer (Alphabet Inc, Mountain View, CA). Residents were given unique identifiers to anonymize resident surveys and metrics of procedural proficiencies.

Materials

We filmed a 2D-360-degree stereoscopic training VR video of an intracavitary cervical brachytherapy procedure on a Zoe pelvic simulator (Gaumard Scientific, Miami, FL). Equipment included a tandem and ring applicator (Mick Radionuclear Instruments, Eckert & Zeigler BeBig, Berlin, Germany) and Alatus vaginal packing balloons (Angiodynamics, Latham, NY). Narration over the video was instructive describing tools, procedure steps, and options. The video is freely available online at https://youtu.be/rs42NXRyhkk. Additional details of the VR SBE video are described in our prior work (Taunk et al., 2021).

The two virtual reality delivery platforms used in this study were the Oculus Go stand-alone VR goggles and the Google cardboard VR viewer. The Google cardboard VR viewer requires a smartphone and a video is available online depicting setup of the Google cardboard VR viewer at https://youtu.be/4zhU20FBLXA. It is important to note that there was one centrally located IHVR system for all residents randomized to the IHVR arm. However, each resident in the CVVR arm was given their own VR device.

IHVR video resolution on Oculus Go was 2560x1140 (1280x1140 per eye), 89° field of view. CVVR resolution was dependent on smartphone but ranged from 1334x750 (667x750 per eye) on iPhone 8, to 2532x1170 (1266x1170 per eye) on iPhone 13 Pro, 60^o field of view on Google Cardboard. In the CVVR arm, YouTube (Alphabet Inc, Mountain View, CA) was used as the hosting platform as it automatically splits content into left and right eye signals. In the IHVR arm, Skybox VR player (SOURCE Studio, Burnsville, NC) in the Oculus environment was used as the hosting platform.

Survey data was collected anonymously from every participant in the study using Qualtrics platform (Qualtrics International Inc, Provo, UT).

Design and procedure

After creation of the VR training video and randomization of participants to either Oculus Go or Google Cardboard, both groups completed validated pre-VR SBE surveys describing demographic information, self-confidence, and knowledge of procedural steps (Pulijala et al., 2018; Zhao et al., 2018; Taunk et al., 2021). Both groups were given unlimited access to not only their randomized VR technology but also to current training materials including slideshows, 2D videos, and use of the pelvic simulator.

After a minimum of 6 weeks of VR SBE access, residents performed a timed intracavitary brachytherapy procedure on the pelvic simulator during which five key objectively validated proficiencies of implant quality were measured (Supplemental Material A) (Zhao et al., 2018).

Following the simulated brachytherapy procedure, residents completed a post-VR SBE survey describing self-confidence and knowledge of procedural steps, and resident perceived usefulness of the VR technology, and their learning experience (Lee, 2010; Tcha-Tokey et al., 2016; Stepan et al., 2017).

A cost-effectiveness analysis comparing each technology was conducted on 20 May 2022. Cost of the currently available VR platform headset was gathered directly from the companies’ e-commerce stores (arvr.google.com/cardboard/get-cardboard/, www.oculus.com/quest-2). Data on radiation oncology training program number and size was gathered from Doximity (Doximity Inc, San Francisco, CA) (residency.doximity.com/specialties/59-radiation-oncology).

This study was approved by our institutional IRB.

Statistical analyses

Likert scale questions were compared using the Exact Wilcoxon Rank sum accompanied with Wilcoxon Rank Sum with Z test. Normality was assessed using the Shapiro Wilks test and equality of variance was tested using Levene’s test. Matched paired analyses were also implemented when appropriate to compare pre-to post-simulation changes in the both VR technology groups. The analysis was not adjusted for multiplicity. Welch’s t test was used to compare implant time differences. All categorical variables (e.g., sex, post-graduate year (PGY) level, last usage, etc) were compared using Fisher’s exact test. Statistical analyses were conducted using SAS version 9.2 and GraphPad Prism 9 (GraphPad Software, San Diego, CA) software packages. Two-tailed p-values less than a type I error rate of 0.05 were considered statistically significant. Figures were created using GraphPad Prism 9. All bar graphs display median values with 95% confidence intervals, unless stated otherwise.

Results

Study population

There were 13 residents who participated in the VR curriculum. These included four post-graduate year (PGY)-2, three PGY-3, two PGY-4, and four PGY-5. All 13 completed the pre-simulation and post-simulation survey. Overall, 84.5% (n = 11) of the participants were male. Seven (53.8%, n = 7) of residents had previously performed an intracavitary brachytherapy procedure, while six (46.2%, n = 6) had no prior experience. There were six residents randomized to IHVR and seven residents randomized to CVVR. There was no difference in sex between the two groups. Although there was not statistically significant difference in PGY level between the groups, there was a numerically higher percentage of PGY-2/3 residents in the CVVR group compared to the IHVR group (71% [n = 5] vs. 33% [n = 2], Fisher’s exact test, p = 0.53). Additionally, a higher percentage for residents in the IHVR had completed prior brachytherapy experience (83% [n = 5] vs. 29% [n = 2]). No residents in both groups had extensive experience with either VR technology, defined as more than 5 uses (Table 1).

TABLE 1

TABLE 1. Baseline Data on VR groups.

Subjective and objective procedural proficiency

On pre-VR SBE and post-VR SBE survey, resident self-confidence, self-report assembling skill, and self-reported insertion skill in both CVVR and IHVR groups were independently collected. In these surveys, a score of five represents confidence of performing the procedure at the level of an attending physician.

Residents in the CVVR reported an improvement in overall self-confidence from a median score of two before VR SBE to a median score of 4 (paired t-test resulting in a t (13) = -4.6, p = 0.0038) see Figure 1. The Shapiro-Wilks test showed a normal distribution (W = 0.86, p = 0.14). Moreover, residents in the CVVR arm described an improvement in self-report assembling skills from a median of two to a median of 4 after VR SBE (paired t-test resulting in a t (13) = -4.6, p = 0.0038). The Shapiro-Wilks test showed a normal distribution (W = 0.86, p = 0.14). Likewise, the perceived insertion skills in the CVVR arm improved from a median of two–four following VR experience (paired t-test resulting in a t (13) = -6.0, p = 0.001). The Shapiro-Wilks test showed a normal distribution (W = 0.83, p = 0.09).

FIGURE 1

FIGURE 1. Median Pre-simulation and Post-simulation self-reported (A) overall confidence, (B) perceived insertion skill, (C) perceived insertion skill, and (D) comfort with independently performing. Bar graph represents the median response score ± 95% confidence interval in each group, CVVR and IHVR, before and after simulation. * represents statistical significant, p < 0.05 based on paired t-test.

A similar assessment of pre- and post-survey self-reported proficiencies was completed in the IHVR arm. Residents in the IHVR group reported an improvement in overall confidence from a median of three–4 (paired t-test resulting in a t (11) = -3.8, p = 0.01). The Shapiro-Wilks test showed a normal distribution (W = 0.87, p = 0.21). Residents in the IHVR arm also noted an improvement in assembling skills from median of three–4 (paired t-test resulting in a t (11) = - 196 3.9, p = 0.01). The Shapiro-Wilks test showed a normal distribution (W = 0.83, p = 0.10). Residents did not report an improvement in perceived insertion skills with a median of three before and median of 3 after the VR SBE (paired t-test resulting in a t (11) = -2.0, p = 0.10). The Shapiro-Wilks test showed a normal distribution (W = 0.82, p = 0.09).

On matched paired analysis, there were also non-statistically significant improvements in the ability to correctly order the steps of the brachytherapy procedure on post-simulation survey compared to pre-simulation survey in both VR viewer groups (CVVR: 71.4% from 42.9%, p = 0.59; IHVR: 83.3% from 50.0%, p = 0.55).

Average time required for implant was similar in both group (mean: CVVR - 200 s vs IHVR - 235 s, Welch’s t test, p = 0.38). Median objective proficiencies of implant quality were 5/5 for both the CVVR and IHVR group.

Comparison of VR technology as learning tool

On Shapiro Wilks test, all Likert scale questions were not normally distributed. On Levene’s test, most questions had equal variance between the two groups. Z test was mostly concordant with the Exact Wilcoxon Rank Sum Test. When evaluated on a Likert scale with 5 as strongly agree and 1 as strongly disagree, the median response rate evaluating CVVR versus IHVR as a useful learning tool (4 vs. five; W [NCVVR = 7, NIHVR = 6] = 53.00, z = 1.61, p = 0.14), enjoyable learning experience (4 vs. five; W [NCVVR = 7, NIHVR = 6] = 52.50, z = 1.58, p = 0.11), engaging tool (4 vs. five; W [NCVVR = 7, NIHVR = 6] = 46.50, z = 1.35, p = 0.18), and easy to use (5 vs. 5, W [NCVVR = 7, NIHVR = 6] = 48.00, z = 0.97, p = 0.49) was similar. The residents in both groups felt strongly that they would recommend a similar technology to another trainee (median: CVVR - 4 vs. IHVR - 5, W [NCVVR = 7, NIHVR = 6] = 54.00, z = 2.01, p = 0.07). Residents in both the CVVR and the IHVR groups felt their respective VR technology made them more interested to learn the topic. Residents felt that both technologies were successful in delivering factual information, improving understanding of basic concepts, helping identify the main/important issues of brachytherapy, and helping summarize material. Residents in groups felt that their VR technology stimulated them to learn more, increased their learning/academic performance, enhanced effectiveness of learning, allowed them to progress at their own pace, and supported their learning. Residents in both groups agree their respective VR technologies were meaningful in their learning. Moreover, residents believed what they learned from the VR technology could be applied in a real context. Both the CVVR and the IHVR created a sense of presence “being there” for the residents. A representative figure of some of these key findings is shown in Figure 2.

FIGURE 2

FIGURE 2. Representative self-reported evaluation of virtual reality technology. The heat map shows the percentage of trainees in each group, CVVR and IHVR, responding to each question (x-axis) to options in the Likert scale (y-axis). Yellow represents 100% response and blue box represents 0% response.

The usability and flow of the VR-based simulation experience was evaluated on a scale of 1–10 with 10 representing strongly agree. The median response rates were largely similar in the two groups. Residents felt that both VR platforms involved them (CVVR–9 vs. IHVR–9.5, W [NCVVR = 7, NIHVR = 6] = 46.50, z = 0.61, p = 0.57) were compelling (CVVR–10 vs. IHVR–9.5, W [NCVVR = 7, NIHVR = 6] = 43.50, z = 0.16, p = 0.86), was responsive (CVVR–9 vs. IHVR–8.5, W [NCVVR = 7, NIHVR = 6] = 43.00, z = 0.07, p = 0.92), and felt natural (CVVR–9 vs. IHVR–8.5, W [NCVVR = 7, NIHVR = 6] = 44.00, z = 0.22, p = 0.84). Both groups were actively able to survey the environment (CVVR–9 vs. IHVR–10, W [NCVVR = 7, NIHVR = 6] = 44.50, z = 0.32, p = 0.71) and felt proficient in interacting with the virtual environment (CVVR–9 vs. IHVR–8.5, W [NCVVR = 7, NIHVR = 6] = 39.50, z = -0.30, p = 0.72). Neither group felt that their VR technology was inconsistent (CVVR–1 vs. IHVR–1.5, W [NCVVR = 7, NIHVR = 6] = 34.50, z = -1.09, p = 0.36). More residents felt that the CVVR was more cumbersome than IHVR (CVVR–3 vs. IHVR–1, W [NCVVR = 7, NIHVR = 6] = 27.50, z = -2.08, p = 0.04). Overall, both groups felt that their VR technology was enjoyable (CVVR–8 vs. IHVR–9.5, W [NCVVR = 7, NIHVR = 6] = 40.00, z = -0.22, p = 0.82). Comprehensive Likert scale response data is depicted in Table 2.

TABLE 2

TABLE 2. Evaluation of virtual reality technology as a learning tool.

There were stark differences in the time between last usage of the VR technology with 50% of the residents in the IHVR group (n = 6) completing the last VR SBE more than 2 weeks before the procedure (Figure 3). In contrast, the majority of residents in the CVVR group (n = 7, 57%) completed the last VR SBE within 1 h before the procedure. Despite these qualitative differences, there was not enough power to show statistical significance with these findings on Fisher’s Exact test (p = 0.20).

FIGURE 3

FIGURE 3. Distribution of time of last use of virtual reality technology prior to the actual procedure. The bar graph shows percentage (y-axis) of trainees in CVVR (green) and IHVR (red) based on their last use of the virtual reality technology (x-axis). Fisher’s Exact test (p = 0.20) shows no difference between the two groups for last use of virtual reality technology.

There were some challenges with both VR technologies expressed by the residents in free text comments. The IHVR group would have liked to have the VR technology available closer to the time of the actual procedure and wished the IHVR had better battery life. The CVVR group expressed challenges with fitting their phones into the cardboard and would have liked if the viewer had straps, so the residents did not have to hold them.

Cost effectiveness analysis

There are 90 radiation oncology residency programs with a total of 813 resident physicians in the United States as of 2020. The costs of the VR platforms were based on analysis completed in May 2022. The Cardboard viewer can be made for free using everyday items available at the local hardware store but is also commercially sold. One model Maxbox VR (Pyrite VR LTD, Bromley, UK) costs $9 + tax USD. CVVR requires smartphone usage and therefore is another cost associated with creating a CVVR-based VR SBE curriculum. The cost of a smartphone was excluded from this analysis because prior studies of US medical residents have shown 90–100% of medical residents already own smartphones (Katz-Sidlow et al., 2012; Boruff and Storie, 2014; Terry and Terry, 2018). The cost of WiFi and cellular services is excluded. All residents in this study used their own smartphone and all smartphones were compatible with CVVR. Each resident already had pre-downloaded YouTube app and no cost was incurred for using the hosting platform. No resident in this study had their own IHVR device available so there was a centrally located Oculus Go Stand-alone headset with equal access to all residents. Because the Oculus Go Stand-alone is no longer commercially available by the manufacturer, Quest two was utilized for cost effectiveness analysis. Quest two costs $299 + tax USD. Table 3 shows the total cost for the VR technology. If purchasing a single VR headset for each residency program, the cost of initiating a IHVR program is ∼33x or 3,322% increase in cost than CVVR program ($26,910 vs. $810, respectively). Similarly, if a VR headset was given to each individual resident, the program would cost ∼33x or 3,322% more with IHVR program compared to CVVR program ($243,097 for IHVR compared to $7,317 for CVVR). Even in a setup similar to this study, in which each resident receives a CVVR ($7,317) or there is only one IHVR for the whole program ($26,910), CVVR programs cost $19,593 less or result in 368% savings.

TABLE 3

TABLE 3. Cost of virtual reality delivery platform for residency training.

Discussion

Medical education evolved over time. Initially, education relied on a combination of patient experiences and 2D pictures, text, slideshows and non-immersive videos. SBE tools have been shown to be effective complements to real patient experiences. However, the COVID-19 pandemic has unmasked the needs to develop of distributed, immersive and repeatable education curriculum, such as VR SBE. Our group previously demonstrated that use of IHVR-based simulation can augment resident confidence and technical skill when performing brachytherapy procedures (Taunk et al., 2021). However, with expansion of VR use in medical education, we need to identify technology that is not only an effective educational tool but also cost-effective so a scalable VR SBE program can be created. To our knowledge, this study is the first to compare virtual reality technologies for medical education.

Virtual reality as an educational tool has been demonstrated as effective in multiple different medical sub-disciplines (Ahlberg et al., 2007; Khan et al., 2019; Baniasadi et al., 2020; Ros 2021). Early VR studies utilized computer-based VR delivery platforms. A study utilizing LapSim (Surgicalscience, Goteborg, Sweden) VR simulator showed that VR education for trainees performing laparoscopic cholecystectomy improved resident error rate and speed. LapSim is a software program used on Windows XP computer with a virtual laparoscopic interface (Ahlberg et al., 2007). Thirteen residents, who were inexperienced in laparoscopic procedures, were randomized to VR training or control arm. Residents had their first 10 procedures video-taped and independently reviewed. Residents in the VR-based education arm consistently had lower rates of errors and shortened surgical times when compared to the control arm. In another study, sixteen surgical residents were randomized to VR training or control for laparoscopic cholecystectomy. Non-VR trained residents were nine times more likely to injure the gallbladder or burn non-target tissue. Moreover, there were six times less errors during the procedure for residents who underwent VR training. Similar to the prior study, the virtual reality platform (MIST VR system) is a desktop PC based VR delivery system (Seymour et al., 2002). One limitation of a computer-based system is it does not create a 360-degree immersive environment. This study found residents who use an immersive system, such as CVVR and IHVR platform, were stimulated to learn more, felt the information they learned could be applied to a real context, and felt a sense of presence “being there” during the virtual procedure.

With advancements in VR delivery techniques, Pulijala et al. showed that Oculus Rift based immersive VR experience improved self-confidence in oral facial maxillary surgical residents compared to a control group when performing Le Fort I osteotomy (Pulijala et al., 2018). Similarly, Stepan et al. showed that Oculus Rift VR system could be used as an educational tool for teaching neuroanatomy. Moreover, sixty-six medical students were randomized to VR experience versus online textbook based educational control. Medical students in the VR group felt that the learning experience was more engaging, enjoyable, and useful. Furthermore, they scored higher on motivation assessment (Stepan et al., 2017). Ros et al. evaluated the pedagogical value of virtual reality training in lumbar puncture training. Learners were randomized to traditional education or virtual reality. They found the learners in the virtual reality group performed the procedure with less error and in less time (Ros et al., 2021). Similar to these studies, our prior study used a similar IHVR system and showed similar results with improvement in self-confidence and engagement in residents undergoing cervical intracavitary brachytherapy training (Taunk et al., 2021). Although an effective educational tool, IHVR is expensive and can make scaling the VR SBE challenging. In this study, CVVR platform was found to be a cost-effective alternative to IHVR as an immersive video educational tool. Both VR delivery systems improve resident self-confidence and self-perceived technical proficiency. There were also no significant differences in average time to implant and median objective technical proficiencies between the two groups. Trainees found both platforms to be useful, engaging, and enjoyable. Residents would also recommend the technology to other trainees. Of note, in this study, there was only one IHVR headset in a central location for the residents in the IHVR arm but each resident in the CVVR arm had their own viewer, which may account for a lack of difference noted between the two VR platforms.

There were several limitations to the current study. One limitation is that this is a single-institution study. There were small resident numbers which may limit statistical power for some comparisons conducted, and further investigation with larger learners need to be completed to validate the results. In light of that, this is the only and largest series comparing effectiveness of two separate VR delivery platforms in medical education. Furthermore, it is important to note the baseline characteristics of this study. 84.5% of participants were male. Although this was representation of our training institution, it is not representation of the gender distribution in training nationally. As such, any future validation study should endeavor to have a more comprehensive representation of gender. Moreover, a higher percentage of residents in the IHVR had completed at least 2 years of training and had prior brachytherapy experience. Even given that difference, it is encouraging to see that those in the IHVR group still reported improvement in overall confidence.

Another limitation is that this study has a narrow scope and only investigates immersive video as an educational tool. It does not compare other features of IHVR such as hand tracking that can improve interaction with the virtual learning environment. The IHVR technology is constantly advancing and newer IHVR systems, such as the Quest two utilized in the cost effective analysis, may continue to improve immersive experience with outward facing camera allowing hand tracking and limited haptic feedback. Although this feature may be available in CVVR in the future, it will likely need expensive smartphones to utilize this feature. Another limitation is that the educational video was filmed in 2D-360°. 3D-360-degree video acquisition may also improve immersive educational experience for residents. However, 3D-360-degree video acquisition does require an additional camera and more sophisticated post production for stereoscopic stitching that may pose a challenge for expansion of the VR curriculum. Moreover, this study is limited to 360-degree video capturing and does not discuss the role of 180-degree virtual reality experience and educational value. Both technologies assume some degree of technology literacy for appropriate use. This study does not investigate difference in symptoms related to technology use such as dizziness/vertigo, which would be an important consideration to expand the virtual reality education program. Future studies should determine difference in technology-related side effects. The cost effectiveness analysis is restricted to the dissemination of a virtual reality based training program and does not include projected monetary value from improved training experience.

However, these limitations present significant opportunities. A low-cost delivery system would allow for faster and more widespread distribution of academic medical content without large limiting cost burdens. This assumes the presence of smartphones with either preloaded content, or access to fast internet with WiFi or cellular service. Expansion of VR curriculum would simply require additional video content which can be scaled not just within radiation oncology procedures, but also across other disciplines of medicine, both procedural and anatomic education. Moreover, this VR delivery system could be expanded to education of nursing, other healthcare staff, and even patients.

In conclusion, this pilot study compared IHVR and CVVR as VR delivery systems. Both VR SBE platforms showed improvement in residents’ self-confidence, and objective and subjective technical skills when performing intracavitary cervical brachytherapy. Resident physicians found both IHVR and CVVR immersive, highly engaging, and enjoyable educational tools. This study found CVVR as a cost effective alternative to IHVR when developing a video-based VR SBE curriculum.

Data availability statement

The raw data supporting the conclusion of this article will be made available by the authors, without undue reservation.

Ethics statement

The studies involving human participants were reviewed and approved by the Institutional Review Board, University of Pennsylvania. The patients/participants provided their written informed consent to participate in this study.

Author contributions

NKS, NKT, TL developed the concept. NKS collected the data. NKS, RM and XW completed statistical analysis. NKS wrote the manuscript. NKT, RM, EH, SA, JWT, XW TL revised the manuscript. NKS and TL obtained funding for the study.

Funding

This work was supported in part by the Evan McCabe Fund Grant (TL) and ACRO Luther Brady Educational Grant (NS).

Acknowledgments

Evan McCabe Fund, American College of Radiation Oncology and American Brachytherapy Society.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

Ahlberg, G., Enochsson, L., Gallagher, A. G., Hedman, L., Hogman, C., McClusky, D. A., et al. (2007). Proficiency-based virtual reality training significantly reduces the error rate for residents during their first 10 laparoscopic cholecystectomies. Am. J. Surg. 193, 797–804. doi:10.1016/j.amjsurg.2006.06.050

PubMed Abstract | CrossRef Full Text | Google Scholar

Bambakidis, N. C., and Tomei, K. L. (2020). Editorial. Impact of COVID-19 on neurosurgery resident training and education. J. Neurosurg. 2, 10–11. doi:10.3171/2020.3.JNS20965

PubMed Abstract | CrossRef Full Text | Google Scholar

Baniasadi, T., Ayyoubzadeh, S. M., and Mohammadzadeh, N. (2020). Challenges and practical considerations in applying virtual reality in medical education and treatment. Oman Med. J. 35 (3), e125. doi:10.5001/omj.2020.43

PubMed Abstract | CrossRef Full Text | Google Scholar

Boruff, J. T., and Storie, D. (2014). Mobile devices in medicine: A survey of how medical students, residents, and faculty use smartphones and other mobile devices to find information. J. Med. Libr. Assoc. 102 (1), 22–30. doi:10.3163/1536-5050.102.1.006

PubMed Abstract | CrossRef Full Text | Google Scholar

Dimopoulos, J. C., Potter, R., Lang, S., Fidarova, E., Georg, P., Korr, W., et al. (2009a). Dose-effect relationship for local control of cervical cancer by magnetic resonance image-guided brachytherapy. Radiother. Oncol. 93, 311–315. doi:10.1016/j.radonc.2009.07.001

PubMed Abstract | CrossRef Full Text | Google Scholar

Dimopoulos, J. C., Lang, S., Kirisits, C., Fidarova, E. F., Berger, D., Georg, P., et al. (2009b). Dose-volume histogram parameters and local tumor control in magnetic resonance image guided cervical cancer brachytherapy. Int. J. Radiat. Oncology*Biology*Physics 75, 56–63. doi:10.1016/j.ijrobp.2008.10.033

PubMed Abstract | CrossRef Full Text | Google Scholar

Donnelly, E. D., Sachdev, S., Zhang, H., Kang, Z., Broadwater, K., and Strauss, J. B. (2020). Development of a gynecologic brachytherapy curriculum and simulation modules to improve radiation oncology trainees' skills and confidence. Brachytherapy 19, 732–737. doi:10.1016/j.brachy.2020.09.016

PubMed Abstract | CrossRef Full Text | Google Scholar

Grassini, S., Laumann, K., and Skogstad, M. (2020). The use of virtual reality alone does not promote training performance (but sense of presence does). Front. Psychol. 11, 1743. doi:10.3389/fpsyg.2020.01743

PubMed Abstract | CrossRef Full Text | Google Scholar

Gupta, D. K., Khandker, N., Stacy, K., Tatsuoka, C. M., and Preston, D. C. (2017). Utility of combining a simulation-based method with a lecture-based method for fundoscopy training in neurology residency. JAMA Neurol. 74 (10), 1223–1227. doi:10.1001/jamaneurol.2017.2073

PubMed Abstract | CrossRef Full Text | Google Scholar

Han, K., Milosevic, M., Fyles, A., Pintilie, M., and Viswanathan, A. N. (2013). Trends in the utilization of brachytherapy in cervical cancer in the United States. Int. J. Radiat. Oncology*Biology*Physics 87, 111–119. doi:10.1016/j.ijrobp.2013.05.033

PubMed Abstract | CrossRef Full Text | Google Scholar

Irvin, W., Rice, L., Taylor, P., Andersen, W., and Schneider, B. (2003). Uterine perforation at the time of brachytherapy for carcinoma of the cervix. Gynecol. Oncol. 90, 113–122. doi:10.1016/s0090-8258(03)00230-0

PubMed Abstract | CrossRef Full Text | Google Scholar

Katz-Sidlow, R. J., Ludwig, A., Miller, S., and Sidlow, R. (2012). Smartphone use during inpatient attending rounds: Prevalence, patterns and potential for distraction. J. Hosp. Med. 7 (8), 595–599. doi:10.1002/jhm.1950

PubMed Abstract | CrossRef Full Text | Google Scholar

Khan, R., Plahouras, J., Johnston, B. C., Scaffidi, M. A., Grover, S. C., and Walsh, C. M. (2019). Virtual reality simulation training in endoscopy: A cochrane review and meta-analysis. Endoscopy 51 (7), 653–664. doi:10.1055/a-0894-4400

PubMed Abstract | CrossRef Full Text | Google Scholar

Lee, M. C. (2010). Explaining and predicting users' continuance intention toward e-learning: An extension of the expectation–confirmation model. Comput. Educ. 54 (2), 506–516. doi:10.1016/j.compedu.2009.09.002

CrossRef Full Text | Google Scholar

Lorello, G. R., Cook, D. A., Johnson, R. L., and Brydges, R. (2014). Simulation-based training in anaesthesiology: A systematic review and meta-analysis. Br. J. Anaesth. 112 (2), 231–245. doi:10.1093/bja/aet414

PubMed Abstract | CrossRef Full Text | Google Scholar

Lynch, B., Einspruch, E. L., Nichol, G., Becker, L. B., Aufderheide, T. P., and Idris, A. (2005). Effectiveness of a 30-min CPR self-instruction program for lay responders: A controlled randomized study. Resuscitation 67, 31–43. doi:10.1016/j.resuscitation.2005.04.017

PubMed Abstract | CrossRef Full Text | Google Scholar

Marcrom, S. R., Kahn, J. M., Colbert, L. E., Freese, C. M., Doke, K. N., Yang, J. C., et al. (2019). Brachytherapy training survey of radiation oncology residents. Int. J. Radiat. Oncology*Biology*Physics 103 (3), 557–560. doi:10.1016/j.ijrobp.2018.10.023

PubMed Abstract | CrossRef Full Text | Google Scholar

Mesko, S., Chapman, B., Tang, C., Kudchardker, R. J., Bruno, T. L., Sanders, J., et al. (2020). Development, implementation, and outcomes of a simulation-based medical education (SBME) prostate brachytherapy workshop for radiation oncology residents. Brachytherapy 19, 738–745. doi:10.1016/j.brachy.2020.08.009

PubMed Abstract | CrossRef Full Text | Google Scholar

Monaghan, A. (2020). Medical teaching and assessment in the era of COVID-19. J. Med. Educ. Curric. Dev. 7, 238212052096525. doi:10.1177/2382120520965255

PubMed Abstract | CrossRef Full Text | Google Scholar

National Cancer Institute (NCI) (2016). Radiation therapy and you: Support for people with cancer. Bethesda, Maryland: NCI. Available at: https://www.cancer.gov/publications/ (Accessed May 12, 2022).

Google Scholar

Portelli, M., Bianco, S. F., Bezzina, T., and Abela, J. E. (2020). Virtual reality training compared with apprenticeship training in laparoscopic surgery: A meta-analysis. annals 102 (9), 672–684. doi:10.1308/rcsann.2020.0178

PubMed Abstract | CrossRef Full Text | Google Scholar

Pulijala, Y., Ma, M., Pears, M., Peebles, D., and Ayoub, A. (2018). Effectiveness of immersive virtual reality in surgical training – a randomized control trial. J. Oral Maxillofac. Surg. 76 (5), 1065–1072. doi:10.1016/j.joms.2017.10.002

PubMed Abstract | CrossRef Full Text | Google Scholar

Ros, M., Neuwirth, L. S., Ng, S., Debien, B., Molinari, N., Gatto, F., et al. (2021). The effects of an immersive virtual reality application in first person point-of-view (IVRA-FPV) on the learning and generalized performance of a lumbar puncture medical procedure. Educ. Technol. Res. Dev. 69, 1529–1556. doi:10.1007/s11423-021-10003-w

CrossRef Full Text | Google Scholar

Seymour, N. E., Gallagher, A. G., Roman, S. A., O’Brien, M. K., Bansal, V. K., Andersen, D. K., et al. (2002). Virtual reality training improves operating room performance results of a randomized, double-blinded study. Ann. Surg. 236 (4), 458–464. doi:10.1097/00000658-200210000-00008

PubMed Abstract | CrossRef Full Text | Google Scholar

Stepan, K., Zeiger, J., Hanchuk, S., Signore, A. D., Shrivastava, R., Govindaraj, S., et al. (2017). Immersive virtual reality as a teaching tool for neuroanatomy. Int. Forum Allergy Rhinol. 7 (10), 1006–1013. doi:10.1002/alr.21986

PubMed Abstract | CrossRef Full Text | Google Scholar

Taunk, N. K., Shah, N. K., Hubley, E., Anamalayil, S., Trotter, J. W., and Li, T. (2021). Virtual reality-based simulation improves gynecologic brachytherapy proficiency, engagement, and trainee self-confidence. Brachytherapy 20 (4), 695–700. doi:10.1016/j.brachy.2021.03.003

PubMed Abstract | CrossRef Full Text | Google Scholar

Tcha-Tokey, K., Christmann, O., Loup-Escande, E., and Richir, S. (2016). Proposition and validation of a questionnaire to measure the user experience in immersive virtual environments. Int. J. Virtual Real. 16 (1), 33–48. doi:10.20870/ijvr.2016.16.1.2880

CrossRef Full Text | Google Scholar

Terry, D. L., and Terry, C. P. (2018). Smartphone use and professional communication among medical residents in primary care. PRiMER 2, 18. doi:10.22454/PRiMER.2018.766371

PubMed Abstract | CrossRef Full Text | Google Scholar

Viswanathan, A. N., Moughan, J., Small, W., Levenback, C., Iyer, R., Hymes, S., et al. (2012). The quality of cervical cancer brachytherapy implantation and the impact on local recurrence and disease-free survival in radiation therapy oncology group prospective trials 0116 and 0128. Int. J. Gynecol. Cancer 22, 123–131. doi:10.1097/IGC.0b013e31823ae3c9

PubMed Abstract | CrossRef Full Text | Google Scholar

Zhao, S., Francis, L., Todor, D., and Fields, E. C. (2018). Proficiency-based cervical cancer brachytherapy training. Brachytherapy 17, 653–659. doi:10.1016/j.brachy.2018.03.001

PubMed Abstract | CrossRef Full Text | Google Scholar