Practical considerations of clinical XR (AR/VR) deployments

Extended Reality (XR), which includes both Augmented Reality (AR) and Virtual Reality (VR), shows great promise in healthcare, with applications ranging from surgical simulations to patient rehabilitation and education. To ensure successful deployment, it is essential to address a wide range of different challenges, including those related to clinical efficacy, safety, ethics, technical requirements, institutional demands, provider and hardware considerations, as well as regulatory and reimbursement issues, all within the broader context of the healthcare system. Artificial intelligence, monopolistic payers, and the lack of a clear boundary between the consumer and healthcare spaces all represent both new challenges as well as opportunities. To fully harness XR’s potential, collaboration among technologists, clinicians, and policymakers is essential, ensuring the technology enhances patient care and education while maintaining safety and effectiveness.

1 Introduction

Extended Reality (XR), encompassing both Augmented Reality (AR) and Virtual Reality (VR), has shown significant promise in clinical settings. Its applications range from surgical simulations and medical training to patient assessments, rehabilitation and therapeutic interventions (Figure 1 below illustrates the overall landscape of the XR within the healthcare space). However, the deployment of XR in clinical environments necessitates careful consideration of several practical aspects to ensure usability, efficacy, safety, and user acceptance. While there is a myriad of available literature surrounding the use of XR for clinical scenarios, illnesses, or disease states, there is a paucity of available information surrounding the practical translation of these advances to the bedside. From the perspective of clinicians that build solutions, this overview explores some of the common challenges, considerations, and approaches to overcome these challenges to accelerate the development of health applications leveraging XR. We begin with a brief literature review to illustrate clinical potential; we then dive into a discussion surrounding practical considerations and offer potential approaches to overcome them. Given the need, we aim for a broadly applicable perspective that delivers value for clinicians, developers, and institutional decision makers alike.

Figure 1

Figure 1. Landscape of the XR in healthcare space.

2 Literature review

While an in-depth systematic literature review is beyond the scope of this article, the unstructured literature review in Table 1 below titled “Literature Overview Demonstrating Broad Applicability of XR Within Multiple Healthcare Domains” is meant to demonstrate some of the diversity and breadth of clinical applicability of XR within healthcare.

Table 1

Table 1. Literature overview demonstrating broad applicability of XR within multiple healthcare domains.

3 Discussion

The practical challenges facing XR applications within healthcare largely stem from this diverse set of possible use cases, from institutional policies, and from the patients and providers themselves. Please refer to Table 2: Challenges, Considerations, and Potential Approaches or Solutions for Applications in XR for Healthcare below. This includes designing intuitive interfaces for individuals with varying levels of technical proficiency as well as ensuring accessibility for individuals with physical disabilities. Research has shown that immersive technologies can significantly improve engagement and learning outcomes, making them highly effective for medical education and training. For example, AR has been utilized to overlay anatomical structures onto patients, aiding in anatomical education and pre-surgical planning (Peterson and Mlynarczyk, 2016; Moro et al., 2017). Additionally, the design of experiences for clinically related XR systems must be optimized to prevent fatigue, adverse events, and discomfort during use (Chen and Wu, 2023).

Table 2

Table 2. Challenges, considerations, and potential approaches or solutions for applications in XR for healthcare.

Appropriate hardware is essential for translating the potential of the technology into actual capabilities that can benefit patients. This represents a significant challenge given the hardware manufacturers propensity to protect (or limit) access data, and capabilities native to the device itself that may be essential for the implementation of certain use cases. Software interoperability with existing systems like Electronic Health Records (EHR) is also crucial to streamline workflows and avoid errors, yet developing standardized protocols for data flow and compatibility remains a complex set of tasks that has not yet been achieved at scale (Figure 2 below is a flattened diagram depicting the multifactorial nature of data as it relates to XR within the healthcare landscape). Fortunately, initial efforts such as the recently published taxonomy defining the landscape of medical extended reality (Spiegel et al., 2024) are an important step in the right direction that lay the foundation needed to be able to move forward. Protecting sensitive patient information within XR applications necessitates data security measures, which can be both technically demanding and resource-intensive to implement. Issues relating to security can further compound to additionally represent a significant potential source of technical debt given the lack of clear and widely accepted standards. Furthermore, achieving low latency and high performance for real-time interactions is challenging, requiring widespread high-bandwidth connectivity and advanced computing infrastructure that may not be readily available, especially in resource-constrained settings (e.g., rural areas). Usability and user training are also significant concerns, as complex interfaces and steep learning curves can hinder adoption, necessitating simple user-friendly designs and comprehensive training programs that are neither too time-consuming or costly. Additionally, ongoing maintenance and support are vital to ensuring continued functionality, which demand dedicated resources and expertise. Addressing these challenges is essential for successful XR deployment in clinical practice, necessitating collaborative efforts among technologists, clinicians, healthcare leaders and policymakers to develop reliable, secure, and effective solutions that meet the challenges of healthcare environments.

Figure 2

Figure 2. Key components of health data in XR context.

Clinical efficacy, expanding provider reach and capabilities, and evidence-based validation are all crucial for the acceptance and widespread adoption of XR in healthcare. The clinical efficacy for XR-based therapies has been demonstrated for a variety of use-cases spanning many specialties and settings. For instance, VR has been used successfully for pain management and physical rehabilitation, offering therapeutic exercises in a virtual environment that motivates patients to adhere to treatment regimens (Chan et al., 2018; Fernández-Álvarez et al., 2019). The potential for greater provider capability and greater patient access comes from a variety of different elements. Remote surgery assistance allows surgeons to receive real-time guidance from remotely located specialists, giving widespread access to specialized knowledge. XR provides interactive and immersive educational content for patients, enhancing their understanding of conditions and treatments for better compliance and engagement, and aids in preoperative planning by enabling surgeons to visualize complex anatomy and plan procedures. XR offers personalized rehabilitation programs that engage patients and track their progress in real-time. Furthermore, XR can simulate emergency scenarios for first responders, offering realistic training that improves readiness and decision-making in real emergencies. Given the promise, there is also a need for more clinical trials and rigorous research to establish the effectiveness of XR interventions and their impact on patient outcomes. Ideally, this evidence base will also support the development of guidelines and best practices for the use of XR in healthcare. With growing organizational efforts such as the Journal of Medical Extended Reality, the International Virtual Reality Healthcare Association, and the Medical Device Innovation Consortium, this space seems well prepared to evaluate, critique, and standardize many of the processes relating to evidence-based validation.

Furthermore, cost considerations, including the initial investment in XR technology, ongoing maintenance, and training for healthcare personnel, must be weighed against the potential benefits. Budget constraints and the return on investment (ROI) are significant factors influencing the decision to adopt XR in clinical practice within both public and private sectors. While the cost-benefit ratio must be assessed, it must also consider long-term savings from improved training outcomes, reduced complication rates, and enhanced patient care (including improvements in patient experience). Additionally, XR can decentralize care, expand access, automate processes, and improve productivity–which are also key considerations for any long-term healthcare vision.

Regulatory and ethical considerations present challenges that come with specific advantages and disadvantages. Regulatory bodies need to develop clear frameworks for the approval and oversight of XR applications in healthcare. On the one hand, stringent regulations aim to ensure patient safety, efficacy and consistency in quality, fostering trust among healthcare providers and patients. On the other hand, these regulations can slow down the adoption of innovative technologies and increase the cost and complexity of development and compliance. Any slowdowns are particularly costly given the constant stream of new hardware, and the potential inability to market obsolete technology once regulatory approvals are granted. While this is not a reason or justification for a less rigorous evaluation of safety and/or efficacy, it is an important stakeholder consideration that can likely be addressed through forward-thinking regulatory frameworks that maintain high standards for safety and efficacy while facilitating appropriate clearances in a way that helps to level disparities through improvements in access to care.

Lack of a clear boundary between healthcare and consumer-facing applications represents a potentially existential challenge for XR in healthcare startups. Afterall, who would go through the trouble of developing an application subject to all the types of challenges unique to healthcare only to have their clientele decide that free or extremely low-cost consumer facing applications are adequate despite their lack of domain specificity and diligence? While regulatory barriers and upfront “market research” may help to classify the obvious (e.g., an app to guide surgeons during complex procedures, vs. a relaxation app that plays 360 videos), this ambiguity will likely continue to represent a significant risk for even the most diligent teams developing solutions at the bleeding edge.

Artificial intelligence (AI) represents both a significant opportunity as well as a significant set of challenges for those building XR solutions within healthcare. While the challenges relating to AI within healthcare are beyond the scope of this perspective piece, AI carries its own set of complexities and regulatory challenges, and its meteoric rise also may create an unrealistic set of expectations for the traditionally slower moving XR space. Fortunately, recent efforts by the FDA have begun to provide solution makers with some clarity and guidance for the use of AI with software medical devices (Artificial Intelligence and Machine Learning, 2025; Artificial Intelligence-Enabled Device Software Functions, 2025; Artificial Intelligence and Machine Learning, 2024).

Modern affordable XR relies on AI (i.e., computer vision) to function, and it is perhaps poetic that these two technologies are once again beginning to converge to enable a set of powerful new possibilities including wearable personal health assistants, immersive mental health support, “go anywhere” escapism, voice cloning for mental health (Jumreornvong et al., 2024), personalized exposure therapy, and countless other applications. Unfortunately, despite the enormous potential for these “combination” approaches to improve the lives of patients, such efforts will likely have a compounded set of challenges originating from combining both AI and XR. And while these technologies can be built by small capable teams, without any clear directives this “combination” space runs the risk of being exclusive to large companies with the levels of resources needed to overcome the enormous set of institutional and regulatory challenges that may come with such combination approaches.

Ethical considerations, such as patient consent, privacy, and the potential for XR to alter the patient-provider relationship, also pose significant challenges. Ensuring robust consent processes and maintaining patient privacy are crucial for ethical deployment, but they can also be resource-intensive and can complicate implementation. Additionally, while XR can enhance patient engagement and clinical outcomes, there is a risk of it depersonalizing care or creating dependency on technology. Balancing all these considerations is essential to navigate the regulatory and ethical landscape effectively, ensuring that XR technologies are deployed responsibly and beneficially in clinical settings.

In conclusion, while XR holds substantial potential to transform patient care, clinical practice and education, its deployment requires a comprehensive approach that addresses technical, clinical, financial, regulatory, and ethical dimensions. These practical considerations are critical in translating experiences from the studio to the bedside, and many times issues related to these considerations take far longer to address than the construction and testing of the experiences themselves. Interdisciplinary teams are best suited to overcome these challenges, and successful integration of XR in healthcare depends on collaborative efforts among technologists, clinicians, researchers, administrators and policymakers. As we advance in this new paradigm of immersive care, future research should focus on harmonizing and integrating XR hardware, software, and IT systems. Additional priorities include developing tools to enhance the clinical utility of XR applications, improving privacy and security measures, and intentionally creating solutions at the intersection of AI and XR where it makes sense to do so.

Data availability statement

The original contributions presented in the study are included in the article/supplementary material, further inquiries can be directed to the corresponding author.

Author contributions

JM: Writing–original draft, Writing–review and editing. RP: Writing–original draft, Writing–review and editing. SC: Writing–original draft, Writing–review and editing.

Funding

The author(s) declare that no financial support was received for the research, authorship, and/or publication of this article.

Conflict of interest

The authors declare the following potential conflicts of interest: JM and RP are co-founders, employed by, and hold equity stakes in Waya Health, a for-profit company that develops extended reality (XR) solutions for healthcare. SC, MPH is a founder of the AI consulting business, Zero Hour Medical as well consultant/advisor and equity stakeholder in Turing Biosystems, a for-profit entity involved in healthcare technology innovation. SC receives no renumeration nor currently holds an equity stake in Waya Health All three authors actively participated in the conceptualization, research and writing of the article. Every effort has been made to ensure that the findings, opinions, and recommendations are unbiased and are supported by cited evidence.

Generative AI statement

The author(s) declare that Generative AI was used in the creation of this manuscript. All content in Table 1 was manually curated by the authors. We used a large language model to assist with the formatting this information into Table 1 and performed several checks to ensure accuracy.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

Artificial intelligence and machine learning (AI/ML)-Enabled medical devices (2024). Available at: https://www.fda.gov/medical-devices/software-medical-device-samd/artificial-intelligence-and-machine-learning-aiml-enabled-medical-devices. Accessed December 20, 2024.

Google Scholar

Artificial intelligence and machine learning in software as a medical device (2025). Available at: https://www.fda.gov/medical-devices/software-medical-device-samd/artificial-intelligence-and-machine-learning-software-medical-device. Accessed January 6, 2025.

Google Scholar

Artificial Intelligence-Enabled Device Software Functions (2025). Lifecycle management and marketing submission recommendations. FDA. Available at: https://www.fda.gov/regulatory-information/search-fda-guidance-documents/artificial-intelligence-enabled-device-software-functions-lifecycle-management-and-marketing. Accessed January 6, 2025.

Google Scholar

Chan, E., Foster, S., Sambell, R., and Leong, P. (2018). Clinical efficacy of virtual reality for acute procedural pain management: a systematic review and meta-analysis. PLoS One 13 (7), e0200987. doi:10.1371/journal.pone.0200987

PubMed Abstract | CrossRef Full Text | Google Scholar

Chen, Y., and Wu, Z. (2023). A review on ergonomics evaluations of virtual reality. Work 74 (3), 831–841. doi:10.3233/WOR-205232

PubMed Abstract | CrossRef Full Text | Google Scholar

Fernández-Álvarez, J., Rozental, A., Carlbring, P., Colombo, D., Riva, G., Anderson, P. L., et al. (2019). Deterioration rates in Virtual Reality Therapy: an individual patient data level meta-analysis. J. Anxiety Disord. 61, 3–17. doi:10.1016/j.janxdis.2018.06.005

PubMed Abstract | CrossRef Full Text | Google Scholar

Jumreornvong, O., Mao, H., Povea Galdo, G., Jaro, T., Schuler, K., Christopher, L., et al. (2024). Artificial intelligence enhanced virtual reality: a personalized, multilingual approach to pain management. J. Med. Ext. Real. 1 (1), 276–281. doi:10.1089/jmxr.2024.0038

CrossRef Full Text | Google Scholar

Moro, C., Štromberga, Z., Raikos, A., and Stirling, A. (2017). The effectiveness of virtual and augmented reality in health sciences and medical anatomy. Anat. Sci. Educ. 10 (6), 549–559. doi:10.1002/ase.1696

PubMed Abstract | CrossRef Full Text | Google Scholar

Peterson, D. C., and Mlynarczyk, G. S. A. (2016). Analysis of traditional versus three-dimensional augmented curriculum on anatomical learning outcome measures. Anat. Sci. Educ. 9 (6), 529–536. doi:10.1002/ase.1612

PubMed Abstract | CrossRef Full Text | Google Scholar

Spiegel, B. M. R., Rizzo, A., Persky, S., Liran, O., Wiederhold, B., Woods, S., et al. (2024). What is medical extended reality? A taxonomy defining the current breadth and depth of an evolving field. J. Med. Ext. Real 1 (1), 4–12. doi:10.1089/jmxr.2023.0012

PubMed Abstract | CrossRef Full Text | Google Scholar