Opportunities and Challenges of Virtual Reality in Healthcare – A Domain Experts Inquiry

In recent years, the applications and accessibility of Virtual Reality (VR) for the healthcare sector have continued to grow. However, so far, most VR applications are only relevant in research settings. Information about what healthcare professionals would need to independently integrate VR applications into their daily working routines is missing. The actual needs and concerns of the people who work in the healthcare sector are often disregarded in the development of VR applications, even though they are the ones who are supposed to use them in practice. By means of this study, we systematically involve health professionals in the development process of VR applications. In particular, we conducted an online survey with 102 healthcare professionals based on a video prototype which demonstrates a software platform that allows them to create and utilise VR experiences on their own. For this study, we adapted and extended the Technology Acceptance Model (TAM). The survey focused on the perceived usefulness and the ease of use of such a platform, as well as the attitude and ethical concerns the users might have. The results show a generally positive attitude toward such a software platform. The users can imagine various use cases in different health domains. However, the perceived usefulness is tied to the actual ease of use of the platform and sufficient support for learning and working with the platform. In the discussion, we explain how these results can be generalized to facilitate the integration of VR in healthcare practice.

1 Introduction

There is a wide range of potential applications of VR systems in the healthcare sector. Therapeutic support applications target, for instance, exposure therapy for patients with a fear of heights (Gonzalez et al., 2016; Kaur et al., 2019), spiders (Shiban et al., 2016; Mertens et al., 2019), or public speaking (Herumurti et al., 2019; Glémarec et al., 2021). Others use the effects of embodiment to deal with body perception disorders that often come with obesity or anorexia (Döllinger et al., 2019; Wolf et al., 2021). Physiological health can be supported by exercise-based training applications in which VR promises increased motivational and training effects. The same arguments apply to virtual physiotherapy measures, e.g. to recover from surgery or to counteract gait impairments (Hamzeheinejad et al., 2019; Kern et al., 2019; Gianola et al., 2020). Patients who suffer from the consequences of a stroke can regain motor skills by practicing with virtual replacements of their limbs (Wang W. et al., 2019; Moldoveanu et al., 2019) and even addiction scenarios can be simulated in virtual worlds which can serve as a basis for treatment (Thompson-Lake et al., 2015; Wang Y.-g. et al., 2019).

The fact that the possibilities are so versatile also means that it is relatively difficult for healthcare professionals to get a hold of applications that are suitable for their clientele. While there have been various efforts to making the creation of VR applications easier and more accessible (Latoschik and Tramberend, 2011; Fischbach et al., 2017; Neelakantam and Pant, 2017; Andone and Frydenberg, 2019; Santos and Cardoso, 2019; Zhang and Oney, 2020), they have not focused on the healthcare domain with its specific requirements. Moreover, these works had programmers and designers of virtual words in mind as the targeted users, whereas healthcare professionals likely lack both these backgrounds. In this study, we assume that there was a platform, similar to the concept presented by von Mammen et al. (2019), that specifically allowed health professionals to compose VR experiences that are meant to help them in their daily work. Based on this assumption, we embark on the quest to find out whether such a software platform had the potential to be established in healthcare practice and what the conditions for its acceptance would be. In this work we will refer to this as our “envisioned software platform”. We see potential users of this platform mainly from four specific healthcare domains. In the 1) therapy domain the envisioned software platform can be used to create VR applications for the treatment of anxiety, obsessive-compulsive, and body-awareness disorders. For example, this work by Döllinger et al. (2019) investigates the usage of VR for modulated body image perception for behavior therapy that addresses obesity challenges. In the 2) rehabilitation domain the envisioned software platform can be used to create, for example, VR applications that assist in regaining physical and motor abilities after an accident, surgery, or a stroke. So, this area mainly concerns physiotherapists. Kern et al. (2019); Hamzeheinejad et al. (2019) have examined the impact of VR in gait rehabilitation, and Laver et al. (2017); Zhang (2020) used VR for stroke rehabilitation and helped patients to recover from the impacts after spinal cord injuries. In the domain of 3) training, the envisioned software platform can be used to create controlled and save virtual training environments for the healthcare domain, e.g. for schooling first-aid, nursing, or surgical procedures. An example here comes from Mathur (2015) who worked on a low-cost VR system for medical training and found it to be more effective than a normal setup. In the 4) prevention domain the envisioned software platform can be used to create simulations that showcase the dangers of drug abuse or help with relaxation to prevent stress-related diseases. For example, the work by Nemire et al. (1999) looked into the possibilities of using a Hi-Fi VR system to prevent teens from smoking. So when we talk about the “potential users” of the envisioned software platform or “healthcare professionals” we usually refer to people who are employed in these four domains. There are other relevant areas, but they are not in focus for now; as they affect fewer patients or for other reasons. This includes, for example, the use of VR for training cognitive skills in dementia or for distraction in palliative patients. In the first area, however, good technologically supported applications already exist (Arlati et al., 2017; Zając-Lamparska et al., 2019). The second application area is smaller, and the provision of beautiful environments tends to be sufficient (Niki et al., 2019), which is probably also an important prerequisite for the other application areas.

Surveying the literature, we found no information about the view of healthcare professionals on VR. In particular, we could not find any studies that shed light on the prioritization of aspects for adaptation of VR from the view of healthcare professionals. Yet, their assessment is an invaluable source of information that has not yet been sufficiently embraced. This lack of information motivates this study. We aim at understanding whether and under which conditions healthcare specialists without technical background would embrace VR as a medium to enhance the effectiveness of their work. We are specifically interested in their prerequisites to autonomously create and utilize VR applications. We also want to determine their acceptance of an according software platform that would empower them to do so. We want to capture their opinions and concerns, as well as their suggestions toward design specifics and ideas about requirements of the platform.

As a result, we conducted an online inquiry based on a video that depicts our envisioned software platform. Our questions target the categories 1) perceived usefulness, 2) perceived ease of use, 3) general attitude, and 4) ethical concerns toward the envisioned software platform. Together, they address two aspects which are considered central in the context of technology development: The users and their needs in the sense of user-centered design (e.g., Karat and Karat, 2003) and a basis for ELSI-conform development, i.e., to reflect the ethical, legal, and social implications on individuals as well as on society and to engineer according measures (e.g., Greenbaum, 2015).

The gained knowledge can be used to spur the integration of VR into healthcare work routines and pave the way for VR out of the laboratory and into common practice. The captured requirements can immediately and effectively inform the development of a great number of health-related VR experiences, and be considered in according, generic authoring platforms for VR experiences. In addition, it can provide a basis for future research, especially considering concerns of the potential users.

The rest of this article is structured as follows: In the related work section, we report on VR authoring tools and technology-related acceptance and ethics studies. Then, in the methods section, we explain how the video prototype was created and how we structured and executed the online survey. In the fourth section, we list the results and in section five we discuss these results in terms of what they mean for the envisioned software platform and VR healthcare applications in general. Eventually, we will wrap the article up in the conclusion.

2 Related Work

This chapter presents preceding works that form the basis for our study. With regard to the technological background, this includes an introduction to previously developed systems similar to the envisioned software platform and a summary of their relevant components. To bolster the methodological and conceptual basis for the inquiry, preceding studies on the user acceptance of novel technologies by specific professional groups are presented. Moreover, we describe the Technology Acceptance Model and shed light on preceding research on the ethical aspects of using VR.

2.1 Evolution of Authoring Tools

Different from previous attempts, the market success of VR head-mounted displays and 3D controllers that started in the early 2000s has not been stopped short, yet. With pricing and capabilities making VR technology more accessible than ever and sales numbers growing rapidly (Angelov et al., 2020), the need for authoring platforms of VR experiences has grown as well (Ashtari et al., 2020).

The developments towards authoring tools started in the early nineties with technical libraries and toolkits. Scene graph libraries such as OpenGL Performer (Rohlf and Helman, 1994), Open Inventor (Strauss and Carey, 1992), and OpenSceneGraph (Wang and Qian, 2010) were created to realize visual simulations, virtual reality environments and other real-time 3D graphics applications. But oftentimes, most of these libraries focused on performance over ease of programmability. Also, developers often found it difficult to use these general-purpose libraries for specific problems (Bethel et al., 1999). Either these libraries were extended (Hesina et al., 1999), or additional packages and frameworks were developed upon these toolkits that could be used to bridge this gap (Kelso et al., 2002; Pavlik and Vance, 2012).

Further, many platforms supported networking and enabled the development of multi-user virtual environments (Carlsson and Hagsand, 1993; Greenhalgh and Benford, 1995; Allard et al., 2002). Latoschik and Tramberend (2011) created the open-source research platform Simulator X that is targeted at developing real-time interactive systems. Fischbach et al. (2017), for example, demonstrated how this platform can be used to increase the software quality of real-time interactive systems. Efforts like this made multiple developers and applications work in a common environment and communicate with each other without exposing the complex internal architectures. Frameworks like Avocado/Avango (Tramberend, 1999; Kuck et al., 2008), MR Toolkit (Shaw et al., 1993), or VR Juggler (Bierbaum et al., 2001) provided high-level APIs to mask the system level architecture from the author. Also, the generality of the data-flow architecture used in these platforms enabled an easy exchange of data across modules, and their users were empowered to create a wide variety of virtual environments (Figueroa et al., 2002; Allard et al., 2004, 2010, 2005).

These libraries and frameworks focused on software architectures and were thus to be used by programmers. A user without profound knowledge in software development would find it overwhelming to understand their respective workflows. The early developments, starting from the libraries and toolkits, slowly evolved into high fidelity authoring platforms over time. With the current advancements in technology and with the wide acceptance of VR, researchers and developers are innovating more sophisticated and effective solutions that can be used to produce more immersive and realistic content. A-frame¹ is a relatively new open-source framework from Mozilla to generate WebVR content (Neelakantam and Pant, 2017). Creating a VR experience is cost-effective in A-frame, but the initial learning curve can be an insurmountable obstacle to novice users (Santos and Cardoso, 2019). Many commercial and research solutions are addressing the design challenges faced by educators. CoSpaces², InstaVR³, or WondaVR⁴ are examples for commercial platforms primarily targeting the education sector. Among these, we found CoSpaces to be an easy-to-use, beginner-friendly tool, that allows the users to create simple VR environments using a drag and drop facility and allows to make the models interactive by employing a Scratch-like visual programming environment (Andone and Frydenberg, 2019). FlowMatic (Zhang and Oney, 2020) is a VR experience editor that uses a different approach to the design process by introducing a concept called immersive authoring. Most of the authoring platforms use 2D software to design the VR experience which takes off the freedom of using spatial information into consideration. FlowMatic makes use of VR, to create VR experiences taking advantage of the 3D spatial interaction. However, FlowMatic focuses only on making the newly created VR environment interactive using a visual programming tool. The above mentioned tools provide ways to facilitate the creation of VR experiences. However, there are some features that they are missing. Especially in terms of healthcare applications there could be the need for additional features such as, control over set stimuli and thus the experience, the ability to monitor the state of the end-user, or an easy creation process for virtual environments. These are also ideas that are part of our vision for a software authoring platform.

2.1.1 Content Creation

Many tools address the 3D asset creation process aimed specifically at novice users. For instance, 360proto (Nebeling and Madier, 2019) and Lift-off (Jackson and Keefe, 2016) are tools that allows users to create minimal AR/VR prototypes and 3D models just by drawing the intended diagram or a skeleton of the envisioned 3D asset on paper and directly importing it into the virtual environment, allowing further modification possibilities. On the commercial side, Google Tiltbrush⁵ and Blocks^6,7, a VR drawing tool and virtual modeling tool respectively, are popular among artists and VR enthusiasts. Though it was released for hobbyists, they are now being used in both healthcare and education fields as well (Ying-Chun and Chwen-Liang, 2018; So and Lu, 2019). Developments from the field of 3D asset creation can be used to enable people with no programming skills to create or adapt their own virtual environments.

2.1.2 User Measurements and User Assistance

Making use of physiological measurements in immersive technology alone is a vast area of study. Especially for therapy and training applications there are many possibilities. For a detailed elaboration of these possibilities refer to the literature review of Halbig and Latoschik (2021). To our knowledge, user assistance is addressed in the surveyed authoring platforms and related tools only employing conventional methods, such as tooltips, web-based tutorials, community forums, etc. An intelligent system, that provides contextual assistance would make the user experience easy, and flatten the learning curve.

The major challenge we observed from the literature is that most of the high-fidelity authoring tools, for example, Unity3D, Unreal Engine, or even A-frame are targeted at experienced developers and designers. The other tools such as CoSpaces, which is relatively easy to use, restrict the users from making the most out of VR technology due to its limited designing capacities and experience control features. This obstructs most of the consumers from creating their content for specific purposes and relies on very limited choices of VR experiences developed by others (Conway et al., 2000; Nebeling and Speicher, 2018; Ashtari et al., 2020). From a healthcare perspective, the authoring tools should additionally provide facilities to help the user fulfil a task completely, such as the provisions for monitoring an event or controlling the user’s experience in real-time. Most of the existing tools we came across are lacking those components. We have therefore come to the conclusion that the development of a specific software platform is necessary: A software platform that aims at facilitating the process that healthcare professionals can create and operate VR applications on their own.

2.2 User Acceptance Studies

Understanding a user and their perception of technology plays an important role in the user-centered software development process. A user acceptance study is a critical milestone to find out if users are convinced by the implementations, to see if their expectations are met, and to make it clear to developers and designers if the system is usable for the end user. It is generally carried out before the final release of the software system (Leung and Wong, 1997). This seems to be an ideal procedure in a user-centered approach where the design process happens for the user. However, there is a shift towards participatory design approaches where users are a part of the design and development process from the very beginning till the release of the platform (Sanders, 2002).

There is a vast literature that addresses technology acceptance studies. In the area of VR and healthcare, information from various acceptance studies are available. However, these studies tend to focus on specific applications that have already been completed (Snoswell and Snoswell, 2019; da Costa and de Carvalho, 2004; Garcia-Palacios et al., 2007) or VR-acceptance overall (Syed-Abdul et al., 2019). Some studies also set in at an earlier stage and used participatory approaches. In a study by Karaosmanoglu et al. (2021) healthcare professionals provided feedback via interviews before the development of a VR exergame to enhance physical activities by people suffering from dementia. In a study by Hilton et al. (2011) professionals’ feedback was gathered via questionnaires and focus groups to inform the development of a system for stroke rehabilitation. Such studies are very valuable because they ensure that applications meet the needs of users. Besides, they can serve as a guideline for future research in the fields of interest and are a stepping stone for any researchers who further want to explore a similar area with a similar user base. However, their conclusions are to some point limited to specific applications and user groups. Besides, while health professionals are consulted for the development of these applications, the ultimate focus is on their use by patients. To our knowledge, acceptance studies that focus on the needs of professionals in the healthcare domain and their use of VR applications have not been published so far.

One of the most used theoretical frameworks to assess an individual’s acceptance of any technical applications or devices is the Technology Acceptance Model (TAM), widely categorized under the umbrella of Information Systems (Davis, 1989; Lee et al., 2003). It assesses how different features of a system affect the perception and attitude of a user toward using the actual system (Balog and Pribeanu, 2009). The original TAM proposed two major variables to understand user acceptance, namely, Perceived Usefulness and Perceived ease of Use. Perceived Usefulness is defined as to what level a person thinks using a system would improve his or her work performance. Perceived ease of Use, is defined as the degree to which a user thinks using a system is effortless (Davis, 1989). Over time, the TAM has gone through major revisions (Lee et al., 2003). Even the original authors had extended it to TAM2, adding more constructs to evaluate user acceptance (Venkatesh and Davis, 2000). Researchers have also combined other frameworks and added their questions and constructs considering the existing TAM as a base, to understand the users’ view better (Lee et al., 2019; Fussell and Truong, 2021).

2.3 Ethical Implications

Ethical aspects play an important role for the acceptance of software. Numerous articles have already investigated ethical aspects for VR use in general (e.g., Kuntze et al., 2002), in health-impaired individuals in general (e.g., Kellmeyer et al., 2019), or in specific health domains (e.g., Lewis and Griffin, 1997). The focus of the majority of papers has been on ethical aspects of virtual reality in psychology/psychotherapy (e.g., Yellowlees et al., 2012; Kellmeyer, 2018; Marloth et al., 2020) and in rehabilitation (Lewis and Griffin, 1997; Kellmeyer, 2018). The identified aspects of ethical relevance are manifold. In addition to aspects that concern the specifics of VR, differential ethical challenges arise depending on the health care setting (e.g., Lewis and Griffin, 1997) and patient groups (e.g., Kellmeyer et al., 2019). These papers are mainly theoretical. In contrast, empirical research on health professionals’ ethical evaluation of VR is scarce. A general assessment is desirable regarding ethically relevant principles, such as respect for autonomy or justice (e.g., Beauchamp and Childress, 2019), and how they are positively or negatively affected by VR from the perspectives of employees in health care areas. Positive and negative expectations need to be investigated for a wide range of health care areas, such as psychotherapy and palliative care. This will help to identify critical areas of VR usage, to get an impression of the awareness for ethical issues in the professional group and to assess the acceptance of the platform based on ethical criteria. This is particularly important since VR usage in healthcare is on the rise.

In the introduction, we described the target of understanding the prerequisites for healthcare professionals to autonomously operate and create VR applications. Literature on this question is missing to date. While there are regular acceptance studies for various soft- and hardware products, until now they have never focused on healthcare professionals’ views on VR. Therefore, we address an important gap. In order to proceed as systematically as possible, our study is based on two pillars: the Technology Acceptance Model (TAM) and the envisioned software platform. Thus, we have concrete categories available as well as novel ideas to be presented to potential users. In this way, people from the healthcare sector are encouraged to share their opinions and suggestions regarding our proposals.

3 Methods

3.1 Mockup Trailer

First, we had to solve the problem of measuring the acceptance toward a software platform that does not exist yet. Moreover, we had to assume that for some of the potential respondents, the topic of VR is little or not known at all. Therefore, the subject of our evaluation was a mockup trailer, which is supposed to bring our ideas closer to the potential users from the healthcare sector. It should give them an idea of what using and creating VR applications might look like in their everyday work. The use of a trailer and surveying at an early stage also have the advantage that the results are less dependent on details. Instead, at a more general level, the survey asks what the professional group’s needs are when using VR and how they would envision using it professionally.

First, the trailer shows examples of VR applications in the healthcare sector, i.e., a virtual knee rehabilitation exercise and an example for exposure therapy targeting pyrophobic patients. Moreover, the trailer depicts potential features of the envisioned software platform. The first one would be a supervisor monitor that allows the supervisor of a healthcare application to keep track of the current state of the patient and adapt the virtual environment accordingly. Next, the trailer shows how users of the software platform can create their own VR applications by choosing from standard templates of virtual environments. Those templates are then adapted by creating virtual objects and by altering the sequence of events and interactions. Prototypical user interfaces and example interactions are shown for these operations. Another idea that is depicted in the trailer is the possibility to scan people via the smartphone and thus create custom avatars of a patient as described in Wenninger et al. (2020). So with the fire-exposure therapy and the knee rehabilitation exercise we chose two fundamentally different areas of application, which can nevertheless be presented in a relatively short duration and in an easily understandable way. However, both concepts are transferable to other areas and are thus intended to stimulate further ideas. By showing possible features like creating an avatar, we wanted to show viewers possibilities of VR that they might not have thought of on their own. Combined with expertise in their respective fields, we hope that participants will be stimulated to think about how such opportunities might be applied in their field. Hence, the mockup trailer introduces different facets of VR healthcare applications and gives the viewer a quick and concise impression of it.

Such a mockup trailer has previously been used to give study participants an impression of a new technology (e.g., Haugstvedt and Krogstie, 2012). The biggest advantage of this method is that it provides a good overview that conveniently can be distributed to a large group of people, accordingly supporting gathering larger samples than individual presentations could. Figure 1 shows the screenshots of above mentioned features from the mockup trailer. The trailer is available in an English and a German version. The English version of this trailer can be viewed at https://www.youtube.com/watch?v=HWXp6ke93_w.

FIGURE 1

FIGURE 1. The screenshots from the mockup video used for the study that shows different features of the envisioned software platform. (A,B) Demonstration of virtual leg coordination therapy. (C,D) Demonstrations of Supervision monitor and how it is used to control a fire exposure therapy session. (E) Demonstrations of Behaviour editor used to configure assets. (F) VR environment editor. (G) Avatar scanning process using a mobile phone. (H) Representation of the assistance system.

3.2 Survey

For the actual study, we adopted the TAM that we already introduced in the related work section (Davis, 1989; Lee et al., 2003). From this model, we adapted categories and corresponding questions as the basis for our study. These three categories are the perceived usefulness of the platform, the perceived ease of use, and the user’s attitude toward the platform. We adopted Likert-type questions from the TAM model that capture these categories. Furthermore, we added additional open-ended questions to the categories in order to gather valuable feedback on the prerequisites for the acceptance or rejection of the software platform and VR in general. Moreover, we wanted to get specific feedback on the idea of a supervisor monitor, which allows to surveil and control a VR session. In an additional question, we wanted to find out what kind of support potential users are hoping for when creating their own VR application. Then, questions were asked about ethical aspects of VR use. One question was on overall positive expectations regarding the effect of VR on people and society. Ultimately, expectations and concerns regarding the use of VR in different health areas were inquired.

That means in summary, six questions were asked about the perceived usefulness, four questions were asked about the perceived ease of use, four questions were asked about the attitude toward using the envisioned software platform, two questions were asked about a supervisor monitor, one question was asked about assistance, and three questions were asked on ethical topics. The final survey also included a demographics section. An overview of the categories and the asked questions can be found in Table 1.

TABLE 1

TABLE 1. Questions that were asked in the online survey and their corresponding categories and types.

3.3 Execution

The survey was conducted using LimeSurvey. It is an online tool that allows users to create their own surveys from a wide variety of question types and edit them at will. Before the actual survey started, participants confirmed their voluntary participation and the use of their anonymized data for research purposes. With the help of LimeSurvey, we were also able to integrate the trailer directly into the survey. That means participants first watched the trailer and then answered the questions right after. The tool was hosted on a university server to protect the corresponding data. Closed-ended questions were answered on a 7-point Likert scale. Survey participants could skip questions or stop answering at any time if they wanted. Before the actual survey started, we asked the participants for permission to store their data anonymously. The language of the survey was German. The link to the survey was distributed to potential users via email, either directly or via mail distribution lists. Each invitation email was also accompanied by a request to forward the survey to colleagues. The link was also shared in Facebook groups to which only persons of the respective professional groups had access. In addition, a professional association’s website invited participation in the survey. In this way, we ensured that we surveyed a broad user group. At the same time, we were able to comply with pandemic containment regulations, which is of course a priority when working together with people from the healthcare sector.

3.4 Data Analysis

The analysis of the quantitative data was done using SPSS. For the qualitative analysis, we transferred all the answers to MAXQDA⁸, which is a software that helps with transcribing, ordering, and analyzing qualitative data. In this software, an entry for each subject was created which included all their respective statements. Subsequently, we developed a code system according to a methodology following Mayring (2015). In an iterative process, we analyzed the statements and grouped them by content. This created categories, which we also grouped again until a hierarchical system of categories and statements emerged. The coding was checked a second time after some weeks to uncover inconclusive categorizations. With this procedure, qualitative data could be converted into quantitative data. The frequency with which certain topics were mentioned can serve as one indicator of their importance.

4 Results

In this paragraph, we will first explain the demographic characteristics of the study participants, especially in terms of their professional roles in healthcare and their prior experience with VR. We then report the results on the three categories of the TAM; perceived usefulness, perceived ease of use, and attitude toward the envisioned software platform. Here we first explain the answers to each of the closed-ended questions and then the answers to the open-ended questions. We then explain the results for the questions on the supervisor monitor and the required assistance. We conclude with the results of the ethics-related questions. For the quantitative questions, means and the t- and p-values for one-sample t-tests are reported that indicate whether the means deviate significantly from the mean of the scale. Thus, we appraise whether the evaluations are above or below the average. For the answers to the open questions, we indicate in parentheses how often the respective statement was made and what proportion of all answers to the respective question that makes.

4.1 Demographics

A total of 102 participants took part in the survey. In order to ensure the voluntary nature of the survey, it was possible not to answer each question. This resulted in different numbers of cases for the various questions. The average age of the participants was 42.4 years and varied between 24 and 73. Of the participants, 58 were female, 42 were male, and 2 did not provide this information. Just under half of the participants in the overall sample were working in psychotherapy, 24 were working in physical therapy, and 11 were working in training and teaching. The rest of the participants were distributed among research, development, consulting, prevention, and other discrete medical fields. Another important piece of information to us was the extent to which participants were using technology to support their daily activities. Forty participants had prior experience with VR. Of these forty participants, 16 had experience with VR applications that were related to medical applications such as physical therapy, exposure therapy, various simulators, or anatomy studies. The rest of the participants that had prior VR experience had seen demos during exhibitions, worked on research projects, or just experienced common entertainment applications. Merely eight of the participants use VR frequently and only three of them for their work. In addition, 58 participants regularly use computers to support their work. Due to the voluntary nature of the study and also because the ethical aspects were addressed at the end of the survey, up to 54 valid answers were provided for the ethical questions.

4.2 Perceived Usefulness

The envisioned software platform is intended to serve as a tool that people can use to create VR environments without major hurdles. From the demographic data we collected, we found that only three of the participants use VR for their work. For most of the other participants, VR is new territory. Especially here it is important to see if the potential users even consider the concept of the envisioned software platform as useful for their own work. For this exact purpose, the TAM offers questions about the perceived usefulness of a product. Thus, participants were asked whether the envisioned software platform could improve their work effectiveness, enhance their work performance, increase their work productivity, and whether it could prove useful for their work.

An exact overview of the distribution of responses in this category can be found in Table 2. On a Likert scale of 1−7, the mean score in this category for 64 participants was 4.4, indicating that they regard the envisioned software platform as a potentially useful tool for their work (t (63) = 2.00, p = 0.050).

TABLE 2

TABLE 2. Summary of the results regarding the perceived usefulness of the envisioned software platform.

4.2.1 Use Cases of VR in Healthcare

Since there is a wide range of potential use cases of VR in healthcare, we first wanted to know from the participants what specific use cases they foresee in the context of their areas of work.

Exposure therapies for anxiety and trauma were mentioned most frequently (n = 39 (43.3%)). Scenarios that are not easily representable in real life were emphasized, e.g., heights, crowds, job interviews, or animals.

Another mentioned therapy area was the treatment of eating and body image disorders (n = 4 (3.6%)). For example, virtual therapy applications could reinforce an examination of one’s own body image by having a subject embody avatars with different characteristics. In addition, coping with potentially stressful situations could be simulated and practiced, such as grocery shopping or observing one’s own body in the mirror.

Survey participants also saw the potential to use VR to foster teaching scenarios (n = 9 (10%)). For example, the teaching of cardiac catheter interventions or the training of future physiotherapists and physicians was mentioned. Simulation of the anatomy via 3D models was mentioned to be helpful here. The training for the acquisition of various competencies was also mentioned quite often (n = 14 (15.6%)). Here, VR was suggested to be used to simulate critical interpersonal situations. There is potential here in the training of all personnel who have contact with customers or patients. In VR, these critical situations could be represented with the help of virtual agents. This simulation of social situations also has the potential to enable a change of perspective. Roleplays were mentioned that could, for example, be used in couple therapy to increase understanding of others. If people had the opportunity to take both positions in a conflict, this could also help to reduce discrimination.

Participants also emphasized the potential of VR for physiotherapy and rehabilitation (n = 8 (8.9%)). Physiotherapists also hope that VR could provide relief in everyday work. Patients could perform exercises from home or independently in the virtual world, thus avoiding 1-to-1 supervision. This idea could potentially counteract staff shortages.

4.2.2 Desired Functions

In the second open question in the category of perceived usefulness, we asked the participants to name features and functions that a VR authoring platform must bring along in order to have a benefit for their personal work.

Many of the potential users asked for individual customization in the creation and use of virtual applications (n = 18 (21.7%)). Participants described a desire to be able to customize the intensity of virtual stimuli, for example, by selecting specific fear stimuli in an exposure, changing the degree of realism of the graphics, or by adjusting the necessary repetitions in a physical rehabilitation exercise (n = 5 (6.0%)). Other specific requests included the ability to simulate different times of day, the ability for users to observe themselves in the virtual world (mirror), the ability to interrupt stimuli (by switching to a relaxation room), or an automatic collection and analysis of data. All these examples reflect the desire for customizability. A feature that was also requested more often was the embedding of avatars (n = 6 (7.2%)). End users should see their own virtual representations or the representations of other users in shared VR experiences. This would be an important factor in simulating and practicing various social situations, e.g., a dispute between two sides.

Participants emphasized that they would like to be provided with certain standard scenarios and templates in which one only has to add minor components or change a few settings (n = 19 (22.9%)). Such standard scenarios could exist for common types of disorders, for example, the most common anxiety scenarios. Examples that were mentioned are train rides, narrow spaces, elevator rides and crowded or wide places. In addition to the standard scenarios already mentioned, some users here suggested a modular system, i.e., a kind of building block system that allows individual adaptation of the applications to the clients.

Some of the participants also demanded assurance of a certain basic quality and support when it comes to medical applications (n = 7 (8.4%)). The envisioned software platform should be aligned with medical guidelines and even offer warnings or corrections in case someone wants to create an incorrect or potentially dangerous application. In addition, the logging of progress over several sessions was mentioned. In general, certain technical requirements were named, which should be fulfilled (n = 13 (15.5%)). This includes a low error susceptibility, stable performance, the possibility to do remote sessions, and compliance with data protection guidelines.

4.3 Perceived Ease of Use

Next, we wanted to find out if potential users can imagine that the envisioned software platform could be useable for them. Therefore, we asked two questions from the TAM that focus on the perceived ease of use of a product. Here we asked how easy the potential users thought it was to use the envisioned software platform and how easy they imagined the learning process of the platform.

An overview of the responses to this category can be found in Table 3. On a Likert scale of 1-7, the mean of responses for all questions about the perceived ease of use was 5.30 (t (68) = 9.86, p < 0.001). When asked if potential users would find it easy to learn how to use the envisioned software platform, 54 of 69 people responded with a 5 (somewhat agree) or higher. This indicates that users tend to be confident that they can master the use of a software platform that allows them to create their own healthcare applications.

TABLE 3

TABLE 3. Summary of the results regarding the perceived ease of use of the envisioned software platform.

4.3.1 Support for Learning

With the first open question in the ease of use category, we wanted to find out what features the envisioned software platform must have to easily learn its usage.

As before, one of the most central points that very often resonates here is that the learning process should not be too laborious (n = 26 (13.3%)). Many participants demanded a certain simplicity from the platform (n = 30 (15.3%)). For example, there should be only a few setting variations or not too many preliminary settings should be necessary (n = 10 (5.1%)), there should not be too much necessary equipment and in general, the learning process should be simple and short. This simplicity was also demanded on a technical level (n = 15 (7,7%)). Here the point was mentioned that the platform should work on laptops with common operating systems to avoid the extra purchase of hardware. One suggestion for making it easier to incorporate VR in working routines was to provide templates, or demo versions (n = 19 (9.7%). Many of the potential users would like to have ready-made scenarios where they only have to modify a few points. Such scenarios could be available for typical use cases and could also serve as a basis for the learning process. This idea was also mentioned in the desired functions in paragraph 4.2.2.

Another point that was raised repeatedly is intuitive usage (n = 20 (10.2%)). The potential users do not want an intensive learning process, but an interaction that builds on their existing knowledge. A further specific aspect that was mentioned here is the language (n = 16 (8.2%)). Many of the potential users explicitly pointed out that the user interface of the software platform should work with their native language. At the same time, the use of technical terms should be avoided.

Furthermore, ideas were put forward as to how learning to use VR could look in concrete terms. Some of the participants demand supporting materials (n = 19 (9.7%)) like tutorial videos, a manual with extensive descriptions, or demo examples. Others wish for support through experts (n = 17 (8.7%)), e.g. support via email and phone or through online/offline training. Seven people suggested that there could be support among the different users of the envisioned software platform, for example over community chats or forums.

4.3.2 Obstacles to Learning

Complementary to the previous question, we next wanted to know which aspects might rather prevent potential users from learning how to integrate VR in their working routines. However, the results here were very similar to those of the last question. A majority of the potential users emphasized that too much effort in learning and using the platform is a criterion for exclusion. The main warnings here were that it could be too time-consuming and too complicated to use.

In addition, some of the potential users expressed that not meeting prerequisites could be an obstacle in learning to create and operate their own VR applications. On the one hand, these prerequisites may be on the side of the users. They fear that they do not have the necessary technical knowledge and equipment to guarantee effective use of VR (n = 11). On the other hand, these preconditions can also be on the software side. Frequent error messages, long loading times, hanging graphics, or poor usability (n = 15) would be reasons not to learn how to use the envisioned software platform.

4.4 Attitude Toward Using

Next we wanted to know whether the respondents would actually use the platform to support their work - in essence, we wanted to find out about the attitudes toward using the platform. To understand this, two questions were asked. With one, we wanted to know if the participants thought that introducing the envisioned software platform into their daily work routine was a good idea, and with the other, if they were positive about using the software platform. These questions were also adopted from the TAM.

On a Likert scale of 1-7, the mean of all 65 responses was 5.66 (t (64) = 11.45, p < 0.001)–indicating that participants agreed that the envisioned software platform can actually help them in their job. The results are depicted in Table 4.

TABLE 4

TABLE 4. Summary of the results regarding the attitude toward using the envisioned software platform.

4.4.1 Reasons for the Use

In order to better understand how basic attitudes toward the envisioned software platform arose, we asked the participants what reasons they see for using the platform.

Many of the participants expect an advantage for their clients from the use of the platform. It was frequently stated that the virtual applications could increase the clients’ motivation (n = 24 (22.0%)). Especially the variety and novelty of the virtual therapy and exercise systems could contribute to this motivation boost. At the same time, potential users also believe that the use of virtual applications can increase the fun factor for clients. In particular, gamification of content can help to increase the fun and at the same time create new incentives.

Many of the participants also simply hope for practical benefits in implementing their treatments. Here, the creation of conditions was frequently mentioned that would be impossible or very difficult to realize in the real world (n = 6 (5.5%)), e.g., the realization of a (virtual) flight in the treatment of aviophobia. In addition, many of the potential users hope that it will be easier to create new treatment options that are adapted to the clients and thus simplify current processes (n = 8 (7.3%)). This is also hoped to increase the effectiveness (n = 5 (4.6%)) and flexibility (n = 10 (9.2%)) of treatments. Another opportunity to increase one’s own treatment options with the help of the envisioned software platform is seen by many of the participants in the fact that the virtual applications could function regardless of location (n = 22 (20.2%)). Here, the idea that clients could run the applications themselves in their own homes was frequently described. In this way, the workload for therapists and trainers could be reduced and inflexible or limited clients could be more easily involved. Especially in times of a global pandemic, this is an interesting aspect for some of the respondents.

4.4.2 Reasons against the Use

Of course, it is also important for us to understand why some participants would rather refrain from using VR. Therefore, we asked the potential users what reasons they have against using the envisioned software platform.

For some participants, the use of virtual applications means too much mechanization of their work (n = 13 (10.1%)). The importance of direct interaction and the associated relationship with clients was often emphasized. Psychotherapists in particular stated that direct interaction and direct face-to-face conversations are of great importance for their work. Therefore, they view the virtual component rather critically. Another disadvantage that many psychotherapists mentioned is the danger of a blurring between reality and VR or an escape from reality (n = 6 (4.7%)). Clients would flee into a virtual world and thus have even more problems dealing with reality. In the treatment of depression, for example, it is more important for clients to go outside and establish activities there. Escaping into a virtual world could be more of a hindrance to this.

Another aspect that was also frequently mentioned here is the fear of too much extra effort that would come with using VR. The participants fear that the possibilities would be too diverse and thus the learning process and the creation of new applications would be too time-consuming (n = 41 (31.8%)). This fear was also expressed with regard to the technical equipment. Procuring and setting up the hardware could be too costly, which would make the envisioned software platform unattractive for certain users (n = 9 (7.0%)). Also, financial concerns were expressed with regard to hardware procurement (n = 12 (9.3%)).

Some participants expressed that they simply do not work with the right target audience that would be eligible to use a virtual application (n = 8 (6.2%)). The use of VR technology for severely affected patients or patients in intensive care units is not conceivable for the respective participants. Potential users who work with elderly people, for example, also fear a lack of acceptance of VR among their clients (n = 4 (3.1%)).

Some participants expressed the fear that virtual therapies might be too ineffective, e.g., because clients get used to them too quickly (n = 8 (6.2%)). Furthermore, some even fear negative consequences for the clients when using virtual applications (n = 8 (6.2%)). This could lead to an aversion to VR if the technology is used improperly by laypersons. Hazards from cybersickness, dizziness, improper use, or falls were mentioned. In general, there were still some concerns about an unsuitable design of the software platform (n = 13 (10.1%)). These included that the platform could be too inflexible, that it could communicate in an incomprehensible language, or that too many pre-settings were necessary.

The results in concerning the three surveyed categories of the TAM can be found in Figure 2 and Table 5. Overall, the TAM shows a mean score of 5.19 on a scale of 7 which is significantly above the mean (t (66) = 7.14, p < 0.001).

FIGURE 2

FIGURE 2. Summary of the results regarding the three surveyed categories of the Technology Acceptance Model (TAM).

TABLE 5

TABLE 5. Summary of the results regarding the TAM and its surveyed dimensions.

4.5 Supervisor Monitor

The planned implementation of a supervisor monitor was also part of our acceptance study. For this purpose, the participants once again saw a short clip of the trailer showing a prototype of the monitor.

4.5.1 Components of the Supervisor Monitor

Subsequently, we wanted to find out which components this monitor must have from the perspective of potential users.

Many of the participants’ suggestions related to the recording and display of the client’s condition. For example, the desire to display the client’s view through the head-mounted display in the monitor was expressed in some cases (n = 11 (15.7%)). In addition, physiological parameters were mentioned as important information (n = 16 (22.9%)). First and foremost, the display of the heart rate was mentioned, but also the display of the skin conductivity, the movement parameters, and the pupil reactions were desired. The recording of the users does not always have to refer to the current situation. Participants also asked for a display of progress over several applications (n = 3 (4.3%)). For example, for a movement exercise in the physiological domain, the monitor could display the intensity and number of repetitions of past sessions. For anxiety therapy, the intensity of the anxiety stimulus achieved could be documented.

In addition to the display of the patient’s condition, many of the potential users would also like to be able to intervene in the current application via the supervisor monitor (n = 13 (18.6%)). This includes, above all, the adjustment of the set stimulus in order to control the intensity of the current application or the activation of assistance. Often, even the possibility to completely switch off the current situation was desired, i.e., to get clients out of the situation immediately if necessary. Some suggested, for example, switching to a relaxation environment.

Other ideas for the supervisor monitor included displaying a map of the virtual environment and the possibility of operating the monitor via a tablet or smartphone. In general, a clear layout with simple operations was desired (n = 10 (14.3%)).

4.5.2 Capturing the User State

To be able to monitor a VR application really effectively, a supervisor needs information about the state of the client immersed in the virtual environment. Therefore, we wanted to know from the participants what tools they usually use to capture the state of their clients. This provides us with a basis for discussing which of these tools could also find their way into VR healthcare applications.

Methods in written form were mentioned most frequently. Primarily questionnaires (n = 20 (19.0%)) and the writing of protocols (n = 15 (14.3%)) were named. In addition, standardized rating scales were mentioned (n = 8 (7.6&)). Here, also specific examples were mentioned sporadically, like the Borg scale Borg (1998) for recording subjective exhaustion or the SUD score Hartanto et al. (2012) for recording subjective anxiety. Probably the simplest form of assessing the user’s condition is observation. Many of the potential users report that direct observation of body language, behavior, facial expressions and gestures is an important indicator of the state of the current treatment (n = 22 (21%)). Especially on the side of the psychotherapists, the conversation or interviews are often used to find out about the course and the effect of the treatment (n = 8 (7.6%)).

Physiological parameters were also mentioned as a method for recording the user’s condition (n = 19 (18.1%)). Among the mentions were heart rate, motion data, skin conductance, eye tracking, or blood pressure.

4.6 Assistance System

If people from the healthcare sector should really be able to integrate VR into their everyday work, then they need to be supported as well as possible. To better understand how assistance should look like we asked potential users what kind of support they would want when creating a virtual environment.

Users want continuous information about the next steps to take (n = 8 (36,4%)). There should therefore be as linear a process as possible when creating VR applications, with the next step always being clearly indicated. This support was requested across the whole process, e.g., support in designing the virtual environment, or help in finding suitable applications and templates.

Again, some of the participants took the opportunity to express that they would like templates provided to them (n = 7 (31.9%)). Frequently used standard applications should be identified and made available as a basis for further adjustments. Other features that were requested are the explanation of the different functions, the highlighting of often-used applications, or the linking to further help.

While few actionable responses were provided here, the answers to previous questions also offer many insights into what type of support is necessary when people want to create their own VR applications. More on this is outlined in the discussion.

4.7 Ethics

The questions on ethical issues covered two topics: an evaluation of how several ethical principles are affected as well as an assessment of benefits and concerns expected for different health areas. Due to the novelty of the research, a principle-based approach provides an excellent starting point for an ethical assessment. This approach relates to a range of ethical principles that were previously defined as being important with regard to innovative health technologies (Walker and Morrissey, 2014). Principles considered relevant were quality of life as the key outcome of all health care activities (Musschenga, 1997), respect for autonomy and justice drawn from the four principles of biomedical ethics (Beauchamp and Childress, 2019), safety/security, privacy, participation, and a positive self-conception delineated in a model for the ethical evaluation of age-appropriate assistive systems, ambient-assisted living (AAL), and other health-related technologies (Manzeschke et al., 2015; Nelles et al., 2016), as well as transparency (e.g., Turilli and Floridi, 2009). The term support substituted the principle of beneficence (Beauchamp and Childress, 2019) because it was considered more concrete and tangible. Due to the wide range of areas in which VR is used, it was considered important to compare benefits and concerns for VR use in different health care areas. Examining professionals’ views of benefits and concerns reveals areas where VR use requires particular caution, for example for vulnerable persons with psychiatric symptoms like psychosis (Rizzo et al., 2003) or adolescents and children (Kaimara et al., 2021). For all questions, 7-point scales were utilized. The endpoints for the question on the agreement to how VR would support the selected ethical principles were 1-totally disagree, 7-totally agree. For the questions on expectations, the endpoints were 1-very low expectations/morally very low concerns; 7-very high expectations/morally very high concerns. T- and p-values are reported to indicate whether the single means deviate significantly from the mean of the scale to appraise whether the evaluation is above or below the average.

4.7.1 Effects on Different Ethical Principles

Participants agreed most that VR would provide support for its users (M = 5.44, t (51) = 11.09, p < 0.001), which indicates that they overall expect benefits for their patients. Safety/Security was the second highest rated dimension (M = 4.94, t (50) = 5.22, p < 0.001), which shows that health professionals were hardly worried that VR would do any harm. The evaluations of respect for autonomy, quality of life and positive effects on the self-conception were also quite positive (all p < 0.002). Privacy and participation reached values tightly and not significantly above the mean of the scale (both p > 0.102). The evaluations for transparency (M = 3.94) and justice (M = 3.88) were slightly, but not significantly beyond the scale’s mean (both p > 0.564), however indicating that there might be some concerns regarding the effect on these dimensions. All means are shown in Figure 3 and in Table 6.

FIGURE 3

FIGURE 3. Summary of the arithmetic means for the question how ethical principles are positively affected by VR in healthcare.

TABLE 6

TABLE 6. Results for the questions on how the selected ethical principles are positively affected by VR.

4.7.2 Benefits and Concerns for the Use in Different Health Care Areas

Expectations were for all health areas significantly above the mean of the scale (p < 0.001) except for palliative care. Expectations were the highest for rehabilitation (M = 5.96), education/academic studies/training/continuing education (M = 5.77) and physical therapy (M = 5.56). For these areas, concerns were also the lowest, resulting in an also highest positive balance between expectations and concerns. Expectations for all other areas except palliative care ranged between 4.83 and 5.13 and were accordingly quite positive. Expectations were considerably low for the area of palliative care (M = 3.31), where they were significantly below the mean (t (47) = −3.00, p = 0.004), and here, concerns were also highest, resulting in a negative balance (M = −1.09). Concerns were second highest for psychiatry/psychotherapy, but did not differ significantly from the mean (M = 3.76, t (48) = −0.96, p = 0.342). Here the positive balance between expectations and concerns was the second lowest (1.38). Concerns were third highest, but significantly below the mean, for pediatrics (M = 3.46, t (47) = −2.10, p = 0.041). For all other areas, concerns were also significantly below the mean of the scale (all p < 0.005) indicating low concerns. The means are depicted in Figure 4 and in Table 7.

FIGURE 4

FIGURE 4. Summary of the arithmetic means regarding the questions on positive expectations and ethical concerns. Education/Academic studies includes training/continuing education.

TABLE 7

TABLE 7. Distribution of results in terms of ethical expectations and concerns with regard to various healthcare domains and VR.

5 Discussion

Overall, the three categories surveyed by the TAM showed a mean of 5.19. Since the TAM measures how likely a system is to actually be used, we can conclude, that our envisioned software platform and its features actually have the potential to be established in healthcare practice. That means a software platform that allows people to create and customize their own VR content could promote the actual use of VR in this domain. Many of the responses we received also gave us an idea of what such a platform should look like. But not only that, many of the answers can be generalized to show what should overall be considered in VR applications that are intended to be suitable for healthcare practice. We will discuss both facets in this section.

5.1 Insights About This Study

When we asked the participants about the potential use of VR in their field, the answers were fairly broad (see Section 4.2.1). This is very important for the basic motivation behind the envisioned software platform. If experts from different fields can imagine a concrete use of VR for their work, then this speaks for the potential of a broad-based software platform that can support many of them. It should also be considered that many of the study participants had no previous contact with VR. The fact that we received a total of 88 mentions about possible use cases for VR shows that the mockup trailer was successful in getting our ideas across.

In terms of the perceived ease of use, the figures also show that potential users tend to feel able to operate the envisioned software platform. This also shows a positive basic attitude of the participants toward the platform. However, questions and statements relating to the user-friendliness and operation of the system should always be treated with caution in this case. After all, the basis of this evaluation was a video. The actual usability of a user interface can only be determined on the basis of actual prototypes and in controlled test environments. Only then can we really make statements about what intuitive use of the platform could look like.

5.2 Customization and Simplicity

It is particularly interesting that there is a certain trade-off in desired functions. The people demand a certain amount of customization, as the multitude of different clients put different requirements to the VR applications. At the same time, they want the envisioned software platform to be very simple and straightforward without too much effort. In general, healthcare applications in VR have to manage this balancing act. Therefore, when implementing the envisioned software platform, we should ensure that standard applications and templates are available to users. We should thus prevent users from being forced to build their own applications from scratch and thus being overwhelmed with a multitude of options. At the same time, these standard applications must be modifiable in the right places so that they cover as many use cases as possible. So when implementing VR healthcare applications, one should always have an overview of what the most commonly used default scenarios and settings are. These must then be adaptable in the simplest possible way.

For exposure therapy, this could mean that common fear stimuli are available (animals, height, confinement, presentation in front of an audience), but that their peculiarities can be adapted. For example, with a virtual audience for virtual speech training, one should be able to modify the number of virtual agents, their degree of realism, and their behavior. At the same time, there could also be standard environments in which such an exposition can take place, e.g., in a large hall (for more stress) or on a beach (for more relaxation).

For physical rehabilitation, this could mean that one can choose between standard muscle groups that should be trained, e.g., the arm/shoulder area or the leg/knee area. Standard exercises could comprise the continuous lifting, holding up, or rotation of those extremities, while the number of repetitions or the time of the exercise could be adjustable and visible for the immersed person. Adaptability can also come into its own when it comes to embodiment. As mentioned before, it is possible to create custom avatars of people quite comfortably with just a smartphone Wenninger et al. (2020). However, one could also provide standard avatars for those who find it too time-consuming to carry out a scan process. This could also be useful in some applications, as already the sense of embodiment toward a regular hand can be sufficient to alter emotional responses to virtual stimuli (Gall et al., 2021). It would also be conceivable here to exploit the Proteus Effect, for example by providing athletic avatars when physical exercise is required.

5.3 Learn to Create and Operate VR Applications

The most important thing when it comes to the learning process of autonomously creating and operating VR applications is simplicity. For the envisioned software platform this means that we should refrain from overwhelming users with too many options. A large proportion of the participants in this survey had little to no prior experience with VR. This also explains the wish for close support. The potential users, therefore, attach less importance to having a large number of functions and options available. Rather, they want a relatively straightforward and simple path to their own virtual application. Again, this is generally true for healthcare applications in VR. One should not try to cover every extraordinary use case, but rather make standard use cases available quickly and easily. If it takes too long to learn how to use the VR application and to prepare it, it makes it more or less useless for practical use in the healthcare sector. Their already stressful working day does not allow for too much additional effort. This does not only apply to the software itself, but also to the equipment or hardware. Again, the fact that many of the participants had less prior technical experience comes into play here. On the one hand, this means that the requirements for operating VR applications should not be too high. On the other hand, one should make suitable hardware recommendations or minimum requirements for the applications. In this way, one can save potential users from having to search for their own hardware solutions. This can apply to both computers and head-mounted displays.

The desire for appropriate language is also relevant in this context. The focus should be on the use of the native language. Technical terms could additionally be provided with their own explanations via mouseover effect or in separate documentation. To be absolutely sure, one should test a VR application beforehand with potential users in the healthcare sector to find out whether the descriptions are comprehensible and appropriate. If one wants to ensure an intuitive and smooth user interaction, this step is inevitable anyway.

As far as the method of learning is concerned, there should definitely be documentation on the envisioned software platform. However, to further ease the burden on potential users, the suggestion of interactive tutorials is quite interesting. If the envisioned software platform provides the standard applications already mentioned, then these could also be used relatively easily as the basis for a tutorial. For each demo application provided, there could be a tutorial that shows where one has to change which parameters and what you achieve with them. This idea is also transferable to all VR applications that are to be used autonomously by people from the healthcare sector.

5.4 Support for the Supervisor

If healthcare professionals are to be truly capable of creating and operating VR applications on their own, they need support in different dimensions. Based on the answers to the survey, there are a few points that can be addressed. Based on our ideas for a software platform, an assistance system seems to be a really interesting solution. This system could offer personalized assistance, based on the experience of the user, the environment that needs to be created, or the type of treatment that is aimed for. The exchange of ideas with other users and communication with experts can also be supported by the assistance system. Ideas and approaches of other users on a certain topic could be shared and then, if relevant, suggested by the assistance system. The templates created by other users can also be shared if they have been published for reuse. The templates could be categorized via a tagging system and thus be found and tested more quickly by others. These tags could refer to diseases, forms of treatment, or medical specialties. This would also greatly facilitate an active search for relevant templates by potential users and optimize possible recommendations by the system. Thus, we could also fulfill the collaborative character that some of the participants have been asking for.

The desire of the potential users that some of the applications should also function without direct supervision, i.e. be carried out by the clients themselves, is quite interesting. In some areas, this could be an interesting approach to further relieve staff in the healthcare sector. People who want to make their VR applications more attractive might keep this option in mind. Suggestions in this direction came mainly from participants in the field of physiotherapy. This is probably less relevant for the treatment of mental disorders, as more intensive care is sometimes necessary here. For our envisioned software platform this means that it should allow users to decide which form of application they want to create. Either a form of application that is controlled by a supervisor or the form in which the clients themselves are responsible for the progress of the application. Automated logs could then still inform caregivers and clients of the achieved progress. In physiotherapy, this could result in some kind of scoreboard. Repetitions of lifting exercises or endurance exercises, e.g., of a treadmill or ergometer training, could serve as evidence of continuous training. Thus, the supervisor would have a convenient overview and clients could draw additional motivation from the introduction of gamification elements.

To make a VR application as attractive as possible for usage in healthcare, it should support the minimization of potential negative consequences and thus also take weight off the shoulders of a supervisor. This includes the definition of an exact play area to prevent falls. To prevent the symptoms of cybersickness, applications should give warnings when the frame rate is too low or the latency is too high. The dynamic restriction of the field of view (Groth et al., 2021; Teixeira and Palmisano, 2021) or the adaption of navigation velocity and acceleration (Plouzeau et al., 2018; Chardonnet et al., 2021) are possible automatic methods that can reduce the hazards of cybersickness. In this way, the supervisor could be further relieved.

The specification or recommendation of certain hardware can also help to ensure minimum quality. Another starting point is the collection of physiological data. It can help to detect cybersickness and general discomfort (Cebeci et al., 2019; Islam et al., 2021). Here, too, the automatic recording, analysis, and visualization of the user’s condition can help to relieve the supervisor. More on this when we talk about the supervisor monitor in the next subsection.

5.5 Supervisor Monitor

The idea of implementing a supervisor monitor also seems to be a suitable measure to give people from the healthcare sector the necessary support and security to integrate VR into their work. Frequently requested was information about the current view of the VR users and information about their physiological state. Wearable sensors are particularly interesting here, as they have a very short setup time. Parameters like heart rate and skin conductivity can be conveniently captured with wristbands such as the Empatica E4⁹, that has been used before in VR studies (Šalkevicius et al., 2019). Gaze and pupil behavior information could also be captured relatively conveniently using head-mounted displays. Eye trackers from tobii¹⁰ or Pupil Labs¹¹ are commonly used to augment VR headsets. Different characteristics of experience can thus be classified automatically, e.g. stress (Ham et al., 2017; Robitaille and McGuffin, 2019), anxiety (Šalkevicius et al., 2019; Bălan et al., 2020), or cognitive workload (Currie et al., 2019). Corresponding classification results can then either be used to directly control the virtual environment, can be visualized for the supervisor, or be used to determine the progress over multiple sessions. The various options that exist here are discussed in Halbig and Latoschik (2021). In any case, the capture, analysis, and visualization of physiological data can help to give the supervisor more confidence in dealing with VR. In practice, of course, one has to weigh how much and to what extent this information is displayed at once. Dangers and unusual physiological signals must be presented in such a way that they can be perceived immediately, e.g. via acoustic signals and a salient layout. So, if possible, not all details should be shown to the supervisor, such as raw physiological or movement data. Rather, the supervisor should be shown the interpretation of the data (e.g., high or low stress) by default. Raw data could be displayed on request. The data and its interpretation could also be part of automated logs of the specific sessions.

Another interesting approach is the ability to intervene in what is happening in the virtual world via the supervisor monitor. Especially the attenuation or amplification of stimuli plays an important role in applications that are supposed to trigger fear or stress. Feedback from potential users also speaks in favor of a kind of emergency off switch on the display that allows the situation to be completely resolved. Here, more control and security can be brought to the supervisor.

The possibility to take notes and save them for the respective client and session could provide basic support for interviews and observations. Such a digital notepad could then also be helpful when documenting progress across multiple sessions. In addition, one has all the information about the stimuli that were set in the virtual environment. Automated logs can then be used to store information about the stimuli and the client’s behavior. In this way, it should be possible for therapists and trainers to analyze the progress of the treatments.

5.6 Ethics

Overall, the results demonstrate that professionals expect positive outcomes for their patients with regard to ethical principles and for most health areas. Nevertheless, negative outcomes are mostly perceived for transparency and justice. Doubts with regard to transparency refer to the possibility that users might unconsciously be manipulated by VR (Kool, 2016; Spiegel, 2018) or that wrong information about VR effectiveness is given (Madary and Metzinger, 2016). With regard to justice, only privileged groups could benefit from VR (Madary and Metzinger, 2016). These issues were previously mentioned in the literature. Future research needs to address the reasons for these concerns and derive conclusions to avoid these negative effects. Besides, psychiatry/psychotherapy and palliative care are the areas with the highest ethical concerns. These disciplines have previously been identified as ethically critical (e.g., Rizzo et al., 2003; Weijers and DiSilvestro, 2017), and the concerns should be addressed adequately. VR has already been proven beneficial in these areas, at least in certain contexts (e.g., Dellazizzo et al., 2020; Johnson et al., 2020). Accordingly, it is important to reflect on these concerns and to carefully consider potential applications, but also to resolve concerns by informing about proven positive effects and positive perceptions by users.

5.7 Limitations

Although this study provides many useful information, it also has limitations. Probably the most severe is that the findings of this study are based on a video and not on actual interaction with a prototype. It is therefore very difficult to make statements about what a functioning interaction with our envisioned software platform would look like. Nevertheless, it is important to collect the opinion of potential users at the very beginning. In addition, the video was also a suitable means to convey ideas of VR and its connection with healthcare to a wide range of people.

Another point to mention here is the fact that this is an online study. Interviews would have made it possible to get more in-depth information. In addition, ambiguities could have been explained. However, the decision to conduct the study online also had its advantages. First, it allowed us to reach more different people as we were able to send invitations around easily. These could then spread further and reach many experts to whom we would not have had access at all. Second, this also enabled us to comply with the pandemic restrictions and get in contact with healthcare personnel at the same time.

5.8 Prospects

In the course of the discussion, we elucidated how the knowledge gained in the present study can be used to develop VR applications that respect more of the requirements of healthcare practice. However, there is still a lot of unfinished work here, especially considering the concrete implementation of the discussed ideas. While it became clear that some kind of supervisor monitor should be helpful to spur the integration of VR into healthcare practice, it remains unclear how exactly this should look. It needs to be clarified how this monitor represents the current state of the immersed person, may it be the location in 3D space or the physiological status. This is accompanied by the question of how a supervisor would alter the sequence of events and how possible dangers are communicated perceptibly and understandably via the monitor.

Further research potential lies in the question of how exactly the support looks like that is received by a person that creates a VR environment. An AI could be designed that understandably supports laymen to create their own virtual applications. The exact solution of the trade-off between adaptability of such applications and low effort in their creation process is another unknown. While we have already shown possible approaches, the details of which features should be customizable and which not remain unclear. Just as with all the other open points, progress will only be made here with concrete user studies. Only a series of iterative tests with concrete interfaces and actual users can provide accurate information on what the implementation of the discussed features would look like. User studies with healthcare professionals and clients using demonstrator applications are planned as a next step.

6 Conclusion

The results of the three surveyed areas of the TAM show that the trailer left a positive impression on potential users. Overall, the TAM shows a mean score of 5.19 on a scale of 7, indicating that the majority of participants agreed with the concept that was depicted by the mockup trailer and that the envisioned software platform has the potential to be accepted by the target users. Of course there were also potential users that pointed out critical aspects about the integration of VR into their working routines.

Looking at the statements of the potential users, there is one point that could prevent almost all users from using our envisioned software platform. This is the potentially too high effort that could be associated with learning and using the platform. Since this is an online survey, we have not used direct quotes of respondents so far. At this point, however, there is a comment that illustrates the main threat for VR in healthcare vividly. The respondent with ID 102 describes it as follows: “We already maintain and document so much that I’m frankly just not into that platform. In the video, I dropped out after the third ‘Design … , Do…, Add … ”. This was one of the most negative comments we received and should not be seen as representative of all the respondents. Nevertheless, it sums up well what the biggest threat to the success of integrating VR applications into healthcare routines is. If users are confronted with too much complexity and too many options, it will discourage them from using it. The day-to-day work of most people in the healthcare sector simply does not allow for too much extra efforts. So when working on VR healthcare applications, one should not fall into aimless actionism and try to enable the users with a myriad of options and settings that allow them to cover every conceivable use case. Rather, applications themselves should take care of a lot of things and provide help wherever possible. This can be done, for example, by automatically analyzing the user’s state, by providing a supervisor monitor, or by providing customizable templates. The focus should rather be on standard use cases that are fast and feasible. This ensures that most people from the healthcare sector will even consider using a product for their work.

Of course, there is still a lot of work to be done to integrate VR into healthcare routines. For the future work we are planning to actually implement the envisioned software platform, taking into account the findings from this work. Studies based on actual prototypes will be necessary to see how well the actual integration into working routines works. Especially ideas like the supervisor monitor will require a close cooperation with the potential users in form of usability tests. Only if there is an understanding of potential users and how they work, the ambitious goal of promoting an autonomous use of VR in healthcare practice can be achieved. We have laid the foundation with this study.

Data Availability Statement

The original contributions presented in the study are included in the article/Supplementary Material, further inquiries can be directed to the corresponding author.

Ethics Statement

Ethical review and approval was not required for the study on human participants in accordance with the local legislation and institutional requirements. Written informed consent for participation was not required for this study in accordance with the national legislation and the institutional requirements.

Author Contributions

SG and KB were responsible for the ethics part of the survey and the interpretation of the data. SG was responsible for the statistical analysis of the ethics part. AH and SB were responsible for the remainder of the survey and the corresponding evaluation and interpretation of the data. SvM, KB, and ML were supervising this project and helped with the structuring of the survey and with writing the article.

Funding

This publication was supported by the Open-Access Publication Fund of the University of Würzburg. The research has been funded by the German Federal Ministry of Education and Research in the project VIA-VR (project numbers 16SV8444 and 16SV8445).

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors, and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Footnotes

¹https://aframe.io/

²https://cospaces.io/edu/

³https://www.instavr.co/

⁴https://www.wondavr.com/

⁵https://www.tiltbrush.com/

⁶https://arvr.google.com/blocks/

⁷All websites were accessed and checked on 16 November 2021

⁸https://www.maxqda.de

⁹https://www.empatica.com/research/e4/

¹⁰https://www.tobiipro.com/de/anwendungsfelder/virtual-reality/

¹¹https://pupil-labs.com/products/vr-ar/

References

Allard, J., Gouranton, V., Lecointre, L., Limet, S., Melin, E., Raffin, B., et al. (2004). “Flowvr: a Middleware for Large Scale Virtual Reality Applications,” in European Conference on Parallel Processing (Springer), 497–505. doi:10.1007/978-3-540-27866-5_65

CrossRef Full Text | Google Scholar

Allard, J., Gouranton, V., Lecointre, L., Melin, E., and Raffin, B. (2002). “Net Juggler: Running Vr Juggler with Multiple Displays on a Commodity Component Cluster,” in Proceedings IEEE Virtual Reality 2002 (IEEE), 273–274.

Google Scholar

Allard, J., Lesage, J.-D., and Raffin, B. (2010). Modularity for Large Virtual Reality Applications. Presence: Teleoperators and Virtual Environments 19, 142–161. doi:10.1162/pres.19.2.142

CrossRef Full Text | Google Scholar

Allard, J., Ménier, C., Boyer, E., and Raffin, B. (2005). “Running Large Vr Applications on a Pc Cluster: the Flowvr Experience,” in 9th International Workshop on Immersive Projection Technology (Aalborg, Denmark: Eurographics Association), 59–68.

Google Scholar

Andone, D., and Frydenberg, M. (2019). “Creating Virtual Reality in a Business and Technology Educational Context,” in Augmented Reality and Virtual Reality (Springer), 147–159. doi:10.1007/978-3-030-06246-0_11

CrossRef Full Text | Google Scholar

Angelov, V., Petkov, E., Shipkovenski, G., and Kalushkov, T. (2020). “Modern Virtual Reality Headsets,” in 2020 International Congress on Human-Computer Interaction, Optimization and Robotic Applications (New York: HORA), 1–5. doi:10.1109/HORA49412.2020.9152604

CrossRef Full Text | Google Scholar

Arlati, S., Greci, L., Mondellini, M., Zangiacomi, A., Santo, D., Franchini, F., et al. (2017). “A Virtual Reality-Based Physical and Cognitive Training System Aimed at Preventing Symptoms of Dementia,” in International Conference on Wireless Mobile Communication and Healthcare (Springer), 117–125.

Google Scholar

Ashtari, N., Bunt, A., McGrenere, J., Nebeling, M., and Chilana, P. K. (2020). “Creating Augmented and Virtual Reality Applications: Current Practices, Challenges, and Opportunities,” in Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (IEEE), 1–13. doi:10.1145/3313831.3376722

CrossRef Full Text | Google Scholar

Bălan, O., Moise, G., Moldoveanu, A., Leordeanu, M., and Moldoveanu, F. (2020). An Investigation of Various Machine and Deep Learning Techniques Applied in Automatic Fear Level Detection and Acrophobia Virtual Therapy. Sensors (Basel) 20, 496. doi:10.3390/s20020496

CrossRef Full Text | Google Scholar

Balog, A., and Pribeanu, C. (2009). Developing a Measurement Scale for the Evaluation of Ar-Based Educational Systems. Stud. Inform. Control. 18, 137–148.

Google Scholar

Beauchamp, T. L., and Childress, J. F. (2019). Principles of Biomedical Ethics. 8 edn. New York: Oxford University Press.:

Google Scholar

Bethel, W., Bass, C., Clay, S. R., Hook, B., Jones, M. T., Sowizral, H., et al. (1999). Scene Graph apis: Wired or Tired? ACM SIGGRAPH 99 Conf. Abstr. Appl. 8, 136–138.

Google Scholar

Bierbaum, A., Just, C., Hartling, P., Meinert, K., Baker, A., and Cruz-Neira, C. (2001). “Vr Juggler: A Virtual Platform for Virtual Reality Application Development,” in Proceedings IEEE Virtual Reality 2001 (IEEE), 89–96.

Google Scholar

Borg, G. (1998). Borg’s Perceived Exertion and Pain Scales. Champaign, Illinois: Human Kinetics.

Google Scholar

Carlsson, C., and Hagsand, O. (1993). DIVE-A Platform for Multi-User Virtual Environments. Comput. Graphics 17, 663–669. doi:10.1016/0097-8493(93)90115-p

CrossRef Full Text | Google Scholar

Cebeci, B., Celikcan, U., and Capin, T. K. (2019). A Comprehensive Study of the Affective and Physiological Responses Induced by Dynamic Virtual Reality Environments. Computer Animation Virtual Worlds 30, e1893. doi:10.1002/cav.1893

CrossRef Full Text | Google Scholar

Chardonnet, J.-R., Mirzaei, M. A., and Merienne, F. (2021). Influence of Navigation Parameters on Cybersickness in Virtual Reality. Virtual Reality 25, 565–574. doi:10.1007/s10055-020-00474-2

CrossRef Full Text | Google Scholar

Conway, M., Audia, S., Burnette, T., Cosgrove, D., and Christiansen, K. (2000). “Alice: Lessons Learned from Building a 3d System for Novices,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (IEEE), 486–493.

Google Scholar

Currie, J., Bond, R. R., McCullagh, P., Black, P., Finlay, D. D., Gallagher, S., et al. (2019). Wearable Technology-Based Metrics for Predicting Operator Performance during Cardiac Catheterisation. Int. J. CARS 14, 645–657. doi:10.1007/s11548-019-01918-0

CrossRef Full Text | Google Scholar

da Costa, R. M. E. M., and de Carvalho, L. A. V. (2004). The Acceptance of Virtual Reality Devices for Cognitive Rehabilitation: a Report of Positive Results with Schizophrenia. Comp. Methods Programs Biomed. 73, 173–182. doi:10.1016/s0169-2607(03)00066-x

CrossRef Full Text | Google Scholar

Davis, F. D. (1989). Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology. MIS Q. 13, 319–340. doi:10.2307/249008

CrossRef Full Text | Google Scholar

Dellazizzo, L., Potvin, S., Luigi, M., and Dumais, A. (2020). Evidence on Virtual Reality-Based Therapies for Psychiatric Disorders: Meta-Review of Meta-Analyses. J. Med. Internet Res. 22, e20889. doi:10.2196/20889

PubMed Abstract | CrossRef Full Text | Google Scholar

Döllinger, N., Wienrich, C., Wolf, E., Botsch, M., and Latoschik, M. E. (2019). “Vitras - Virtual Reality Therapy by Stimulation of Modulated Body Image - Project Outline,” in Mensch und Computer 2019-Workshop band (Bonn, Germany: Gesellschaft für Informatik e.V.), 606–611.

Google Scholar

Figueroa, P., Green, M., and Hoover, H. J. (2002). “Intml: a Description Language for Vr Applications,” in Proceedings of the Seventh International Conference on 3D Web Technology (IEEE), 53–58.

Google Scholar

Fischbach, M., Wiebusch, D., and Latoschik, M. E. (2017). Semantic Entity-Component State Management Techniques to Enhance Software Quality for Multimodal Vr-Systems. IEEE Trans. Vis. Comput. Graphics 23, 1342–1351. doi:10.1109/tvcg.2017.2657098

PubMed Abstract | CrossRef Full Text | Google Scholar

Fussell, S. G., and Truong, D. (2021). Accepting Virtual Reality for Dynamic Learning: an Extension of the Technology Acceptance Model. Interactive Learn. Environments 3, 1–18. doi:10.1080/10494820.2021.2009880

CrossRef Full Text | Google Scholar

Gall, D., Roth, D., Stauffert, J.-P., Zarges, J., and Latoschik, M. E. (2021). Embodiment in Virtual Reality Intensifies Emotional Responses to Virtual Stimuli. Front. Psychol. 12, 1. doi:10.3389/fpsyg.2021.674179

CrossRef Full Text | Google Scholar

Garcia-Palacios, A., Botella, C., Hoffman, H., and Fabregat, S. (2007). Comparing Acceptance and Refusal Rates of Virtual Reality Exposure vs. In Vivo Exposure by Patients with Specific Phobias. CyberPsychology Behav. 10, 722–724. doi:10.1089/cpb.2007.9962

CrossRef Full Text | Google Scholar

Gianola, S., Stucovitz, E., Castellini, G., Mascali, M., Vanni, F., Tramacere, I., et al. (2020). Effects of Early Virtual Reality-Based Rehabilitation in Patients with Total Knee Arthroplasty. Medicine 99, e19136. doi:10.1097/md.0000000000019136

PubMed Abstract | CrossRef Full Text | Google Scholar

Glémarec, Y., Lugrin, J.-L., Bosser, A.-G., Collins Jackson, A., Buche, C., and Latoschik, M. E. (2021). Indifferent or Enthusiastic? Virtual Audiences Animation and Perception in Virtual Reality. Front. Virtual Reality 2, 72. doi:10.3389/frvir.2021.666232

CrossRef Full Text | Google Scholar

Gonzalez, D. S., Moro, A. D., Quintero, C., and Sarmiento, W. J. (2016). “Fear Levels in Virtual Environments, an Approach to Detection and Experimental User Stimuli Sensation,” in 2016 XXI Symposium on Signal Processing, Images and Artificial Vision (STSIVA) (IEEE), 1–6. doi:10.1109/stsiva.2016.7743323

CrossRef Full Text | Google Scholar

Greenbaum, D. (2015). Expanding Elsi to All Areas of Innovative Science and Technology. Nat. Biotechnol. 33, 425–426. doi:10.1038/nbt.3193

PubMed Abstract | CrossRef Full Text | Google Scholar

Greenhalgh, C., and Benford, S. (1995). “Massive: a Distributed Virtual Reality System Incorporating Spatial Trading,” in Proceedings of 15th International Conference on Distributed Computing Systems (IEEE), 27–34.

Google Scholar

Groth, C., Tauscher, J.-P., Heesen, N., Castillo, S., and Magnor, M. (2021). “Visual Techniques to Reduce Cybersickness in Virtual Reality,” in 2021 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW) (IEEE), 486–487. doi:10.1109/vrw52623.2021.00125

CrossRef Full Text | Google Scholar

Halbig, A., and Latoschik, M. E. (2021). A Systematic Review of Physiological Measurements, Factors, Methods, and Applications in Virtual Reality. Front. Virtual Reality 2, 89. doi:10.3389/frvir.2021.694567

CrossRef Full Text | Google Scholar

Ham, J., Cho, D., Oh, J., and Lee, B. (2017). “Discrimination of Multiple Stress Levels in Virtual Reality Environments Using Heart Rate Variability,” in 2017 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) (IEEE), 3989–3992. doi:10.1109/embc.2017.8037730

CrossRef Full Text | Google Scholar

Hamzeheinejad, N., Roth, D., Götz, D., Weilbach, F., and Latoschik, M. E. (2019). “Physiological Effectivity and User Experience of Immersive Gait Rehabilitation,” in 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR) (IEEE), 1421–1429. doi:10.1109/vr.2019.8797763

CrossRef Full Text | Google Scholar

Hartanto, D. W., Kang, N., Brinkman, W. P., Kampmann, I. L., Morina, N., Emmelkamp, P. G., et al. (2012). Automatic Mechanisms for Measuring Subjective Unit of Discomfort. Stud. Health Technol. Inform. 181, 192–196.

PubMed Abstract | Google Scholar

Haugstvedt, A.-C., and Krogstie, J. (2012). “Mobile Augmented Reality for Cultural Heritage: A Technology Acceptance Study,” in 2015 IEEE Mixed and Augmented Reality (IEEE), 247–255. doi:10.1109/ismar.2012.6402563

CrossRef Full Text | Google Scholar

Herumurti, D., Yuniarti, A., Rimawan, P., and Yunanto, A. A. (2019). “Overcoming Glossophobia Based on Virtual Reality and Heart Rate Sensors,” in 2019 IEEE International Conference on Industry 4.0, Artificial Intelligence, and Communications Technology (IAICT) (IEEE), 139–144. doi:10.1109/iciaict.2019.8784846

CrossRef Full Text | Google Scholar

Hesina, G., Schmalstieg, D., Furhmann, A., and Purgathofer, W. (1999). “Distributed Open Inventor: A Practical Approach to Distributed 3d Graphics,” in Proceedings of the ACM Symposium on Virtual Reality Software and Technology (New York: ACM), 74–81.

Google Scholar

Hilton, D., Cobb, S., Pridmore, T., Gladman, J., and Edmans, J. (2011). “Development and Evaluation of a Mixed Reality System for Stroke Rehabilitation,” in Advanced Computational Intelligence Paradigms in Healthcare 6. Virtual Reality in Psychotherapy, Rehabilitation, and Assessment (Springer), 193–228. doi:10.1007/978-3-642-17824-5_10

CrossRef Full Text | Google Scholar

Islam, R., Ang, S., and Quarles, J. (2021). “Cybersense: A Closed-Loop Framework to Detect Cybersickness Severity and Adaptively Apply Reduction Techniques,” in 2021 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW) (IEEE), 148–155. doi:10.1109/vrw52623.2021.00035

CrossRef Full Text | Google Scholar

Jackson, B., and Keefe, D. F. (2016). Lift-off: Using Reference Imagery and Freehand Sketching to Create 3d Models in Vr. IEEE Trans. Vis. Comput. Graphics 22, 1442–1451. doi:10.1109/tvcg.2016.2518099

PubMed Abstract | CrossRef Full Text | Google Scholar

Johnson, T., Bauler, L., Vos, D., Hifko, A., Garg, P., Ahmed, M., et al. (2020). Virtual Reality Use for Symptom Management in Palliative Care: A Pilot Study to Assess User Perceptions. J. Palliat. Med. 23, 1233–1238. doi:10.1089/jpm.2019.0411

PubMed Abstract | CrossRef Full Text | Google Scholar

Kaimara, P., Oikonomou, A., and Deliyannis, I. (2021). Could Virtual Reality Applications Pose Real Risks to Children and Adolescents? a Systematic Review of Ethical Issues and Concerns. Virtual Reality 11, 1–39.

Google Scholar

Karaosmanoglu, S., Rings, S., Kruse, L., Stein, C., and Steinicke, F. (2021). Lessons Learned from a Human-Centered Design of an Immersive Exergame for People with Dementia. Proc. ACM Hum.-Comput. Interact. 5, 1–27. doi:10.1145/3474679

CrossRef Full Text | Google Scholar

Karat, J., and Karat, C. M. (2003). The Evolution of User-Centered Focus in the Human-Computer Interaction Field. IBM Syst. J. 42, 532–541. doi:10.1147/sj.424.0532

CrossRef Full Text | Google Scholar

Kaur, R., Sun, R., Ziegelman, L., Sowers, R., and Hernandez, M. E. (2019). “Using Virtual Reality to Examine the Neural and Physiological Responses to Height and Perturbations in Quiet Standing,” in 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) (IEEE), 5233–5236. doi:10.1109/embc.2019.8857647

PubMed Abstract | CrossRef Full Text | Google Scholar

Kellmeyer, P., Biller-Andorno, N., and Meynen, G. (2019). Ethical Tensions of Virtual Reality Treatment in Vulnerable Patients. Nat. Med. 25, 1185–1188. doi:10.1038/s41591-019-0543-y

PubMed Abstract | CrossRef Full Text | Google Scholar

Kellmeyer, P. (2018). Neurophilosophical and Ethical Aspects of Virtual Reality Therapy in Neurology and Psychiatry. Camb Q. Healthc. Ethics 27, 610–627. doi:10.1017/s0963180118000129

PubMed Abstract | CrossRef Full Text | Google Scholar

Kelso, J., Arsenault, L. E., Satterfield, S. G., and Kriz, R. D. (2002). “Diverse: A Framework for Building Extensible and Reconfigurable Device Independent Virtual Environments,” in Proceedings IEEE Virtual Reality 2002 (IEEE), 183–190.

Google Scholar

Kern, F., Winter, C., Gall, D., Käthner, I., Pauli, P., and Latoschik, M. E. (2019). “Immersive Virtual Reality and Gamification within Procedurally Generated Environments to Increase Motivation during Gait Rehabilitation,” in 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR) (IEEE), 500–509. doi:10.1109/vr.2019.8797828

CrossRef Full Text | Google Scholar

Kool, H. (2016). The Ethics of Immersive Journalism: A Rhetorical Analysis of News Storytelling with Virtual Reality Technology. Intersect: Stanford J. Sci. Technol. Soc. 9, 1–11.

Google Scholar

Kuck, R., Wind, J., Riege, K., Bogen, M., and Birlinghoven, S. (2008). “Improving the Avango Vr/ar Framework: Lessons Learned,” in Workshop Virtuelle und Erweiterte Realität (IEEE), 209–220.

Google Scholar

Kuntze, M. F., Stoermer, R., Mueller-Spahn, F., and Bullinger, A. H. (2002). Ethical Codes and Values in a Virtual World. CyberPsychology Behav. 5, 203–206. doi:10.1089/109493102760147187

PubMed Abstract | CrossRef Full Text | Google Scholar

Latoschik, M. E., and Tramberend, H. (2011). “Simulator X: A Scalable and Concurrent Architecture for Intelligent Realtime Interactive Systems,” in 2011 IEEE Virtual Reality Conference (IEEE), 171–174. doi:10.1109/vr.2011.5759457

CrossRef Full Text | Google Scholar

Laver, K. E., Lange, B., George, S., Deutsch, J. E., Saposnik, G., and Crotty, M. (2017). Virtual Reality for Stroke Rehabilitation. Cochrane database Syst. Rev. 11 (11), CD008349. doi:10.1002/14651858.cd008349.pub4

PubMed Abstract | CrossRef Full Text | Google Scholar

Lee, J., Kim, J., and Choi, J. Y. (2019). The Adoption of Virtual Reality Devices: The Technology Acceptance Model Integrating Enjoyment, Social Interaction, and Strength of the Social Ties. Telematics Inform. 39, 37–48. doi:10.1016/j.tele.2018.12.006

CrossRef Full Text | Google Scholar

Lee, Y., Kozar, K. A., and Larsen, K. R. (2003). The Technology Acceptance Model: Past, Present, and Future. Commun. Assoc. Inf. Syst. 12, 50. doi:10.17705/1cais.01250

CrossRef Full Text | Google Scholar

Leung, H. K. N., and Wong, P. W. L. (1997). A Study of User Acceptance Tests. Softw. Qual. J. 6, 137–149. doi:10.1023/a:1018503800709

CrossRef Full Text | Google Scholar

Lewis, C. H., and Griffin, M. J. (1997). “Human Factors Consideration in Clinical Applications of Virtual Reality,” in Virtual Reality in Neuro-Psycho-Physiology: Cognitive, Clinical and Methodological Issues in Assessment and Treatment. Editor G. Riva (Amsterdam, Netherlands: IOS Press), 35–56.

Google Scholar

Madary, M., and Metzinger, T. K. (2016). Real Virtuality: a Code of Ethical Conduct. Recommendations for Good Scientific Practice and the Consumers of Vr-Technology. Front. Robotics AI 2016, 3. doi:10.3389/frobt.2016.00003

CrossRef Full Text | Google Scholar

Manzeschke, A., Weber, K., Rother, E., and Fangerau, H. (2015). Results of the Study ”Ethical Questions in the Area of Age Appropriate Assisting Systems”. Berlin: VDI.

Google Scholar

Marloth, M., Chandler, J., and Vogeley, K. (2020). Psychiatric Interventions in Virtual Reality: Why We Need an Ethical Framework. Camb Q. Healthc. Ethics 29, 574–584. doi:10.1017/s0963180120000328

PubMed Abstract | CrossRef Full Text | Google Scholar

Mathur, A. S. (2015). “Low Cost Virtual Reality for Medical Training,” in 2015 IEEE Virtual Reality (VR) (IEEE), 345–346. doi:10.1109/vr.2015.7223437

CrossRef Full Text | Google Scholar

Mayring, P. (2015). Qualitative Inhaltsanalyse. Grundlagen und Techniken. 12 edn. Weinheim: Beltz.

Google Scholar

Mertens, G., Wagensveld, P., and Engelhard, I. M. (2019). Cue Conditioning Using a Virtual Spider Discriminates between High and Low Spider Fearful Individuals. Comput. Hum. Behav. 91, 192–200. doi:10.1016/j.chb.2018.10.006

CrossRef Full Text | Google Scholar

Moldoveanu, A., Ferche, O.-M., Moldoveanu, F., Lupu, R. G., Cinteza, D., Constantin Irimia, D., et al. (2019). The Travee System for a Multimodal Neuromotor Rehabilitation. IEEE Access 7, 8151–8171. doi:10.1109/access.2018.2886271

CrossRef Full Text | Google Scholar

Musschenga, A. W. (1997). The Relation between Concepts of Quality-Of-Life, Health and Happiness. J. Med. Philos. 22, 11–28. doi:10.1093/jmp/22.1.11

CrossRef Full Text | Google Scholar

Nebeling, M., and Madier, K. (2019). “360proto: Making Interactive Virtual Reality & Augmented Reality Prototypes from Paper,” in Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (IEEE), 1–13.

Google Scholar

Nebeling, M., and Speicher, M. (2018). “The Trouble with Augmented Reality/virtual Reality Authoring Tools,” in 2018 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) (IEEE), 333–337. doi:10.1109/ismar-adjunct.2018.00098

CrossRef Full Text | Google Scholar

Neelakantam, S., and Pant, T. (2017). “Introduction to A-Frame,” in Learning Web-Based Virtual Reality (Springer), 17–38. doi:10.1007/978-1-4842-2710-7_4

CrossRef Full Text | Google Scholar

Nelles, J., Kohns, S., Spies, J., Bröhl, C., Brandl, C., Mertens, A., et al. (2016). “Best-practice Approach for a Solution-Oriented Technology Assessment: Ethical, Legal, and Social Issues in the Context of Human-Robot Collaboration,” in Advances in Ergonomic Design of Systems, Products and Processes: Proceedings of the Annual Meeting of GfA 2016 (Berlin: Springer), 1–14. doi:10.1007/978-3-662-53305-5_1:

CrossRef Full Text | Google Scholar

Nemire, K., Beil, J., and Swan, R. W. (1999). Preventing Teen Smoking with Virtual Reality. CyberPsychology Behav. 2, 35–47. doi:10.1089/cpb.1999.2.35

PubMed Abstract | CrossRef Full Text | Google Scholar

Niki, K., Okamoto, Y., Maeda, I., Mori, I., Ishii, R., Matsuda, Y., et al. (2019). A Novel Palliative Care Approach Using Virtual Reality for Improving Various Symptoms of Terminal Cancer Patients: a Preliminary Prospective, Multicenter Study. J. Palliat. Med. 22, 702–707. doi:10.1089/jpm.2018.0527

CrossRef Full Text | Google Scholar

Pavlik, R. A., and Vance, J. M. (2012). “Vr Jugglua: A Framework for Vr Applications Combining Lua, Openscenegraph, and Vr Juggler,” in 2012 5th Workshop on Software Engineering and Architectures for Realtime Interactive Systems (SEARIS) (IEEE), 29–35. doi:10.1109/searis.2012.6231166

CrossRef Full Text | Google Scholar

Plouzeau, J., Chardonnet, J.-R., and Merienne, F. (2018). “Using Cybersickness Indicators to Adapt Navigation in Virtual Reality: a Pre-study,” in 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR) (IEEE), 661–662. doi:10.1109/vr.2018.8446192

CrossRef Full Text | Google Scholar

Rizzo, A., Schulteis, M. T., and Rothbaum, B. O. (2003). “Ethical Issues for the Use of Virtual Reality in the Psychological Sciences,” in Ethical Issues in Clinical Neuropsychology (Lisse, NL: Swets & Zeitlinger Publishers), 243–280.

Google Scholar

Robitaille, P., and McGuffin, M. J. (2019). “Increased Affect-Arousal in Vr Can Be Detected from Faster Body Motion with Increased Heart Rate,” in Proceedings of the ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games (IEEE), 1–6. doi:10.1145/3306131.3317022

CrossRef Full Text | Google Scholar

Rohlf, J., and Helman, J. (1994). “Iris Performer: A High Performance Multiprocessing Toolkit for Real-Time 3d Graphics,” in Proceedings of the 21st Annual Conference on Computer Graphics and Interactive Techniques (IEEE), 381–394.

Google Scholar

Šalkevicius, J., Damaševičius, R., Maskeliunas, R., and Laukienė, I. (2019). Anxiety Level Recognition for Virtual Reality Therapy System Using Physiological Signals. Electronics 8, 1039.

Google Scholar

Sanders, E. B.-N. (2002). “From User-Centered to Participatory Design Approaches,” in Design and the Social Sciences (Boca Raton, Florida: CRC Press), 18–25. doi:10.1201/9780203301302.ch1

CrossRef Full Text | Google Scholar

Santos, S. G., and Cardoso, J. (2019). A-frame Experimentation and Evaluation for the Development of Interactive Vr: A Virtual Tour of the Conimbriga Museum. Coimbra, Portugal: Technical Report.

Google Scholar

Shaw, C., Green, M., Liang, J., and Sun, Y. (1993). Decoupled Simulation in Virtual Reality with the Mr Toolkit. ACM Trans. Inf. Syst. 11, 287–317. doi:10.1145/159161.173948

CrossRef Full Text | Google Scholar

Shiban, Y., Fruth, M. B., Pauli, P., Kinateder, M., Reichenberger, J., and Mühlberger, A. (2016). Treatment Effect on Biases in Size Estimation in Spider Phobia. Biol. Psychol. 121, 146–152. doi:10.1016/j.biopsycho.2016.03.005

PubMed Abstract | CrossRef Full Text | Google Scholar

Snoswell, A. J., and Snoswell, C. L. (2019). Immersive Virtual Reality in Health Care: Systematic Review of Technology and Disease States. JMIR Biomed. Eng. 4, e15025. doi:10.2196/15025

CrossRef Full Text | Google Scholar

So, S., and Lu, E. (2019). “Virtual Reality in Visual Arts Education: A Study on Using Google Tilt brush,” in EdMedia+ Innovate Learning (Association for the Advancement of Computing in Education) (Waynesville: AACE), 1485–1490.

Google Scholar

Spiegel, J. S. (2018). The Ethics of Virtual Reality Technology: Social Hazards and Public Policy Recommendations. Sci. Eng. Ethics 24, 1537–1550. doi:10.1007/s11948-017-9979-y

PubMed Abstract | CrossRef Full Text | Google Scholar

Strauss, P. S., and Carey, R. (1992). An Object-Oriented 3d Graphics Toolkit. SIGGRAPH Comput. Graph. 26, 341–349. doi:10.1145/142920.134089

CrossRef Full Text | Google Scholar

Syed-Abdul, S., Malwade, S., Nursetyo, A. A., Sood, M., Bhatia, M., Barsasella, D., et al. (2019). Virtual Reality Among the Elderly: a Usefulness and Acceptance Study from Taiwan. BMC Geriatr. 19, 223–310. doi:10.1186/s12877-019-1218-8

PubMed Abstract | CrossRef Full Text | Google Scholar

Teixeira, J., and Palmisano, S. (2021). Effects of Dynamic Field-Of-View Restriction on Cybersickness and Presence in Hmd-Based Virtual Reality. Virtual Reality 25, 433–445. doi:10.1007/s10055-020-00466-2

CrossRef Full Text | Google Scholar

Thompson-Lake, D. G. Y., Cooper, K. N., Mahoney, J. J., Bordnick, P. S., Salas, R., Kosten, T. R., et al. (2015). Withdrawal Symptoms and Nicotine Dependence Severity Predict Virtual Reality Craving in Cigarette-Deprived Smokers. Nicotine Tob. Res. 17, 796–802. doi:10.1093/ntr/ntu245

PubMed Abstract | CrossRef Full Text | Google Scholar

Tramberend, H. (1999). “Avocado: A Distributed Virtual Reality Framework,” in Proceedings IEEE Virtual Reality (Cat. No. 99CB36316) (IEEE), 14–21.

Google Scholar

Turilli, M., and Floridi, L. (2009). The Ethics of Information Transparency. Ethics Inf. Technol. 11, 105–112. doi:10.1007/s10676-009-9187-9

CrossRef Full Text | Google Scholar

Venkatesh, V., and Davis, F. D. (2000). A Theoretical Extension of the Technology Acceptance Model: Four Longitudinal Field Studies. Manag. Sci. 46, 186–204. doi:10.1287/mnsc.46.2.186.11926

CrossRef Full Text | Google Scholar

von Mammen, S., Müller, A., Latoschik, M. E., Botsch, M., Brukamp, K., Schröder, C., et al. (2019). “Via Vr: A Technology Platform for Virtual Adventures for Healthcare and Well-Being,” in 2019 11th International Conference on Virtual Worlds and Games for Serious Applications (VS-Games) (IEEE), 1–2. doi:10.1109/VS-Games.2019.8864580

CrossRef Full Text | Google Scholar

Walker, R. L., and Morrissey, C. (2014). Bioethics Methods in the Ethical, Legal, and Social Implications of the Human Genome Project Literature. Bioethics 28, 481–490. doi:10.1111/bioe.12023

PubMed Abstract | CrossRef Full Text | Google Scholar

Wang, R., and Qian, X. (2010). OpenSceneGraph 3.0: Beginner’s Guide. Birmingham: Packt Publishing Ltd.

Google Scholar

Wang, W., Yang, B., Guan, C., and Li, B. (2019a). “A Vr Combined with Mi-Bci Application for Upper Limb Rehabilitation of Stroke,” in 2019 IEEE MTT-S International Microwave Biomedical Conference (IMBioC) (IEEE), 1–4. doi:10.1109/imbioc.2019.8777805

CrossRef Full Text | Google Scholar

Wang, Y.-g., Liu, M.-h., and Shen, Z.-h. (2019b). A Virtual Reality Counterconditioning Procedure to Reduce Methamphetamine Cue-Induced Craving. J. Psychiatr. Res. 116, 88–94. doi:10.1016/j.jpsychires.2019.06.007

CrossRef Full Text | Google Scholar

Weijers, D., and DiSilvestro, R. (2017). “The Morality of Experience Machines for Palliative and End-Of-Life Care,” in Experience Machines: The Philosophy of Virtual Worlds. Editor M. Silcox (London: Rowman & Littlefield), 183–201.

Google Scholar

Wenninger, S., Achenbach, J., Bartl, A., Latoschik, M. E., and Botsch, M. (2020). “Realistic Virtual Humans from Smartphone Videos,” in VRST. Editors R. J. Teather, C. Joslin, W. Stuerzlinger, P. Figueroa, Y. Hu, A. U. Batmazet al. (New York: ACM), 1–29. doi:10.1145/3385956.3418940

CrossRef Full Text | Google Scholar

Wolf, E., Merdan, N., Dölinger, N., Mal, D., Wienrich, C., Botsch, M., et al. (2021). “The Embodiment of Photorealistic Avatars Influences Female Body Weight Perception in Virtual Reality,” in 2021 IEEE Virtual Reality and 3D User Interfaces (VR) (IEEE), 65–74. doi:10.1109/vr50410.2021.00027

CrossRef Full Text | Google Scholar

Yellowlees, P. M., Holloway, K. M., and Parish, M. B. (2012). Therapy in Virtual Environments-Clinical and Ethical Issues. Telemed. e-Health 18, 558–564. doi:10.1089/tmj.2011.0195

CrossRef Full Text | Google Scholar

Ying-Chun, L., and Chwen-Liang, C. (2018). “The Application of Virtual Reality Technology in Art Therapy: A Case of Tilt brush,” in 2018 1st IEEE International Conference on Knowledge Innovation and Invention (ICKII) (IEEE), 47–50. doi:10.1109/ickii.2018.8569081

CrossRef Full Text | Google Scholar

Zając-Lamparska, L., Wiłkość-Dębczyńska, M., Wojciechowski, A., Podhorecka, M., Polak-Szabela, A., Warchoł, Ł., et al. (2019). Effects of Virtual Reality-Based Cognitive Training in Older Adults Living without and with Mild Dementia: a Pretest-Posttest Design Pilot Study. BMC Res. Notes 12, 776–778. doi:10.1186/s13104-019-4810-2

PubMed Abstract | CrossRef Full Text | Google Scholar

Zhang, A. (2020). Immersive Virtual Reality Training Improved Upper Extremity Function in Patients with Spinal Cord Injuries. Omaha, Nebraska: A case series.

Google Scholar

Zhang, L., and Oney, S. (2020). “Flowmatic: An Immersive Authoring Tool for Creating Interactive Scenes in Virtual Reality,” in Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (IEEE), 342–353. doi:10.1145/3379337.3415824

CrossRef Full Text | Google Scholar