Panagiotis Kourtesis
Simona Collina3,4
Leonidas A. A. Doumas
Sarah E. MacPherson- 1Human Cognitive Neuroscience, Department of Psychology, University of Edinburgh, Edinburgh, United Kingdom
- 2Department of Psychology, University of Edinburgh, Edinburgh, United Kingdom
- 3Lab of Experimental Psychology, Suor Orsola Benincasa University of Naples, Naples, Italy
- 4Interdepartmental Centre for Planning and Research “Scienza Nuova”, Suor Orsola Benincasa University of Naples, Naples, Italy
Immersive virtual reality (VR) emerges as a promising research and clinical tool. However, several studies suggest that VR induced adverse symptoms and effects (VRISE) may undermine the health and safety standards, and the reliability of the scientific results. In the current literature review, the technical reasons for the adverse symptomatology are investigated to provide suggestions and technological knowledge for the implementation of VR head-mounted display (HMD) systems in cognitive neuroscience. The technological systematic literature indicated features pertinent to display, sound, motion tracking, navigation, ergonomic interactions, user experience, and computer hardware that should be considered by the researchers. Subsequently, a meta-analysis of 44 neuroscientific or neuropsychological studies involving VR HMD systems was performed. The meta-analysis of the VR studies demonstrated that new generation HMDs induced significantly less VRISE and marginally fewer dropouts. Importantly, the commercial versions of the new generation HMDs with ergonomic interactions had zero incidents of adverse symptomatology and dropouts. HMDs equivalent to or greater than the commercial versions of contemporary HMDs accompanied with ergonomic interactions are suitable for implementation in cognitive neuroscience. In conclusion, researchers' technological competency, along with meticulous methods and reports pertinent to software, hardware, and VRISE, are paramount to ensure the health and safety standards and the reliability of neuroscientific results.
Introduction
In recent years, virtual reality (VR) technology has attracted attention, demonstrating its utility and potency in the field of neuroscience and neuropsychology (Rizzo et al., 2004; Bohil et al., 2011; Parsons, 2015). Traditional approaches in human neuroscience involve the utilization of static and simple stimuli which arguably lack ecological validity (Parsons, 2015). VR offers the usage of dynamic stimuli and interactions with a high degree of control within an ecologically valid environment which enables the collection of advanced cognitive and behavioral data (Rizzo et al., 2004; Bohil et al., 2011; Parsons, 2015). VR can be combined with non-invasive imaging techniques (Bohil et al., 2011; Parsons, 2015) and has been effective in the assessment of cognitive and affective functions and clinical conditions (e.g., social stress disorders) which require ecological validity (Rizzo et al., 2004; Parsons, 2015) for their assessment, rehabilitation and treatment (e.g., post-traumatic stress disorder) (Rizzo et al., 2004; Bohil et al., 2011).
However, researchers and clinicians have reported caveats with the implementation of immersive VR interventions and assessments, particularly when head mounted display (HMD) systems are utilized (Sharples et al., 2008; Davis et al., 2015; de França and Soares, 2017; Palmisano et al., 2017). A predominant concern is the presence of adverse physiological symptoms (i.e., cyber/simulation-sickness which includes nausea, disorientation, instability, dizziness, and fatigue). These undesirable effects are categorized as VR Induced Symptoms and Effects (VRISE) (Sharples et al., 2008; Davis et al., 2015; de França and Soares, 2017; Palmisano et al., 2017), and are evaluated by using questionnaires such as the Simulator Sickness Questionnaire (Kennedy et al., 1993) and the Virtual Reality Sickness Questionnaire (Kim et al., 2018).
VRISE may risk the health and safety of participants or patients (Kane and Parsons, 2017; Parsons et al., 2018), which raises ethical considerations for the adoption of VR HMDs as research and clinical tools. Additionally, the presence of VRISE has modulated substantial decline in reaction times and overall cognitive performance (Plant and Turner, 2009; Nalivaiko et al., 2015; Plant, 2016; Nesbitt et al., 2017; Mittelstaedt et al., 2018), as well as increasing body temperature and heart rates (Nalivaiko et al., 2015). Also, the presence of VRISE robustly increases cerebral blood flow and oxyhemoglobin concentration (Gavgani et al., 2018), the power of brain signals (Arafat et al., 2018), and the connectivity between stimulus response brain regions and nausea-processing brain regions (Toschi et al., 2017). Thus, VRISE could be considered confounding variables, which significantly undermine the reliability of neuropsychological, physiological, and neuroimaging data.
VRISE are predominantly mediated by an oculomotor discrepancy between what is being perceived through the oculomotor (optic nerve) sensor and what is being sensed via the rest of the afferent nerves in the human body (Sharples et al., 2008; Davis et al., 2015; de França and Soares, 2017; Palmisano et al., 2017). Nevertheless, technologically speaking, VRISE are derivatives of hardware and software inadequacies, i.e., the type of display screen, resolution, and refresh rate of the image, the size of the field of view (FOV) as well as non-ergonomic movements within an interaction in the virtual environment (VE; de França and Soares, 2017; Palmisano et al., 2017). Notably, VR HMDs have substantially evolved during the last two decades. Important differences may be seen between the HMDs released before 2013 (old generation) and those released from 2013 onwards (new generation). While the last old generation HMD was released in 2001 (i.e., nVisor SX111), the year 2013 is used to distinguish between old and new generation HMDs, since it is the year that the first new generation HMD prototype (i.e., Oculus Development Kit 1) was released. This systematic review attempts to clarify the technological etiologies of VRISE and provide pertinent suggestions for the implementation of VR HMDs in cognitive neuroscience and neuropsychology. In addition, a meta-analysis of the neuroscience studies that have implemented VR HMDs will be conducted to elucidate the frequency of VRISE and dropout rates as per the VR HMD generation.
Technological Systematic Review
In Table 1, a glossary of the key terms and concepts is provided to assist with comprehension of the utilized terminology. We followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines using a decremental stepwise method to perform the literature review (see Figure 1). The selected papers and book chapters included an explicit explanation and discussion of VRISE and users' experiences pertinent with the specified technological features of the VR hardware and software. Digital databases specialized in technologies were used: (1) IEEE Xplore Digital Library; (2) ACM Digital Library; (3) ScienceDirect; (4) MIT CogNet; and (5) Scopus. Two categories of keywords were used, where each category had three or more keywords and each paper had to include at least one keyword from each category in the main body of the text. The categories were: (1) “virtual reality” OR “immersive virtual reality” OR “head-mounted display;” AND (2) “VRISE” OR “motion sickness” OR “cyber sickness” OR “simulation sickness.” Finally, the extracted information from the identified papers was clustered together under common features (i.e., display, sound, motion tracking, navigation, ergonomic interactions, user experience, and computer hardware).
Table 1. Glossary of key terms and concepts.
Figure 1. Decremental stepwise method for the technological literature review.
Technological Etiologies of VRISE
Display
VR HMDs use the following three types of screens: Cathode Ray Tubes (CRT); Liquid Crystal Display (LCD); and organic light emitting diode (OLED). LCD screens replaced CRT ones due to VRISE (Costello, 1997). LCD, in comparison to CRT, alleviated the probability of visual complications and physical burdens (e.g., fatigue) (Costello, 1997). However, the suitability of LCD was challenged by the emergence of OLED screens. While old generation VR HMDs mainly utilize LCD screens (Costello, 1997), the commercial versions of new generation VR HMDs predominantly use OLED screens (Kim J. W. et al., 2017). The OLED screens have been found to be better than LCD screens for general implementation in VR, because of their faster response times, lighter weight, and better color quality (Kim J. W. et al., 2017). OLED screens decrease the likelihood of VRISE and offer an improved VR display (Kim J. W. et al., 2017).
Three more factors related to display type are crucial for the avoidance of VRISE: the width of the FOV (Rakkolainen et al., 2016; Kim J. W. et al., 2017); the resolution of the image per eye (Hecht, 2016; Rakkolainen et al., 2016; Kim J. W. et al., 2017; Brennesholtz, 2018); and the latency of the images (frames per second) (Hecht, 2016; Rakkolainen et al., 2016; Kim J. W. et al., 2017; Brennesholtz, 2018). A wider FOV significantly decreases the chance of VRISE and increases the level of immersion (Rakkolainen et al., 2016; Kim J. W. et al., 2017). The canonical guidelines suggest a lowest threshold of 110° FOV (diagonal) (Hecht, 2016; Rakkolainen et al., 2016; Kim J. W. et al., 2017; Brennesholtz, 2018). In addition, an increased refresh rate and resolution alleviates the danger of discomfort or VRISE (Hecht, 2016; Rakkolainen et al., 2016; Kim J. W. et al., 2017; Brennesholtz, 2018). The refresh rate should be ≥75 Hz (i.e., ≥75 frames per s) (Goradia et al., 2014; Hecht, 2016; Brennesholtz, 2018), while the resolution is required to be higher than 960 × 1,080 sub-pixels per eye (Goradia et al., 2014).
Sound
A second important consideration for a user's experience in VR is the sound quality. The integration of spatialized sounds (e.g., ambient and feedback sounds) in the VE may increase the level of immersion, pleasantness of the experience, and successful navigation (Vorländer and Shinn-Cunningham, 2014), while they significantly decrease the likelihood of VRISE (Viirre et al., 2014). However, the volume and localization of sounds need to be optimized in terms of audio spatialization to ensure a user's experience is pleasant without adverse VRISE (Viirre et al., 2014; Vorländer and Shinn-Cunningham, 2014).
Motion Tracking
Motion tracking in VR is a pre-condition for naturalistic movement within an immersive VE (Slater and Wilbur, 1997; Stanney and Hale, 2014). Motion tracking allows the precise tracking of the user's physical body within the VE (i.e., it allows the computer to provide accurate environmental feedback, which modulates and consolidates the awareness of the position and movement of the user's body). This phenomenon is called proprioception or kinesthesia (Slater and Wilbur, 1997) and is linked with vestibular and oculomotor mediated VRISE (Slater and Wilbur, 1997; Plouzeau et al., 2015; Caputo et al., 2017). Hence, motion tracking should be adequately rapid and accurate to facilitate ergonomic interactions in the VE (Caputo et al., 2017).
Navigation
A highly important factor in the quality of VR software and to avoid VRISE is the movement of the user in the VE (Porcino et al., 2017). New generation HMDs deliver an adequate play area for interactions to facilitate ecologically valid scenarios (Porcino et al., 2017; Borrego et al., 2018). However, there are restrictions in the size of the play area, which does not permit navigation solely by physical walking (Porcino et al., 2017; Borrego et al., 2018). Teleportation allows movement beyond the play area size and elicits a high-level of immersion and pleasant user experience, whilst alleviating VRISE (Bozgeyikli et al., 2016; Frommel et al., 2017; Porcino et al., 2017). In contrast, movement dependent on a touchpad, keyboard, or joystick results to high occurrences of VRISE (Bozgeyikli et al., 2016; Frommel et al., 2017; Porcino et al., 2017). Therefore, teleportation in conjunction with physical movement (i.e., free movement of the upper limbs and walking in a small-restricted area) is the most suitable method for movement in VR (Bozgeyikli et al., 2016; Frommel et al., 2017; Porcino et al., 2017). Yet, there are additional factors such as external hardware (i.e., controllers and wands), which are needed to facilitate optimal ergonomic interactions in VR.
Ergonomic Interactions
Ergonomic and naturalistic interactions are essential to minimize the risk of VRISE, while non-ergonomic and non-naturalistic interactions increase the occurrence of them (Slater and Wilbur, 1997; Stanney and Hale, 2014; Plouzeau et al., 2015; Caputo et al., 2017; Porcino et al., 2017). Importantly, controllers, joysticks, and keyboards do not support ergonomic and naturalistic interactions in VR (Plouzeau et al., 2015; Bozgeyikli et al., 2016; Caputo et al., 2017; Frommel et al., 2017; Porcino et al., 2017; Sportillo et al., 2017; Figueiredo et al., 2018). Instead, wands with 6 degrees of freedom (DoF) of movement (e.g., Oculus Rift and HTC Vive wands), and realistic interfaces with direct hand interactions (e.g., Microsoft's Kinect) facilitate naturalistic and ergonomic interactions (Sportillo et al., 2017; Figueiredo et al., 2018). Both hardware systems facilitate easy familiarization with their controls and their utilization (Sportillo et al., 2017; Figueiredo et al., 2018). However, direct hand interactions are easier than 6DoF controllers-wands in terms of familiarization with their controls and efficiency (Sportillo et al., 2017; Figueiredo et al., 2018). Direct hand interactions were also found to offer more pleasant user experiences (Sportillo et al., 2017; Figueiredo et al., 2018), although, they are substantially less accurate than 6DoF controller-wands (Sportillo et al., 2017; Figueiredo et al., 2018).
User Experience
Notably, ergonomic interactions might be available to the user; however, the user is required to learn the necessary interactions and how the VE functions to facilitate a pleasant user experience (Gromala et al., 2016; Jerald et al., 2017; Brade et al., 2018). The inclusion of comprehensible tutorials where the user may spend an adequate amount of time acquiring the necessary skills (i.e., navigation, use and grab of items, two-handed interactions) and knowledge of the VE (i.e., how it reacts to your controls) is crucial (Gromala et al., 2016; Jerald et al., 2017; Brade et al., 2018). Additionally, in-game instructions and prompts should be offered to the user through interactions in the VE (e.g., directional arrows, non-player characters, signs, labels, ambient sounds, audio, and videos) (Gromala et al., 2016; Jerald et al., 2017; Brade et al., 2018).
Computer Hardware
The computer hardware (i.e., the processor, graphics card, sound card) should at least meet the minimum requirements of the VR software and HMD (Anthes et al., 2016). The performance of VR HMDs is analogous to the computing power and the quality of the hardware (Stanney and Hale, 2014; Anthes et al., 2016; Borrego et al., 2018). The processor, graphics card, sound card, and operating system (e.g., Windows) need to be considered and reported because they modulate the performance of the software (Plant and Turner, 2009; Plant, 2016; Kane and Parsons, 2017; Parsons et al., 2018). Research software developers and researchers are required to be technologically competent in order to opt for the appropriate hardware and software to achieve their research and/or clinical aims (Plant, 2016; Kane and Parsons, 2017; Parsons et al., 2018).
Conclusions
Based on the outcomes of the above technological review, VR HMDs should have a good quality display-screen (i.e., OLED or upgraded LCD), an adequate FOV (i.e., diagonal FOV ≥ 110°), adequate resolution per eye (i.e., resolution > 960 × 1,080 sub-pixels per eye), and an adequate image refresh rate (i.e., refresh rate ≥ 75 Hz) to safeguard the health and safety of the participants and the reliability of the neuroscientific results (see Table 2). Also, the VR HMD should have external hardware which offers an adequate VR area, fast and accurate motion tracking, spatialized audio, and ergonomic interactions. The computer's processor, graphics card, and sound card should meet the minimum requirements of the VR software and HMD too. New generation VR HMDs appear to have all the necessary hardware characteristics (i.e., graphics, level of immersion, and sound) to be used in ecological valid research and clinical paradigms (Borrego et al., 2018; see Table 2 for a comparison between old and new generation HMDs). New generation VR HMDs have the required hardware to support and produce high-quality spatialized sounds in VEs (Borrego et al., 2018). Additionally, new generation VR HMDs have integrated rapid and precise motion tracking which facilitates naturalistic and ergonomic interactions within the VE (Borrego et al., 2018).
Table 2. Minimum hardware criteria: old and new generation VR HMDs.
Both the Oculus development kit (DK) 1 and DK2 do not meet the minimum hardware features highlighted by the technological review, despite being new generation VR HMDs (see Table 2). The DK1 has substantially lower resolution per eye and image refresh rates, while the DK2 has marginally acceptable refresh rates, yet a slightly lower resolution per eye. These DKs are not available for general use but are used by professional developers to produce beta (early) versions of their games or apps (Goradia et al., 2014; Suznjevic et al., 2017). Moreover, they were removed from the market after the release of the Oculus Rift CV. VR HMDs should have hardware characteristics equal to or better than the commercial versions (CV) of the Oculus Rift and HTC Vive in order to ensure the health and safety of the participants, as well as the reliability of the neuroscientific results (i.e., physiological, neuropsychological, and neuroimaging data). The researchers and clinicians should have the technological competence to choose an HMD which is equal to or greater than the CVs of the Oculus Rift and HTC Vive (e.g., Valve Index, HTC Vive Pro, Oculus Quest, Pimax VR, and StarVR).
However, the VR software's features are equally important. The VR software should include an ergonomic interaction and navigation system, as well as tutorials, in-game instructions, and prompts. A suitable navigation system should combine teleportation and physical movement, while ergonomic interactions should include those that simulate real-life interactions by using a direct hands system or 6DoF controllers. Also, the tutorials, in-game instructions, and prompts should be informative and easy to follow, especially for experimental or clinical purposes where users should be equally able to interact with the VE (Plant and Turner, 2009; Plant, 2016; Kane and Parsons, 2017; Parsons et al., 2018). The criteria for effective VR software are displayed in Table 3. These criteria should be met before implementing VR software for research and/or clinical purposes. Otherwise, researchers or clinicians may compromise the reliability of their study's results (Plant and Turner, 2009; Plant, 2016; Kane and Parsons, 2017; Parsons et al., 2018), and/or jeopardize the health and safety of their participants/patients (Kane and Parsons, 2017;Parsons et al., 2018).
Table 3. Criteria for suitable VR software in cognitive neuroscience and neuropsychology.
The above features enable researchers or clinicians to administer a sophisticated and pleasant VR experience, which substantially alleviates or eradicates adverse VRISE. Therefore, the technological competency of neuroscientists and neuropsychologists is a pre-condition for the efficient adoption and implementation of innovative technologies like VR HMDs in cognitive neuroscience or neuropsychology.
Meta-Analysis of VR Studies in Cognitive Neuroscience
Literature Research and Inclusion Criteria
We followed the PRISMA guidelines to conduct the literature research using a decremental approach, where the selection commenced with a relatively vast accumulation of abstracts and concluded with a diminished list of full papers that comprise standardized and detailed VR research paradigms. The procedure is described in Figure 2. The following databases were used for the literature research: (1) PsycInfo; (2) PsycArticles; (3) PubMed; and (4) Medline. Two categories of keywords were used, with three keywords in each category. The minimum threshold for each study was the inclusion of at least one keyword from each category in the main body of text. The keywords for each category were: (1) “virtual reality;” OR “Immersive;” OR “Head Mounted Display;” AND (2) “Psychology;” OR “Neuropsychology” OR “Neuroscience.” Additional filters and criteria were: (1) chronological specification (2004 and later); and (2) a comprehensive description of the VR research methods in conjunction with the research aims and results. Finally, the selected studies were allocated into two groups according to the generation of the implemented VR HMD. Two tables display the studies that utilize old generation (Table 4) and new generation (Table 5) HMDs.
Figure 2. Decremental stepwise method for the literature review of VR studies.
Table 4. Neuroscience studies employing old generation VR HMDs.
Table 5. Neuroscience studies employing new generation VR HMDs.
Data Collection and Coding
Target Variables
The principal aim of the meta-analysis was to measure the frequency of VRISE in neuroscience or psychology studies using a VR HMD. However, only six studies reported VRISE quantitatively (i.e., using a questionnaire). For this reason, we considered only the presence or absence of VRISE. The dichotomous VRISE variable (i.e., presence or absence of VRISE) was quantified (i.e., absent VRISE = 0; present VRISE = 1) to facilitate a comparison (i.e., Bayesian t-tests) between the studies that used old generation HMDs, new generation DK HMDs, and new generation CV HMDs, as well as the examination of potential correlations with other variables (i.e., Bayesian Pearson's correlation analysis).
secondary aim of the meta-analysis was to inspect the dropout rates in neuroscience or psychology studies that used VR HMDs. However, as the vast majority of studies had no dropouts, studies with some dropouts (e.g., 3, 5, 6) were statistically considered as outliers. For this reason, we considered the existence of dropouts in each study. The dropout variable was dichotomized as presence = 1 and absence = 0. This dichotomous dropout variable was used to investigate whether using a certain generation HMD (i.e., old generation HMDs, new generation DK HMDs, or new generation CV HMDs) could increase/decrease the dropout size. We compared (i.e., Bayesian t-tests) the dropout rate across studies that used old generation HMDs, new generation DK HMDs, and new generation CV HMDs. We also inspected whether the dropout rates correlated with other variables by using Bayesian Pearson's correlation analysis.
Grouping Variables
We subdivided studies into groups based on the HMD generation they used. Hence, two groups of studies were created and compared by using Bayesian t-tests; the first group included studies that utilized old generation HMDs, while the second group included studies which utilized new generation HMDs (i.e., both DKs and CVs).
The new generation studies were further distinguished and compared by using Bayesian t-tests based on the type of new generation HMDs adopted (i.e., DK or CV). Two sub-groups were formed; the first group included studies that utilized DK HMDs, and the second group included studies that utilized CV HMDs.
Furthermore, the recency of the HMD technology was compared by using an ordinal variable where 1 indicated old generation HMDs, 2 indicated new generation DKs, and 3 indicated new generation CVs. This ordinal variable allowed us to inspect whether the HMD generation correlated with other variables by using Bayesian Pearson's correlation analysis.
Lastly, we considered the type of interactions, where the type of interactions were expressed in a binary form (i.e., non-ergonomic interactions = 0 and ergonomic interactions = 1). This allowed a comparison between the VR studies which had ergonomic interactions and the VR studies which had non-ergonomic interactions by using a Bayesian t-test. It also allowed us to inspect whether the interaction type correlated with other variables (i.e., Bayesian Pearson's correlation analysis).
Definition of Ergonomic Interactions
In line with the definition of ergonomic interactions in our technological review, we considered interactions to be ergonomic or non-ergonomic based on their proximity to real-life interactions. We provide some examples below to clarify our criteria:
Example 1—Ergonomic Interaction: if the VR software required the participant to look around moving his or her head.
Example 2—Non-Ergonomic Interaction: if the VR software required the participant to look around by using a joystick or mouse.
Example 3—Ergonomic Interaction: if the VR software required the user to interact with objects (e.g., pushing a button, holding an item) in the VE or to navigate within the VE by using either 6DoF controllers or direct-hand interactions.
Example 4—Non-Ergonomic Interaction: if the VR software required the user to interact with objects (e.g., pushing a button, holding an item) in the VE or to navigate within the VE by using a keyboard or joystick (e.g., Xbox controller).
Statistical Analyses
Bayesian statistics were preferred over null hypothesis significance testing (NHST). The Bayesian factor (BF10) was therefore used instead of p-values for statistical inference, although we do report both BF10 and p-values. P-values measure the difference between the data and the null hypothesis (H0) (e.g., the assumption of no difference or no effect), while the BF10 calibrates p-values by converting them into evidence in favor of the alternative hypothesis (H1) over the H0 (Cox and Donnelly, 2011; Bland, 2015; Held and Ott, 2018). BF10 is considered substantially more parsimonious than the p-value in evaluating the evidence against the H0 (Cox and Donnelly, 2011; Bland, 2015; Held and Ott, 2018). Also, the difference between BF10 and the p-value in evaluating the evidence against H0 is even greater in small sample sizes (Held and Ott, 2018). Bayesian Factor (BF10) threshold ≥ 10 was set for statistical inference in all analyses, which indicates strong evidence in favor of the H1 (Rouder and Morey, 2012; Wetzels and Wagenmakers, 2012; Marsman and Wagenmakers, 2017), and corresponds to a p-value < 0.01 (e.g., BF10 = 10) (Cox and Donnelly, 2011; Bland, 2015; Held and Ott, 2018). JASP software was used to perform the statistical analyses (JASP Team, 2018). Bayesian independent samples t-tests were conducted to investigate the difference in VRISE frequency and dropout occurrence between old and new generation HMDs, as well as between new generation DKs and CVs. A Bayesian Pearson's correlations analysis examined the possible statistical relationships amongst the HMD generations, VRISE presence, the type of interactions, and dropout occurrences.
Results
The Implementation of Old and New Generation HMDs in Cognitive Neuroscience
The studies that utilized old generation HMDs are displayed in Table 4 and recruited 1,200 participants in total. Nine out of 22 studies examined stress disorders, 7 of these were VR exposure therapy (VRET) studies either for phobias or post-traumatic stress disorder (PTSD), while 2 studies attempted to assess stress levels in context (e.g., assessment of social stress during a job interview). In 9 studies, there were VR assessments of cognitive functions, 2 studies assessed memory, 3 studied attention, 3 examined executive functions, and 1 examined visuospatial ability. Two of the studies involved social cognition while only one involved paranoid thinking. Lastly, only one study provided rehabilitation sessions in VR for patients with spinal injuries. The targeted age groups were young adults in 18 studies, middle-aged adults in 8 studies, older adults in one study, and children in one study.
The studies that utilized new generation HMDs are displayed in Table 5 and recruited 982 individuals in total. Specifically, 376 individuals were recruited in 10 studies where new generation DKs were used, while 606 individuals were recruited in 12 studies where new generation CVs were used. Nine out of the 22 studies attempted to assess cognitive functions (i.e., memory, attention, visuospatial ability, executive functions), 4 investigated anxiety disorders (i.e., fear of death, social stress, general anxiety disorder), 3 provided sensorimotor rehabilitation interventions, 3 studies examined the effects of presence in specific VEs, 2 assessed social cognition and 1 study offered a psychoeducational session to patients with motor-related disorders. Lastly, the targeted age groups were young adults in 18 studies, middle-aged adults in 6 studies, and older adults in 4 studies.
Meta-Analysis
The descriptive statistics are presented in Figures 3, 4. In Figure 3, the number of studies with VRISE are displayed according to their HMD generation and interaction type. In Figure 4, the dropouts and sample sizes are presented according to their HMD generation, VRISE presence, and interaction type. The presence of VRISE substantially becomes less frequent when new generation HMDs are implemented (Figure 3). In new generation HMDs, VRISE are present in only 4 out of 22 studies, while across 982 participants, there are only 11 dropouts. In contrast, in old generation HMDs, VRISE are present in 14 out of 22 studies, while in a total sample size of 1,200 participants, there are 58 dropouts.
Figure 3. VRISE per HMD generation and ergonomic interactions. ABSENT, Absence of VRISE; PRESENT, Presence of VRISE; OLD, Old Generation HMD; DK, New Generation HMD—Development Kit; CV, New Generation HMD—Commercial Version; Ergonomic, Ergonomic Interactions; Non-Ergonomic, Non-Ergonomic Interactions.
Figure 4. Droupouts and sample size per HMD, VRISE presence, and ergonomic interactions. ABSENT, Absence of VRISE; PRESENT, Presence of VRISE; OLD, Old Generation HMD; DK, New Generation HMD—Development Kit; CV, New Generation HMD—Commercial Version; Ergonomic, Ergonomic Interactions; Non-Ergonomic, Non-Ergonomic Interactions.
In the 14 old generation HMDs studies where VRISE are present, half of them involved ergonomic interactions and the other half involved non-ergonomic interactions. Similarly, there is an equal distribution of dropouts (29 in each) between the old generation HMDs studies that had ergonomic and non-ergonomic interactions. When only old HMDs with ergonomic interactions are considered, VRISE are present in 7 out of 12 studies, while in an entire sample size of 598, the dropouts are 29. In the studies with new generation DKs, non-ergonomic interactions had an increased presence of VRISE than the ones with ergonomic interactions. Also, in the studies which used DKs with non-ergonomic interactions, 7 participants out of 158 dropped out, while in studies with ergonomic interactions, only one participant out of 218 dropped out. Importantly, when new generation CVs with ergonomic interactions are exclusively considered, there are no VRISE or dropouts in any of the 11 studies with 546 participants. Finally, VRISE were only present in one study using a new generation CV HMD, where 3 participants dropped out. This study was the only one with a new generation CV HMD that did not involve ergonomic interactions.
The Bayesian independent samples t-test highlighted that studies involving new generation VR HMDs have significantly less frequent VRISE (BF10 = 144.68; p < 0.001). The difference in the existence of dropouts was not substantial, yet the studies with new generation HMDs have less frequent dropouts (BF10 = 4.69; p < 0.05) than studies with old HMDs. Notably, the studies which used a new generation CV HMD have significantly less frequent VRISE (BF10 = 46.39; p < 0.001) but not less frequent dropouts (BF10 = 1.66; p = 0.16) than the studies which used a new generation DK HMD. Finally, the studies which implemented VR software with ergonomic interactions had substantially less frequent VRISE (BF10 = 19.54; p < 0.001) and dropouts (BF10 = 16.01; p < 0.001) than studies which used VR software with non-ergonomic interactions.
The Bayesian Pearson's correlations demonstrated a substantial negative correlation between the presence of VRISE and the HMD generation [BF10 = 328.03; r(44) = −0.56, p < 0.001], while the presence of VRISE robustly demonstrated a positive correlation with the existence of dropouts [BF10 = 83510.53; r(44) = 0.68, p < 0.001]. Also, the utilization of ergonomic interactions was significantly negatively correlated with VRISE [BF10 = 20.11; r(44) = −0.42, p < 0.001] and the existence of dropouts [BF10 = 16.11; r(44) = −0.41, p < 0.001].
Discussion
The results of the meta-analysis indicated that VR HMDs have been implemented in diverse clinical conditions and age groups, as well as the unquestionable difference between old generation and new generation HMDs. There were significantly more frequent VRISE in the studies involving old generation VR HMDs compared to studies with new generation HMDs. Additionally, the frequency of VRISE correlated negatively with the HMD generation. Hence, the older the utilized HMD, the higher the VRISE frequency. Moreover, the existence of dropouts significantly and positively correlated with the presence of VRISE.
Nevertheless, one potential reason for the higher dropouts in old generation studies is that several studies included follow-up sessions (e.g., VRET) and participants may have opted not to return to complete the remaining sessions for reasons other than the presence of VRISE. However, in the old generation studies, the dropout rates were low in relation to the size of the population, albeit there were VRISE present. The low dropout rates in the old generation HMD studies may be due to the fixed intervals between the VR sessions where the participants were able to rest and obtain relief from any adverse effects they were experiencing.
Furthermore, the incidence of VRISE in old generation HMD studies may be due to anxiety levels (Bouchard et al., 2011) or be self-induced (Almeida et al., 2017) as several of the studies had either stress-related aims or included participants with stress disorders. However, several of the new generation HMD studies also had comparable aims and/or populations and included patients with clinical conditions (e.g., Alzheimer's disease, Parkinson's disease, stroke, and movement disorders) which have high comorbidity with stress and anxiety (Factor et al., 1995; Smith et al., 2000; Jenner, 2003; Allen and Bayraktutan, 2009). Also, the rates of self-induced VRISE are expected to be equal in both new and old generation HMD studies. In addition, the reporting of VRISE may be for reasons unrelated to the quality of the hardware or software (e.g., subjectivity in the reporting of VRISE, individual differences in the experience of VRISE) (Kortum and Peres, 2014; Almeida et al., 2017). However, this modulation is again expected to have affected both new and old generation HMD studies in a similar way.
Beyond the difference between old and new generations HMDs, a substantial difference is observed between DK and CV new generation HMDs. Significantly fewer VRISE were present in the studies that used a CV, indicating the superiority of new generation CV HMDs compared to new generation DK HMDs. Furthermore, the studies (i.e., both old and new generation studies) which utilized VR software with ergonomic interactions had robustly less frequent VRISE and dropouts than the studies which implemented VR software with non-ergonomic interactions. However, the ergonomic interactions do not appear to mitigate the dropout frequency and the incidence of VRISE in old generation HMDs. In contrast, VRISE were present in more DK studies with non-ergonomic interactions compared to DK studies with ergonomic interactions. Similarly, more participants dropped out from DK studies with non-ergonomic interactions. Notably, there were no VRISE or dropouts in CV studies with ergonomic interactions. Therefore, the contribution of ergonomic interactions in the reduction of VRISE increases when newer and better HMDs are utilized. To conclude, the findings of the meta-analysis are aligned with the outcomes of the technological review.
General Discussion
Technological Competence in VR Neuroscience and Neuropsychology
The findings of our technological literature review suggest that the hardware features of old generation HMDs and new generation DKs do not meet the minimum hardware features that alleviate or eradicate VRISE. Instead, the technological literature review postulates the suitability of new generation CVs which have specific hardware capabilities to alleviate VRISE. However, VR software attributes (e.g., ergonomic interactions) are equally vital.
Secondly, the findings of our meta-analysis of 44 neuroscientific or neuropsychological studies using VR are aligned with the outcomes of our technological review, where VRISE were substantially less frequent in studies which utilized new generation VR HMDs. In particular, the studies which used new generation CVs accompanied by ergonomic interactions did not have any VRISE or dropouts. Therefore, the combined outcomes of the technological review and the meta-analysis indicate that the appropriate VR HMDs are those with hardware characteristics equal to or greater than the HTC Vive and Oculus Rift, though the VR HMD should be implemented in conjunction with VR software which offers ergonomic interactions.
However, researchers may have to opt for an HMD based on their available budget. For example, the Oculus Rift costs around $400, while the HTC Vive costs around $500. Moreover, the majority of HMDs also require a VR-ready desktop PC or a laptop to be operated, so a researcher needs to additionally spend around $500–$1,500 for a desktop computer or laptop to utilize these HMDs. Hence, the combined cost would be between $800 and $1,900. The cost of VR equipment (e.g., both HMD and computer) may lead researchers to use HMDs that are cheaper, albeit that they are more likely to result in VRISE. However, in the market, there are plenty of cost-effective alternatives that meet the minimum hardware criteria. For example, the Oculus Quest is a standalone HMD (i.e., it does not require a PC, a laptop, or a smartphone to be operated) and it costs approximately $400. Hence, a researcher can spend the equivalent of the price of a neuropsychological test or a smartphone to acquire and use an HMD that meets the minimum hardware criteria to lower the presence of VRISE.
Nonetheless, the selection of an appropriate VR HMD and software requires technological competency from the researchers, clinicians, and/or research software developers. Unfortunately, the meta-analysis results do not indicate that technological knowledge of VR has been well-established in neuroscience. Of course, the utilization of old generation HMDs and new generation DKs pre-2016 is justified as the new generation CVs were not available. However, in our meta-analysis, 25 studies were conducted between 2016 and 2019, where half of these studies (13/25) implemented an inappropriate HMD (i.e., old generation HMD or new generation DK). However, 10 studies used a DK2 which has a marginally lower resolution than the minimum hardware criteria, while our meta-analysis results indicated that its utilization in conjunction with ergonomic interactions appears to alleviate the frequency of VRISE, but not as effectively as the CV HMDs. Furthermore, one fifth of the studies did use a new generation HMD, but they did not have ergonomic interactions in their VR software. Therefore, at this time, VR technological competence does not seem to have been well-established in neuroscience. As a result, in the studies since 2016, the health and safety of the participants may not substantially guaranteed, and the reliability of the results may be questionable, as VRISE substantially decreases reaction times and overall cognitive performance (Plant and Turner, 2009; Nalivaiko et al., 2015; Plant, 2016; Nesbitt et al., 2017; Mittelstaedt et al., 2018), as well as confounding neuroimaging and physiological data (Toschi et al., 2017; Arafat et al., 2018; Gavgani et al., 2018). The selection of an appropriate HMD is paramount for successfully implementing VR HMDs in cognitive neuroscience and neuropsychology.
However, the implementation of the currently available and appropriate HMDs in neuroscience and neuropsychology should be compatible with the research aims. For example, in research designs where the user should be active (i.e., navigating, walking, and interacting within the VE) instead of being idle, or in a standing or a seated position, the researcher should opt for the best HMD that permits intense body movement and activity. In this setting, the Oculus Rift was found to be inferior to the HTC Vive on pick-and-place (i.e., relocating objects) tasks, whilst the HTC Vive also provided a substantially superior VR experience for users compared to the Oculus Rift (Suznjevic et al., 2017). Moreover, the HTC Vive provides an interactive area that is twice the size (25 m2) of the Oculus Rift, albeit that both are very accurate in tracking (Borrego et al., 2018). Nevertheless, the HTC Vive was found to lose motion-tracking and the ground level becomes slanted when the user goes out of bounds (Niehorster et al., 2017). This shortcoming solely affects studies where the participant needs to go out of the tracking area. In most neuroscientific designs, the recommended maximum play area by HTC (6.25 m2) or by Borrego et al. (2018) (25 m2) are both substantially adequate for conducting ecological valid experiments (Borges et al., 2018; Borrego et al., 2018). Nonetheless, the slanted floor or lost tracking is not a hardware problem but a software one (Borges et al., 2018). In cases where the participant is required to go out of the tracking area, the tracking problem or the slanted floor may be easily corrected by adding 3 additional trackers (Peer et al., 2018), using software with an improved algorithm (freely distributed by NASA Ames Research Center) (Borges et al., 2018), or by simply updating the firmware of the lighthouse base stations. In summary, the researchers should be technologically competent to not only identify and implement a safe HMD and software, but an HMD and software that facilitate the optimal research methods pertinent to their research needs and aims.
As discussed in our technological review, the quality of the implemented VR software is equally important to avoid VRISE. Our meta-analysis of VR studies indicated that the utilization of ergonomic interactions is crucial albeit with the utilization of an appropriate HMD. For example, Detez et al. (2019) used the HTC Vive to investigate physiological arousal and behavior during gambling. However, the interactions and navigation within the VE were facilitated by using a typical controller (Detez et al., 2019). Hence, their VR software did not support the utilization of the ergonomic 6DoF controllers (both hands) of the HTC Vive, which facilitate naturalistic navigation (e.g., teleportation) and interaction within the VE. Consequently, Detez et al.'s (2019) participants experienced VRISE and 3 of their participants discontinued their sessions and so their data were discarded (Detez et al., 2019). Importantly, Detez et al. (2019) only reported the presence of VRISE and dropout size. They did not provide any quantitative data on the intensity of VRISE, or the quality of their software attributes (e.g., graphics, sound, tutorials, in-game instructions and prompts) (Detez et al., 2019). Indeed, only six of the studies in the meta-analysis provided adequately explicit reports on VRISE and VR software. Since Detez et al. (2019) assessed reaction times and heart rates, these data are likely to be affected by VRISE, despite the study having a rigorous experimental design and using the HTC Vive. Therefore, it is important to use appropriate VR software and external hardware to prevent risks to the health and safety of the participants as well as the reliability of the results.
Limitations and Future Studies
The above technological review and meta-analysis of VR studies evidenced the importance of technological and methodological features in VR research and clinical designs. However, our meta-analysis of VR studies has some limitations. The meta-analysis considered VR studies with diverse populations and designs, which may have affected the frequency of VRISE and the existence of dropouts. Uniformity across studies (e.g., considering only VRET, assessments or a specific clinical population) was not possible due to the scarcity of neuroscience studies involving VR, especially using new generation HMDs. Moreover, the review did not consider any software details due to the scarceness of such descriptions in published studies. Future VR studies should report software and hardware features to allow an in-depth meta-analysis. Equally, only six studies provided quantitative reports of VRISE intensity, consequently, only the presence or absence of VRISE was considered. The dichotomous consideration of VRISE is susceptible to reports based on subjective criteria and individual differences, but this is likely to have affected the VRISE rates in both old and new generation studies. Future studies should aim to appraise the quality of the software and intensity of VRISE (e.g., using questionnaires). Studies should also attempt to clarify the acceptable duration of immersive VR sessions, which will aid researchers in designing their studies appropriately. Importantly, the cost of the VR software development should also be considered. Finally, studies should attempt to provide software development guidelines that enable researchers and/or research software developers to develop VR research software without depending on third parties (e.g., freelance developers or software development companies) and these guidelines should embed suggestions and instructions for VR software development, which meet the criteria discussed above.
Conclusion
The use of VR HMDs is becoming more popular in neuroscience either for clinical or research purposes and VR technology and methods have been well-accepted by diverse populations in terms of age groups and clinical conditions. A more pleasant VR experience and a reduction in VRISE symptomatology has been found using new generation CV HMDs, which deliver an adequately high display resolution, rapid image refresh rate, ergonomic design and has controllers which allow naturalistic navigation and movement within the VE environment, especially when there is restricted teleportation. The outcomes of the current technological review and meta-analysis support the feasibility of new generation VR CV HMDs to be implemented in cognitive neuroscience and neuropsychology. The findings of the technological review suggest methods that should be considered in the development or selection of VR research software, as well as hardware and software features that should be included in the research protocol. The selected VR HMD and the VR research software should enable suitable ergonomic interactions, locomotion techniques (e.g., teleportation), and kinetic mechanics which ensure VRISE are reduced or completely avoided. A meticulous approach and technological competence are compulsory to consolidate the viability of VR research and clinical designs in cognitive neuroscience and neuropsychology.
Data Availability Statement
All datasets generated for this study are included in the manuscript/supplementary files.
Author Contributions
PK had the initial idea and contributed to every aspect of this study. SC, LD, and SM contributed to the methodological aspects and the discussion of the results.
Conflict of Interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
References
Allen, C. L., and Bayraktutan, U. (2009). Oxidative stress and its role in the pathogenesis of ischaemic stroke. Int. J. Stroke 4, 461–470. doi: 10.1111/j.1747-4949.2009.00387.x
Almeida, A., Rebelo, F., Noriega, P., and Vilar, E. (2017). “Virtual reality self induced cybersickness: an exploratory study,” in International Conference on Applied Human Factors and Ergonomics (Cham: Springer), 26–33. doi: 10.1007/978-3-319-60582-1_3
Anthes, C., García-Hernández, R. J., Wiedemann, M., and Kranzlmüller, D. (2016). “State of the art of virtual reality technology,” in 2016 IEEE Aerospace Conference (Big Sky, MT), 1–19. doi: 10.1109/AERO.2016.7500674
Arafat, I. M., Ferdous, S. M. S., and Quarles, J. (2018). “Cybersickness-provoking virtual reality alters brain signals of persons with multiple sclerosis.” in 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR) (Reutlingen: IEEE), 1–120. doi: 10.1109/VR.2018.8446194
Banakou, D., Kishore, S., and Slater, M. (2018). Virtually being Einstein results in an improvement in cognitive Task performance and a decrease in age bias. Front. Psychol. 9:917. doi: 10.3389/fpsyg.2018.00917
Banville, F., Nolin, P., Lalonde, S., Henry, M., Dery, M. P., and Villemure, R. (2010). Multitasking and prospective memory: can virtual reality be useful for diagnosis? Behav. Neurol. 23, 209–211. doi: 10.1155/2010/320707
Barberia, I., Oliva, R., Bourdin, P., and Slater, M. (2018). Virtual mortality and near-death experience after a prolonged exposure in a shared virtual reality may lead to positive life-attitude changes. PLoS ONE 13:e0203358. doi: 10.1371/journal.pone.0203358
Bioulac, S., Lallemand, S., Rizzo, A., Philip, P., Fabrigoule, C., and Bouvard, M. P. (2012). Impact of time on task on ADHD patient's performances in a virtual classroom. Eur. J. Paediatr. Neurol. 16, 514–521. doi: 10.1016/j.ejpn.2012.01.006
Bohil, C. J., Alicea, B., and Biocca, F. A. (2011). Virtual reality in neuroscience research and therapy. Nat. Rev. Neurosci. 12:752. doi: 10.1038/nrn3122
Borges, M., Symington, A., Coltin, B., Smith, T., and Ventura, R. (2018). “HTC Vive: analysis and accuracy improvement,” in 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (Madrid: IEEE), 2610–2615. doi: 10.1109/IROS.2018.8593707
Borrego, A., Latorre, J., Alcaniz, M., and Llorens, R. (2018). Comparison of oculus rift and HTC vive: feasibility for virtual reality-based exploration, navigation, exergaming, and rehabilitation. Games Health J. 7, 151–156. doi: 10.1089/g4h.2017.0114
Botella, C., García-Palacios, A., Villa, H., Baños, R. M., Quero, S., Alcañiz, M., et al. (2007). Virtual reality exposure in the treatment of panic disorder and agoraphobia: a controlled study. Clin. Psychol. Psychother. 14, 164–175. doi: 10.1002/cpp.524
Bouchard, S., Robillard, G., Renaud, P., and Bernier, F. (2011). Exploring new dimensions in the assessment of virtual reality induced side effects. J. Comput. Inform. Technol. 1, 20–32. Available online at: http://pubs.drdc-rddc.gc.ca/BASIS/pcandid/www/engpub/DDW?W%3DSYSNUM=536530&r=0
Bourdin, P., Barberia, I., Oliva, R., and Slater, M. (2017). A virtual out-of-body experience reduces fear of death. PLoS ONE 12:e0169343. doi: 10.1371/journal.pone.0169343
Bozgeyikli, E., Raij, A., Katkoori, S., and Dubey, R. (2016). “Point and teleport locomotion technique for virtual reality,” in Proceedings of the 2016 Annual Symposium on Computer-Human Interaction in Play (Austin, TX: ACM), 205–216. doi: 10.1145/2967934.2968105
Brade, J., Dudczig, M., and Klimant, P. (2018). “Using virtual prototyping technologies to evaluate human-machine-interaction concepts,” in AW&I Conference (Chemnitz), 3.
Brennesholtz, M. S. (2018). “3-1: Invited Paper: VR standards and guidelines,” in SID Symposium Digest of Technical Papers, 49 (Pleasantville, NY), 1–4. doi: 10.1002/sdtp.12476
Caputo, F., Greco, A., Egidio, D. A., Notaro, I., and Spada, S. (2017). “A preventive ergonomic approach based on virtual and immersive reality,” in International Conference on Applied Human Factors and Ergonomics (Cham: Springer), 3–15. doi: 10.1007/978-3-319-60582-1_1
Carlozzi, N. E., Gade, V., Rizzo, A. S., and Tulsky, D. S. (2013). Using virtual reality driving simulators in persons with spinal cord injury: three screen display versus head mounted display. Disabil. Rehabil. 8, 176–180. doi: 10.3109/17483107.2012.699990
Christou, C. G., Michael-Grigoriou, D., Sokratous, D., and Tsiakoulia, M. (2018). “A virtual reality loop and wire game for stroke rehabilitation,” in 28th International Conference on Artificial Reality and 23rd Telexistence and Eurographics Symposium on Virtual Environments (Limassol).
Collins, M. K., Ding, V. Y., Ball, R. L., Dolce, D. L., Henderson, J. M., and Halpern, C. H. (2018). Novel application of virtual reality in patient engagement for deep brain stimulation: a pilot study. Brain Stimul. 11, 935–937. doi: 10.1016/j.brs.2018.03.012
Costello, P. J. (1997). Health and Safety Issues Associated With Virtual Reality: A Review of Current Literature. Loughborough: Advisory Group on Computer Graphics.
Cox, D. R., and Donnelly, C. A. (2011). Principles of Applied Statistics. Cambridge: Cambridge University Press
Davis, S., Nesbitt, K., and Nalivaiko, E. (2015). “Comparing the onset of cybersickness using the Oculus Rift and two virtual roller coasters,” in Proceedings of the 11th Australasian Conference on Interactive Entertainment (IE 2015), 27 (Sydney), 30.
de França, A. C. P., and Soares, M. M. (2017). “Review of virtual reality technology: an ergonomic approach and current challenges,” in International Conference on Applied Human Factors and Ergonomics (Cham: Springer), 52–61. doi: 10.1007/978-3-319-60582-1_6
Detez, L., Greenwood, L. M., Segrave, R., Wilson, E., Chandler, T., Ries, T., et al. (2019). A psychophysiological and behavioural study of slot machine near-misses using immersive virtual reality. J. Gambl. Stud. 35, 929–944. doi: 10.1007/s10899-018-09822-z
Factor, S. A., Podskalny, G. D., and Molho, E. S. (1995). Psychogenic movement disorders: frequency, clinical profile, and characteristics. J. Neurol. Neurosurg. Psychiatry 59, 406–412. doi: 10.1136/jnnp.59.4.406
Figueiredo, L., Rodrigues, E., Teixeira, J., and Techrieb, V. (2018). A comparative evaluation of direct hand and wand interactions on consumer devices. Comput. Graph. 77, 108–121. doi: 10.1016/j.cag.2018.10.006
Foerster, R. M., Poth, C. H., Behler, C., Botsch, M., and Schneider, W. X. (2016). Using the virtual reality device Oculus Rift for neuropsychological assessment of visual processing capabilities. Sci. Rep. 6:37016. doi: 10.1038/srep37016
Freeman, D., Bradley, J., Antley, A., Bourke, E., DeWeever, N., Evans, N., et al. (2016). Virtual reality in the treatment of persecutory delusions: randomised controlled experimental study testing how to reduce delusional conviction. Br. J. Psychiatry 209, 62–67. doi: 10.1192/bjp.bp.115.176438
Freeman, D., Evans, N., Lister, R., Antley, A., Dunn, G., and Slater, M. (2014). Height, social comparison, and paranoia: an immersive virtual reality experimental study. Psychiatry Res. 218, 348–352. doi: 10.1016/j.psychres.2013.12.014
Frommel, J., Sonntag, S., and Weber, M. (2017). “Effects of controller-based locomotion on player experience in a virtual reality exploration game,” in Proceedings of the 12th International Conference on the Foundations of Digital Games (Hyannis, MA: ACM), 30. doi: 10.1145/3102071.3102082
Gaggioli, A., Pallavicini, F., Morganti, L., Serino, S., Scaratti, C., Briguglio, M., et al. (2014). Experiential virtual scenarios with real-time monitoring (interreality) for the management of psychological stress: a block randomized controlled trial. J. Med. Internet Res. 16:e167. doi: 10.2196/jmir.3235
Gavgani, A. M., Wong, R. H., Howe, P. R., Hodgson, D. M., Walker, F. R., and Nalivaiko, E. (2018). Cybersickness-related changes in brain hemodynamics: a pilot study comparing transcranial Doppler and near-infrared spectroscopy assessments during a virtual ride on a roller coaster. Physiol. Behav. 191, 56–64. doi: 10.1016/j.physbeh.2018.04.007
George, C., Demmler, M., and Hussmann, H. (2018). “Intelligent interruptions for IVR: investigating the interplay between presence, workload and attention,” in Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal, QC: ACM), LBW511. doi: 10.1145/3170427.3188686
Gómez-Jordana, L. I., Stafford, J., Peper, C. L. E., and Craig, C. M. (2018). Virtual footprints can improve walking performance in people with Parkinson's disease. Front. Neurol. 9:681. doi: 10.3389/fneur.2018.00681
Goradia, I., Doshi, J., and Kurup, L. (2014). A review paper on oculus rift and project morpheus. Int. J. Curr. Eng. Technol. 4, 3196–3200. Available online at: http://inpressco.com/wp-content/uploads/2014/09/Paper183196-3200.pdf
Gromala, D., Tong, X., Shaw, C., Amin, A., Ulas, S., and Ramsay, G. (2016). Mobius floe: an immersive virtual reality game for pain distraction. Electron. Imaging 2016, 1–5. doi: 10.2352/ISSN.2470-1173.2016.4.ERVR-413
Hartanto, D., Kampmann, I. L., Morina, N., Emmelkamp, P. G., Neerincx, M. A., and Brinkman, W. P. (2014). Controlling social stress in virtual reality environments. PLoS ONE 9:e92804. doi: 10.1371/journal.pone.0092804
Hasler, B. S., Spanlang, B., and Slater, M. (2017). Virtual race transformation reverses racial in-group bias. PLoS ONE 12:e0174965. doi: 10.1371/journal.pone.0174965
Hecht, J. (2016). Optical dreams, virtual reality. Optics Photon. News 27, 24–31. doi: 10.1364/OPN.27.6.000024
Held, L., and Ott, M. (2018). On p-values and Bayes factors. Annu. Rev. Stat. Appl. 5, 393–419. doi: 10.1146/annurev-statistics-031017-100307
Hsieh, T. J. T., Kuo, Y. H., and Niu, C. K. (2018). “Utilizing HMD VR to improve the spatial learning and wayfinding effects in the virtual maze,” in International Conference on Human-Computer Interaction (Cham: Springer), 38–42. doi: 10.1007/978-3-319-92285-0_6
Jenner, P. (2003). Oxidative stress in Parkinson's disease. Ann. Neurol. 53, S26–S38. doi: 10.1002/ana.10483
Jerald, J., LaViola, J. J. Jr., and Marks, R. (2017). “VR interactions,” in ACM SIGGRAPH 2017 Courses (Los Angeles, CA: ACM), 19. doi: 10.1145/3084873.3084900
Kane, R. L., and Parsons, T. D. (eds.). (2017). The Role of Technology in Clinical Neuropsychology. New York, NY: Oxford University Press.
Kelly, J. W., Cherep, L. A., and Siegel, Z. D. (2017). Perceived space in the HTC vive. ACM Trans. Appl. Percept. 15:2. doi: 10.1145/3106155
Kennedy, R. S., Lane, N. E., Berbaum, K. S., and Lilienthal, M. G. (1993). Simulator sickness questionnaire: an enhanced method for quantifying simulator sickness. Int. J. Aviation Psychol. 3, 203–220. doi: 10.1207/s15327108ijap0303_3
Kim, A., Darakjian, N., and Finley, J. M. (2017). Walking in fully immersive virtual environments: an evaluation of potential adverse effects in older adults and individuals with Parkinson's disease. J. Neuroeng. Rehabil. 14:16. doi: 10.1186/s12984-017-0225-2
Kim, H. K., Park, J., Choi, Y., and Choe, M. (2018). Virtual reality sickness questionnaire (VRSQ): motion sickness measurement index in a virtual reality environment. Appl. Ergon. 69, 66–73. doi: 10.1016/j.apergo.2017.12.016
Kim, J. W., Choe, W. J., Hwang, K. H., and Kwag, J. O. (2017). “78-2: the optimum display for virtual reality,” in SID Symposium Digest of Technical Papers, 48, 1146–1149. doi: 10.1002/sdtp.11845
Kim, K., Kim, J., Ku, J., Kim, D. Y., Chang, W. H., Shin, D. I., et al. (2004). A virtual reality assessment and training system for unilateral neglect. Cyberpsychol. Behav. 7, 742–749. doi: 10.1089/cpb.2004.7.742
Kortum, P., and Peres, S. C. (2014). The relationship between system effectiveness and subjective usability scores using the System Usability Scale. Int. J. Human Comput. Int. 30, 575–584. doi: 10.1080/10447318.2014.904177
Lubetzky, A. V., Kary, E. E., Harel, D., Hujsak, B., and Perlin, K. (2018). Feasibility and reliability of a virtual reality oculus platform to measure sensory integration for postural control in young adults. Physiother. Theory Pract. 34, 935–950. doi: 10.1080/09593985.2018.1431344
Marsman, M., and Wagenmakers, E. J. (2017). Bayesian benefits with JASP. Eur. J. Dev. Psychol. 14, 545–555. doi: 10.1080/17405629.2016.1259614
Matheis, R. J., Schultheis, M. T., Tiersky, L. A., DeLuca, J., Millis, S. R., and Rizzo, A. (2007). Is learning and memory different in a virtual environment? Clin. Neuropsychol. 21, 146–161. doi: 10.1080/13854040601100668
Meyerbroeker, K., Morina, N., Kerkhof, G. A., and Emmelkamp, P. M. G. (2013). Virtual reality exposure therapy does not provide any additional value in agoraphobic patients: a randomized controlled trial. Psychother. Psychosom. 82, 170–176. doi: 10.1159/000342715
Mittelstaedt, J. M., Wacker, J., and Stelling, D. (2018). VR aftereffect and the relation of cybersickness and cognitive performance. Virtual Real. 23, 143–154. doi: 10.1007/s10055-018-0370-3
Montenegro, J. M. F., and Argyriou, V. (2017). Cognitive evaluation for the diagnosis of Alzheimer's disease based on turing test and virtual environments. Physiol. Behav. 173, 42–51. doi: 10.1016/j.physbeh.2017.01.034
Moreau, G., Guay, M. C., Achim, A., Rizzo, A., and Lageix, P. (2006). The virtual classroom: an ecological version of the continuous performance test–A pilot study. Annu. Rev. CyberTherapy Telemed. 4, 59–66. doi: 10.1037/e695432011-008
Mottelson, A., and Hornbæk, K. (2017). “Virtual reality studies outside the laboratory,” in Proceedings of the 23rd ACM Symposium on Virtual Reality Software and Technology (Gothenburg: ACM), 9. doi: 10.1145/3139131.3139141
Nalivaiko, E., Davis, S. L., Blackmore, K. L., Vakulin, A., and Nesbitt, K. V. (2015). Cybersickness provoked by head-mounted display affects cutaneous vascular tone, heart rate and reaction time. Physiol. Behav. 151, 583–590. doi: 10.1016/j.physbeh.2015.08.043
Nesbitt, K., Davis, S., Blackmore, K., and Nalivaiko, E. (2017). Correlating reaction time and nausea measures with traditional measures of cybersickness. Displays 48, 1–8. doi: 10.1016/j.displa.2017.01.002
Niehorster, D. C., Li, L., and Lappe, M. (2017). The accuracy and precision of position and orientation tracking in the HTC vive virtual reality system for scientific research. i-Perception 8:2041669517708205. doi: 10.1177/2041669517708205
Oagaz, H., Schoun, B., Pooji, M., and Choi, M. H. (2018). Neurocognitive assessment in virtual reality through behavioral response analysis. IEEE J. Biomed. Health Inform. 23, 1899–1910. doi: 10.1109/JBHI.2018.2881455
Palmisano, S., Mursic, R., and Kim, J. (2017). Vection and cybersickness generated by head-and-display motion in the Oculus Rift. Displays 46, 1–8. doi: 10.1016/j.displa.2016.11.001
Parsons, T. D. (2015). Virtual reality for enhanced ecological validity and experimental control in the clinical, affective and social neurosciences. Front. Human Neurosci. 9:660. doi: 10.3389/fnhum.2015.00660
Parsons, T. D., and Barnett, M. (2017). Virtual Apartment-Based Stroop for assessing distractor inhibition in healthy aging. Appl. Neuropsychol. Adult 26, 144–154. doi: 10.1080/23279095.2017.1373281
Parsons, T. D., and Carlew, A. R. (2016). Bimodal virtual reality stroop for assessing distractor inhibition in autism spectrum disorders. J. Autism Dev. Disord. 46, 1255–1267. doi: 10.1007/s10803-015-2663-7
Parsons, T. D., Courtney, C. G., and Dawson, M. E. (2013). Virtual reality Stroop task for assessment of supervisory attentional processing. J. Clin. Exp. Neuropsychol. 35, 812–826. doi: 10.1080/13803395.2013.824556
Parsons, T. D., and McMahan, T. (2017). An initial validation of the virtual of the virtual environment grocery store. J. Neurosci. Methods 291, 13–19. doi: 10.1016/j.jneumeth.2017.07.027
Parsons, T. D., McMahan, T., and Kane, R. (2018). Practice parameters facilitating adoption of advanced technologies for enhancing neuropsychological assessment paradigms. Clin. Neuropsychol. 32, 16–41. doi: 10.1080/13854046.2017.1337932
Parsons, T. D., Rizzo, A. A., Bamattre, J., and Brennan, J. (2007). Virtual reality cognitive performance assessment test. Annu. Rev. CyberTherapy Telemed. 5, 163–171. doi: 10.1037/e688982007-001
Peck, T. C., Seinfeld, S., Aglioti, S. M., and Slater, M. (2013). Putting yourself in the skin of a black avatar reduces implicit racial bias. Conscious. Cogn. 22, 779–787. doi: 10.1016/j.concog.2013.04.016
Peer, A., Ullich, P., and Ponto, K. (2018). “Vive tracking alignment and correction made easy,” in 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR) (Reutlingen: IEEE), 653–654. doi: 10.1109/VR.2018.8446435
Plant, R. R. (2016). A reminder on millisecond timing accuracy and potential replication failure in computer-based psychology experiments: an open letter. Behav. Res. Methods 48, 408–411. doi: 10.3758/s13428-015-0577-0
Plant, R. R., and Turner, G. (2009). Millisecond precision psychological research in a world of commodity computers: new hardware, new problems? Behav. Res. Methods 41, 598–614. doi: 10.3758/BRM.41.3.598
Plouzeau, J., Paillot, D., Chardonnet, J. R., and Merienne, F. (2015). “Effect of proprioceptive vibrations on simulator sickness during navigation task in virtual environment,” in International Conference on Artificial Reality and Telexistence (Kyoto).
Porcino, T. M., Clua, E., Trevisan, D., Vasconcelos, C. N., and Valente, L. (2017). “Minimizing cyber sickness in head mounted display systems: design guidelines and applications,” in 2017 IEEE 5th International Conference on Serious Games and Applications for Health (SeGAH) (Perth, WA: IEEE), 1–6. doi: 10.1109/SeGAH.2017.7939283
Quinlivan, B., Butler, J. S., Beiser, I., Williams, L., McGovern, E., O'Riordan, S., et al. (2016). Application of virtual reality head mounted display for investigation of movement: a novel effect of orientation of attention. J. Neural Eng. 13:056006. doi: 10.1088/1741-2560/13/5/056006
Rakkolainen, I., Turk, M., and Höllerer, T. (2016). “A superwide-FOV optical design for head-mounted displays,” in Proceedings of the Joint 26th International Conference on Artificial Reality and Telexistence and the 21th Eurographics Symp. on Virtual Environments (ICAT-EGVE 2016) (Little Rock, AR), 45–48.
Reger, G. M., Holloway, K. M., Candy, C., Rothbaum, B. O., Difede, J., Rizzo, A. A., et al. (2011). Effectiveness of virtual reality exposure therapy for active duty soldiers in a military mental health clinic. J. Traumat. Stress 24, 93–96. doi: 10.1002/jts.20574
Rizzo, A., Difede, J., Rothbaum, B. O., Reger, G., Spitalnick, J., Cukor, J., et al. (2010). Development and early evaluation of the Virtual Iraq/Afghanistan exposure therapy system for combat-related PTSD. Ann. N. Y. Acad. Sci. 1208, 114–125. doi: 10.1111/j.1749-6632.2010.05755.x
Rizzo, A. A., Schultheis, M., Kerns, K. A., and Mateer, C. (2004). Analysis of assets for virtual reality applications in neuropsychology. Neuropsychol. Rehabil. 14, 207–239. doi: 10.1080/09602010343000183
Rooney, B., Burke, C., Bálint, K., O'Leary, T., Parsons, T., Lee, C. T., et al. (2017). “Virtual reality, presence and social cognition: the effect of eye-gaze and narrativity on character engagement,” in 2017 23rd International Conference on Virtual System and Multimedia (VSMM) (Dublin: IEEE), 1–6. doi: 10.1109/VSMM.2017.8346272
Rothbaum, B. O., Price, M., Jovanovic, T., Norrholm, S. D., Gerardi, M., Dunlop, B., et al. (2014). A randomized, double-blind evaluation of D-cycloserine or alprazolam combined with virtual reality exposure therapy for posttraumatic stress disorder in Iraq and Afghanistan War veterans. Am. J. Psychiatry 171, 640–648. doi: 10.1176/appi.ajp.2014.13121625
Rouder, J. N., and Morey, R. D. (2012). Default Bayes factors for model selection in regression. Multivariate Behav. Res. 47, 877–903. doi: 10.1080/00273171.2012.734737
Sharples, S., Cobb, S., Moody, A., and Wilson, J. R. (2008). Virtual reality induced symptoms and effects (VRISE): comparison of head mounted display (HMD), desktop and projection display systems. Displays 29, 58–69. doi: 10.1016/j.displa.2007.09.005
Shiban, Y., Schelhorn, I., Pauli, P., and Mühlberger, A. (2015). Effect of combined multiple contexts and multiple stimuli exposure in spider phobia: a randomized clinical trial in virtual reality. Behav. Res. Therapy 71, 45–53. doi: 10.1016/j.brat.2015.05.014
Slater, M., and Wilbur, S. (1997). A framework for immersive virtual environments (FIVE): speculations on the role of presence in virtual environments. Presence 6, 603–616. doi: 10.1162/pres.1997.6.6.603
Smith, M. A., Rottkamp, C. A., Nunomura, A., Raina, A. K., and Perry, G. (2000). Oxidative stress in Alzheimer's disease. Biochim. Biophys. Acta 1502, 139–144. doi: 10.1016/S0925-4439(00)00040-5
Sportillo, D., Paljic, A., Boukhris, M., Fuchs, P., Ojeda, L., and Roussarie, V. (2017). “An immersive Virtual Reality system for semi-autonomous driving simulation: a comparison between realistic and 6-DoF controller-based interaction,” in Proceedings of the 9th International Conference on Computer and Automation Engineering (Sydney: ACM), 6–10. doi: 10.1145/3057039.3057079
Stanney, K. M., and Hale, K. S. (2014). Handbook of Virtual Environments: Design, Implementation, and Applications. Boca Raton, FL: CRC Press. doi: 10.1201/b17360
Suznjevic, M., Mandurov, M., and Matijasevic, M. (2017). “Performance and QoE assessment of HTC vive and oculus rift for pick-and-place tasks in VR,” in 2017 Ninth International Conference on Quality of Multimedia Experience (QoMEX) (Erfurt: IEEE), 1–3. doi: 10.1109/QoMEX.2017.7965679
Toschi, N., Kim, J., Sclocco, R., Duggento, A., Barbieri, R., Kuo, B., et al. (2017). Motion sickness increases functional connectivity between visual motion and nausea-associated brain regions. Auton. Neurosci. 202, 108–113. doi: 10.1016/j.autneu.2016.10.003
Veling, W., Brinkman, W. P., Dorrestijn, E., and Van Der Gaag, M. (2014). Virtual reality experiments linking social environment and psychosis: a pilot study. Cyberpsychol. Behav. Social Netw. 17, 191–195. doi: 10.1089/cyber.2012.0497
Viirre, E., Price, B. J., and Chase, B. (2014). “Direct effects of virtual environments on users,” in Handbook of Virtual Environments, eds K. S. Hale and K. M. Stanney (Boca Raton, FL: CRC Press), 527–535. doi: 10.1201/b17360-28
Vorländer, M., and Shinn-Cunningham, B. (2014). “Virtual auditory displays,” in Handbook of Virtual Environments, eds K. S. Hale and K. M. Stanney (Boca Raton, FL: CRC Press), 107–134. doi: 10.1201/b17360-7
Wetzels, R., and Wagenmakers, E. J. (2012). A default Bayesian hypothesis test for correlations and partial correlations. Psychon. Bull. Rev. 19, 1057–1064. doi: 10.3758/s13423-012-0295-x
Yeh, S. C., Li, Y. Y., Zhou, C., Chiu, P. H., and Chen, J. W. (2018). Effects of virtual reality and augmented reality on induced anxiety. IEEE Trans. Neural Syst. Rehabil. Eng. 26, 1345–1352. doi: 10.1109/TNSRE.2018.2844083
Keywords: virtual reality, VRISE, HMD, cybersickness, neuroscience, neuropsychology, psychology, VR
Citation: Kourtesis P, Collina S, Doumas LAA and MacPherson SE (2019) Technological Competence Is a Pre-condition for Effective Implementation of Virtual Reality Head Mounted Displays in Human Neuroscience: A Technological Review and Meta-Analysis. Front. Hum. Neurosci. 13:342. doi: 10.3389/fnhum.2019.00342
Received: 27 June 2019; Accepted: 18 September 2019;
Published: 02 October 2019.
Edited by:
Valerio Rizzo, University of Palermo, ItalyReviewed by:
Camila Rosa De Oliveira, Faculdade Meridional (IMED), BrazilUri Maoz, Chapman University, United States
Copyright © 2019 Kourtesis, Collina, Doumas and MacPherson. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Panagiotis Kourtesis, cGtvdXJ0ZXMmI3gwMDA0MDtleHNlZWQuZWQuYWMudWs=