Dimitrios Saredakis1*†
Ancret Szpak1†
Brandon Birckhead2
Hannah A. D. Keage1
Albert Rizzo3
Tobias Loetscher1- 1Cognitive Ageing and Impairment Neurosciences Laboratory, School of Psychology, Social Work and Social Policy, University of South Australia, Adelaide, SA, Australia
- 2Division of Health Services Research, Department of Medicine, Cedars-Sinai Health System, Los Angeles, CA, United States
- 3Institute for Creative Technologies, University of Southern California, Los Angeles, CA, United States
The use of head-mounted displays (HMD) for virtual reality (VR) application-based purposes including therapy, rehabilitation, and training is increasing. Despite advancements in VR technologies, many users still experience sickness symptoms. VR sickness may be influenced by technological differences within HMDs such as resolution and refresh rate, however, VR content also plays a significant role. The primary objective of this systematic review and meta-analysis was to examine the literature on HMDs that report Simulator Sickness Questionnaire (SSQ) scores to determine the impact of content. User factors associated with VR sickness were also examined. A systematic search was conducted according to PRISMA guidelines. Fifty-five articles met inclusion criteria, representing 3,016 participants (mean age range 19.5–80; 41% female). Findings show gaming content recorded the highest total SSQ mean 34.26 (95%CI 29.57–38.95). VR sickness profiles were also influenced by visual stimulation, locomotion and exposure times. Older samples (mean age ≥35 years) scored significantly lower total SSQ means than younger samples, however, these findings are based on a small evidence base as a limited number of studies included older users. No sex differences were found. Across all types of content, the pooled total SSQ mean was relatively high 28.00 (95%CI 24.66–31.35) compared with recommended SSQ cut-off scores. These findings are of relevance for informing future research and the application of VR in different contexts.
Introduction
Despite advancements in virtual reality (VR) technology, many people still report experiencing simulator sickness symptoms from its use (Rebenitsch and Owen, 2016; Gavgani et al., 2017; Duzmanska et al., 2018; Guna et al., 2019). Characterizing and quantifying these symptoms is challenging, as several factors are at play including a diverse range of technologies; the use of inconsistent terminology for sickness from using virtual environments; little consensus on the biological mechanisms of symptoms; the diverse range of VR content; along with user characteristics such as age and sex (Hale and Stanney, 2014). Identifying factors that increase the occurrence of simulator sickness becomes necessary with the increased use of VR for rehabilitation, industry training and gaming/entertainment consumers (Gallagher and Ferrè, 2018; Powell et al., 2018; Wang et al., 2018).
Side effects from virtual environment usage has been referred to by many terms including simulator sickness (Kennedy et al., 1993), cybersickness (LaViola, 2000) and VR sickness (Kim et al., 2018). The term simulator sickness originated from the early use of flight simulators in the military (Kennedy et al., 1993), and is still currently used in research using modern HMD technology (Tyrrell et al., 2018; Ziegler et al., 2018). Cybersickness, originally used to describe side effects from use of virtual environments (McCauley and Sharkey, 1992), has often been mentioned in studies using a variety of technologies including flat screen displays and head-mounted displays (HMD) (Rebenitsch and Owen, 2016). The term VR sickness has typically been used in studies using HMDs (Cobb et al., 1999; Kim et al., 2018). Thus, diverse terminology is often used interchangeably across the virtual environments literature.
This current review focuses on adverse symptoms from HMD use, hence the term “VR Sickness” will be referred to as the symptoms (and their severity) typically reported in the literature from HMD use. The term “motion sickness” will be used to refer to more general reporting of symptoms from motion environments (e.g., air, land, or sea travel), not specific to HMDs, where symptoms can differ. For example, nausea can be more severe in seasickness, compared with simulator use (Kennedy et al., 2010). Symptomatology of sickness also differs between technologies. Compared with simulators, HMDs have been reported to produce higher symptoms related to nausea, dizziness and blurred vision (Kennedy et al., 2003).
Measures of VR sickness are a fundamental part of establishing prevalence and symptomatology in virtual environments. The Simulator Sickness Questionnaire (SSQ) (Kennedy et al., 1993), originally developed for measuring motion sickness in simulators, is the most commonly used measure of sickness in virtual environments (Rebenitsch and Owen, 2016). Alternate measures, such as the Virtual Reality Symptom Questionnaire, which was specifically developed for HMDs (Ames et al., 2005) or the Virtual Reality Sickness Questionnaire (Kim et al., 2018) have yet to be widely adopted. Single item assessments that are easy to administer and monitor symptoms during VR exposure (Bos et al., 2005; Keshavarz and Hecht, 2011) are commonly used, but do not provide comprehensive measurements of the symptoms of VR sickness. Very few studies report on the use of objective physiological measures (e.g., heart rate, skin conductance, electroencephalograms, eye blink rate, and electrogastrogram) that do not rely on individual self-report data (Kim et al., 2005; Dennison et al., 2016).
Recent advances in HMD technology (field of view, resolution, framerate, and ergonomic factors) have increased the levels of immersion and realism that may have an influence on the occurrence of VR sickness (Nichols, 1999; Lee et al., 2017; Kourtesis et al., 2019). For example, if an image is clear and tracking of movement is accurate, there may be fewer sensory conflicts, and that could lead to a reduction in VR sickness symptoms (White et al., 2015; Shin et al., 2016; Ray et al., 2018). However, an increase in the field of view may also increase risk of VR sickness (Fernandes and Feiner, 2016). Despite the improvements in HMD technology, a recent review suggests that the prevalence of VR sickness is still problematic (Rebenitsch and Owen, 2016). In addition to this, Kourtesis et al. (2019), in their review found that although recent hardware features have been an important factor in reducing VR sickness, software features also need to be taken into consideration.
The VR content delivered to users can induce or even reduce VR sickness. A rollercoaster ride may be more likely to induce VR sickness to the level of severity where users will request to discontinue the experience. For example, almost 67% of participants in a study using a rollercoaster virtual environment were unable to complete an exposure time of 14 min (Nesbitt et al., 2017). In contrast, content consisting of low amounts of motion may be less likely to induce VR sickness (Guna et al., 2019), as well as in cases where head movement in a fixed position is concordant with what the user would experience in the real world (Rizzo and Koenig, 2017).
Length of time exposed to a virtual environment may also influence likelihood and severity of VR sickness (Duzmanska et al., 2018). Significant correlations have been found between exposure time and VR sickness, with longer exposure times increasing risk of VR sickness (Stanney et al., 2003). For example, research measuring VR sickness at multiple time points found symptoms increased at 2-min increments, with the highest VR sickness scores measured in the final trial at 10 min (Moss and Muth, 2011). In contrast, a recent review has found that some people may build up a resistance or adapt over time to VR sickness, particularly over multiple sessions (Duzmanska et al., 2018). Although content and duration are significant contributing factors that may increase the likelihood of sickness symptoms, the user also needs to be taken into consideration.
User characteristics adds another layer of complexity in understanding the relationship between hardware, content and VR sickness. Research on sex and age, have generated mixed findings when it comes to the likelihood of sickness from VR (Cheung and Hofer, 2002; Benoit et al., 2015; Munafo et al., 2017; Arcioni et al., 2018). In reference to age, physiological differences over the lifespan (i.e., visual, vestibular senses) (Bermúdez Rey et al., 2016) may influence the occurrence of VR sickness and symptom profiles. For example, hormonal differences in females have been reported to influence and likely to be a factor in increased rates of VR sickness (Clemes and Howarth, 2005). Moreover, females can have a smaller interpupillary distance (Fulvio et al., 2019) and some HMDs may not be able to be adjusted accordingly therefore creating eye strain and general discomfort. Thus, it is important to increase the understanding of the relationship between these user characteristics and VR sickness.
Previous reviews (Rebenitsch and Owen, 2016; Duzmanska et al., 2018; Kourtesis et al., 2019) have focused on temporal or technological aspects of VR sickness. To date, none of the reviews on VR sickness have systematically evaluated VR content and user characteristics in a meta-analysis. The primary aim of this systematic review is to examine if VR sickness symptoms measured with the SSQ using HMDs are influenced by different factors. More specifically, factors that will be examined in this review are content, the amount of visual stimulation (motion of virtual environment), whether a person is stationary or moving in the virtual environment and time. As the SSQ consists of three grouped factors (nausea, oculomotor, and disorientation), a summary of the most common symptoms using HMDs will be provided. Studies with the intention of inducing or not inducing VR sickness will also be compared. A secondary aim is to examine the influence of user characteristics (i.e., age and sex) on SSQ scores and dropout rates.
Methods
Search Strategy
In accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) statement (Liberati et al., 2009), a systematic literature search was conducted to reveal journal and conference papers related to VR sickness from using HMDs. This review included the following search terms: virtual reality OR virtual environment* OR VR OR VR headset OR virtual reality headset OR head-mounted display OR HMD OR helmet mounted display AND cybersickness OR motion sickness OR simulator sickness OR visually induced motion sickness OR virtual reality induced motion sickness OR virtual reality induced symptoms and effects OR virtual reality sickness OR visual-vestibular OR nausea OR aftereffect* OR after effect* OR VIMS. No limiters were inserted in the database searches.
This search was carried out on the 10th October 2018 in the six databases: Cochrane Library, IEEExplore, Medline, PsycINFO, Scopus, and Web of Science. Terms were mapped to subject headings. Both journal and conference articles were included in this review if: participants used a head-mounted display (HMD); VR sickness was measured using the SSQ; articles were peer-reviewed and complete (i.e., includes a full paper, not just an abstract or poster presentation); the text was in English or had been translated for publication. Papers were excluded if they: used augmented reality (AR) or see-through displays; were reviews, dissertations, abstracts or poster presentations; used prototype HMD devices; and were case studies. Papers that included clinical samples were also excluded, however, if the study included a healthy control group, this data was included. Eligibility of studies was assessed by two independent reviewers (DS and AS).
Papers were included if they supplied mean data for the SSQ (either subscales or total scores), if no mean data was supplied they were still included in the dropout analysis if they indicated drop out rates. If papers supplied mean scores without standard deviations, authors were contacted to supply the standard deviations. Current contact details were searched for online in each case. A follow-up email was sent to authors that did not respond to the initial email. If the authors did not respond to the second email the paper was excluded. The calculation of subscale and total SSQ scores required weighting. Subscales are weighted as follows; nausea 9.54; oculomotor 7.58; and disorientation 13.92, while total scores can be calculated by multiplying unweighted subscale scores by 3.74 (Kennedy et al., 1993). This can create some confusion at times, and there were instances where researchers calculated the scores differently. For example, multiplying the weighted subscale scores by 3.74 thereby producing inflated total scores. There were also instances where the total SSQ scores did not match the subscale scores, the same contact procedure was followed for these papers as per the missing standard deviations.
Figure 1 shows the results of the electronic search and article selection as per PRISMA guidelines (Liberati et al., 2009).
Figure 1. The article selection and screening process using the PRISMA flow diagram (Liberati et al., 2009).
Statistical Approach
Comprehensive Meta-Analysis (CMA) Version 3 (Borenstein et al., 2013) was used to conduct meta-analyses. A random effects model was used to calculate pooled effect estimates with 95% confidence intervals. In studies reporting multiple experiments within groups, these means were merged in CMA to produce one mean per study. In studies reporting multiple experiments between groups, these means were calculated separately for each experiment. Pooled means were calculated for all factors separately on each subscale of the SSQ. Pooled means were also calculated for all factors separately for the total SSQ score. Differences between sub-factors within each factor were assessed using the Q-test based on analysis of variance (Borenstein et al., 2011). The Q-value for the between group analyses corresponded to the weighted sum of squared deviations of the subgroup means about the grand mean. P-values were obtained by comparing the Q-values with a chi-squared distribution with degrees of freedom equal to the number of subgroups minus one (Borenstein et al., 2013). A p-value lower than 0.05 was assumed to indicate a significant statistical difference of SSQ scores between the sub-factors. A correlation was performed between the percentage of females in studies and total SSQ scores as breakdowns for sex of means for the SSQ scores were not supplied in most studies.
Operationalisation of Factors Being Examined
All factors were operationalised and independently reviewed by DS and AS. Any disagreements were resolved by discussion.
Content
Four types of content were categorized in studies included for analysis; 360 videos; gaming content; minimalist content; and scenic content. User interaction and environmental features differed for each category. The 360 videos included content captured with a 360 camera or video taken that allowed a 360 view of the virtual environment. Gaming included high detailed content where the user could actively interact and perform tasks in the virtual environment including off-the-shelf games and content developed by researchers. Minimalist content consisted of basic shapes or minimal textures, with typically simple interactions. Scenic content included detailed environments, for example, a landscape or cityscape with no or simple interaction by the user. See Figure 2 for a summary of content characteristics.
Figure 2. Content characteristics and participant's sickness response. Emoticons indicate participant level of discomfort according to total Simulator Sickness Questionnaire (SSQ) scores averaged across all studies.
Visual Stimulation
All studies were categorized based on the amount of visual movement within the content regardless of user-directed movement, such as locomotion and head movement. Low visual stimulation included content with slow visual changes, while high visual stimulation included content with fast visual changes.
Locomotion
Locomotion refers to how a user navigates in the virtual environment. For the analysis in this review, locomotion was classified as either stationary, controller-based movement, or physically walking. With stationary content, the user does not move in the virtual environment. Two moving categories were included; controller and walking. Controller-based movement included the following navigation methods; flying; controller-based walking; teleporting and driving, therefore any movement for navigation by the user. Walking included the following physical movements; walking; walking in place and walking on a treadmill. The two categories of moving were used as physically walking has been found to reduce the incidence of VR sickness compared to controller-based navigation (Chance et al., 1998).
Time
Sickness in virtual environments has been found to increase after 10 min in HMDs and simulator studies (Min et al., 2004; Moss and Muth, 2011). Thus, time was categorized into three intervals of 10 min: <10 min, ≥10 min, or ≥20 min.
VR Sickness Condition
Studies that explicitly set out to increase/decrease the occurrence of VR sickness or measured VR sickness as a secondary aim, were categorized into two conditions: induce, and not induce.
User Characteristics
The user characteristic of age was categorized into a mean age of <35 years old and ≥35 years old. This cut-off was used to correspond with theories of both sensory conflict and postural instability. For example, vestibular function involved in the sensory conflict theory starts to decline around the age of 40 (Bermúdez Rey et al., 2016). With relevance to the postural instability theory, changes in altered postural balance have been reported to commence at the ages of 30–39 (Era et al., 2006).
Mean breakdowns by sex were not supplied in most SSQ studies. Therefore, a correlational analysis was performed looking at the proportion of sex (females) in studies with total SSQ mean scores. This approach aimed to give an approximation due to the lack of available data, a positive correlation in this analysis will indicate higher susceptibility of VR sickness in females.
Dropouts
Dropouts in this review refer to participants that exited an experiment due to VR sickness.
Results
A total of 2,654 publications were identified through the search. A snowballing strategy was used to identify an additional 15 articles for inclusion. These publications were imported into EndNote where 1,045 duplicates were removed. The remaining 1,609 articles were sent to Covidence systematic review management software (Covidence, 2019) for title and abstract screening, which identified 292 articles for full-text screening. A further 237 articles were excluded as outlined in Figure 1. Authors were contacted for 15 papers as per the procedure described in the methods section if mean scores were supplied without standard deviations (10), or if scores did not appear to be weighted correctly (5). A total of 54% of authors replied with 20% supplying raw data to enable calculation of SSQ scores. Hence, 55 publications were identified through the systematic review process and listed in Table 1.
Table 1. Summary of included articles.
Dropouts
The mean dropout rate reported across 46 experiments due to VR sickness was 15.6%. If studies did not report dropouts, they were not included in this analysis as it was unknown whether there were no instances of dropouts or whether they were just not reported.
Description of Studies
Out of the 55 papers included in this review, 20 papers reported both subscale scores and total SSQ scores, 7 papers reported subscale SSQ scores only, and 16 papers reported total SSQ scores only. Twenty papers that reported SSQ scores also reported dropout rates. A further 12 papers that used the SSQ but only reported dropout rates were also included. The total number of experiments from these papers included 54 that reported the total SSQ scores and 38 that reported the subscale SSQ scores (these numbers include between group studies from the same paper). The number of participants included in all experiments represented 3,016 participants. Heterogeneity was consistently high for all analyses (I2 > 90).
Studies came from: Australia (n = 3), Canada (n = 1), Columbia (n = 1), Cyprus (n = 1), Finland (n = 1), Germany (n = 11), Greece (n = 1), Japan (n = 1), Korea (n = 4), Netherlands (n = 3), New Zealand (n = 1), Portugal (n = 1), Slovenia (n = 1), Spain (n = 1), United Kingdom (n = 2), United States of America (n = 22).
The pooled mean age of participants was 24 years (of 45 studies that included mean age), with the youngest sample having a mean age of 19.5 years and the oldest having a mean age of 80 years. Fifty-one studies included both female and male participants, 4 studies did not report sex distributions, and 41% of participants were female. Bivariate correlations between the SSQ and percentage of females in studies were not significant (r = −0.172, p = 0.170).
See Table 2 for a summary of results showing factors associated with both total and subscale SSQ scores.
Table 2. Summary of results.
Discussion
The aim of the review was to synthesize the literature on VR sickness symptoms using HMDs measured using the SSQ. The primary aim was to examine if VR sickness symptoms are influenced by content (four categories), the amount of visual stimulation, how a person moves in the virtual environment and exposure times. With a secondary aim of examining the influence of user characteristics (i.e., age and sex).
SSQ Scores Interpretation
In this review, total SSQ mean scores ranged from 14.30 to 35.27. Pooled total SSQ scores were relatively high across all studies and content type (M = 28.00) with high levels of heterogeneity. Historically the SSQ was intended for military personnel using simulators, however, the different applications and interpretation of the scores have changed with increased use of VR and advancements in technology. When interpreting total SSQ scores, according to Kennedy et al. (2003); scores between 10 and 15 indicate significant symptoms; between 15 and 20 are a concern; and scores over 20 indicate a problem simulator. These cut-off scores were established from military personnel using flight simulators, these scores may differ in the general population, additionally, SSQ scores do tend to be higher in other virtual environments compared to flight simulators (Stanney and Kennedy, 1997; Kennedy et al., 2003). According to the Kennedy et al. (2003) categories, even the lowest total SSQ mean score of 14.30 found in studies including older adults in this current review would be regarded as significant symptoms. All remaining classifications displayed higher means with the highest total SSQ score displayed in studies that set out to induce motion sickness.
VR Sickness Symptom Profiles
Across all studies, this review found the highest pooled SSQ subscale scores for disorientation (M = 23.50), followed by oculomotor (M = 17.09) and nausea (M = 16.72). This subscale distribution demonstrates the difference with the symptom profile of motion sickness where nausea typically has the highest rating, followed by oculomotor and disorientation (Rebenitsch and Owen, 2016). These findings increase awareness of symptoms that may be more likely to develop when using HMDs (e.g., dizziness, blurred vision and difficulty focusing). However, the weighting of these subscales makes it unclear as to what degree these symptoms differ.
VR Content
The content characteristics in Figure 2 highlight the distinguishing features of the four content types that may account for the distribution of SSQ scores in this review. SSQ scores were significantly influenced by content type with gaming content displaying the highest total SSQ mean (M = 34.26). This effect was also seen for subscale SSQ scores with all measured subscale symptoms of nausea, oculomotor and disorientation highest for gaming content compared to other types of content (see Table 4). Consistent with these results, previous studies using gaming content reported the highest dropout rates, ranging from 44 to 100% (Merhi et al., 2007; Dennison et al., 2016; Munafo et al., 2017). The second highest total SSQ means were found in studies using 360 videos. This was followed by minimalist content, with scenic content producing the lowest total SSQ mean. The total SSQ means did not always correspond with dropout rates, for example higher dropout rates were found in scenic content than 360 videos. This discrepancy highlights the variability in how users tolerate HMD use that could be due to other factors. Exposure time, user characteristics or the amount of visual stimulation are all other factors that may have contributed to the high heterogeneity found in this review. Thus, a limitation of this current meta-analysis and meta-analyses in general is that methodological differences between studies are collapsed when pooling results.
Influence of Visual Stimulation on Sickness
Content varies not only by type but also by the amount of visual stimulation offered. For example, all four types of content examined in this review may provide varying degrees of visual movement to the user. Oculomotor subscale SSQ mean scores were significantly higher for high visual stimulation compared with low visual stimulation. Some of the symptoms in the oculomotor subscale relate to eyestrain, difficulty focusing, difficulty concentrating and blurred vision. Despite recent improvements in display technology, stereoscopic HMDs may produce more side effects due to the vergence-accommodation conflict. Vergence refers to the way the eyes move laterally to adjust to items moving toward and away from the eyes combined with the process of focusing (accommodation). These visual processes do not occur naturally in a stereoscopic display as accommodation occurs at a fixed screen depth (Terzić and Hansard, 2016). This conflict may be a reason for the higher SSQ means for high visual stimulation in the oculomotor SSQ subscale. When there is a high level of visual stimulation there are more changes in the stimulus distance compared to content with low visual stimulation. The level of visual stimulation is meaningful, as research examining rapid vs. slow changes in stimulus distance found rapid changes to increase visual discomfort (Kim et al., 2014). In a virtual environment, a conflict may be created due to the differences in what a person sees and what their body experiences. With the emergence of new VR technologies, high-quality stereoscopic HMDs are now capable of simulating the visual and spatial properties of the real-world. Despite improvements, current technology still falls short of replicating how humans see and perceive depth under natural viewing conditions (Howarth and Costello, 1997). There are software solutions that can help to reduce discomfort by introducing blurring during motion (Budhiraja et al., 2017), however, this technique may not be effective for everyone. The shortcomings of current HMDs can produce unnatural visual conflicts, which have been shown to play a role in VR sickness (Carnegie and Rhee, 2015), especially when they are combined with visually stimulating VR environments (Kim et al., 2014).
Locomotion Type in Virtual Environment
SSQ scores were significantly influenced by locomotion type with controller-based movement displaying the highest total SSQ mean (M = 32.55). Both nausea and oculomotor subscale SSQ scores means were also significantly influenced by locomotion type with higher scores when stationary as opposed to both controller-based moving and walking (see Table 4), high heterogeneity between studies has contributed to these differences. There are several other factors that can account for differences between total and subscale SSQ scores for locomotion between controller-based and stationary content. This includes differences in the number of studies, with seven stationary and five walking studies that reported subscale SSQ data, compared with 12 studies that reported total SSQ data for these locomotion categories. Additionally, relatively high total SSQ scores were reported for controller-based studies (Merhi et al., 2007; Budhiraja et al., 2017; Ragan et al., 2017) that did not report any subscale scores. Finally, these differences between SSQ totals and subscales may result from certain methods of locomotion having a greater impact on specific symptoms in the subscale SSQ scores depending on locomotion type that would not be reflected in the total SSQ scores. For example, being stationary in the real world may induce a greater conflict in a virtual environment where there is movement and hence may increase nausea symptoms. This is consistent with research that indicates a reduction in symptoms when user-initiated movement is matched to the environment (Lee et al., 2017; Misha et al., 2018), these findings also support the sensory conflict theory relating to a visual-vestibular conflict (Reason and Brand, 1975). Thus, the visual-vestibular conflict may be exacerbated by the type of content (moving vs. static) being viewed combined with the locomotion method. A reduction in visual-vestibular conflict may be the reason that the lowest total and subscale SSQ scores for locomotion were consistently reported in studies that included physically walking content. More research is needed to increase the understanding of how the type of locomotion can influence specific symptoms of VR sickness.
VR Exposure Time on VR Sickness
Both nausea and disorientation SSQ subscale scores in studies for exposure times of <10 min were lower than those that were equal to or >10 min. Interestingly scores were lower for studies that were equal to or >20 min than those equal to or >10 min (see Figure 3). This contradicts a recent summary in a review suggesting that longer exposure times are more likely to increase VR sickness (Duzmanska et al., 2018). Content may have been a factor contributing to this pattern of results within each of the time categories. In examining the distribution of content among the time breakdowns ≥10 min studies did have the highest percentage of gaming content (62%), compared to studies with the shortest (<10) and longest exposure times (≥20). In addition to this 50% of studies with the longest exposure times (≥20) consisted of minimalist or scenic content. More research is needed to determine the relationship between content and exposure time. Within-subject designs with different exposure times and controlled content may assist with answering questions around safe exposure times as this information is important when planning clinical trials to avoid VR sickness and dropouts and establish safe use procedures.
Figure 3. VR sickness symptoms across the different SSQ subscales of all measured factors including (A) Visual Stimulation, (B) Time, (C) VR Sickness Condition, (D) Locomotion, (E) Age, and (F) Content. SSQ, Simulator Sickness Questionnaire; Error bars represent standard error.
Age and VR Sickness
Four studies included older samples (studies with a mean age range ≥35 years; n = 64) that reported total SSQ scores. Not only did these studies report lower total SSQ scores for older samples (M = 14.30) compared to younger samples (M = 28.44), these studies reported the lowest SSQ scores when compared with all other examined factors (see Table 3). Two of the four studies with older samples also included subscale SSQ scores with 37 participants in total. The disorientation subscale recorded significantly lower SSQ scores for the older samples compared with the younger samples. While scores for nausea and oculomotor subscales were higher for the older adult samples compared with younger samples, they were not statistically significant. Previous research has found inconsistent findings when looking at older samples (Kennedy et al., 2010; Benoit et al., 2015). Even though age has been reported as a user characteristic likely to predict motion sickness (Golding, 2006), the results from this review support previous research that there may be a decline in susceptibility to VR sickness as a person ages (Paillard et al., 2013). However, as there are a limited number of studies including older samples, these results should be interpreted with caution. Additionally, three of the studies used scenic content and one study used gaming content. What also needs to be considered is that the VR content for the studies including older adults may be assessing specific symptoms, and the virtual environments may be designed to reduce the likelihood of side effects. For example, two of the studies (Parijat and Lockhart, 2011; Kim et al., 2017) involved walking on a treadmill to assess gait or balance and consisted of content with the lowest total SSQ mean scores in this review (scenic content). It is also possible that older adults may experience symptoms that differ to younger adults as indicated with lower disorientation subscale SSQ scores found in the older samples (symptoms related to dizziness, vertigo, blurred vision, nausea and difficulty focusing). With many companies offering VR services to aged care facilities (Aged Care Virtual Reality, 2018; Reminiscience, 2018; Rendever, 2018), the use by older adults will continue to increase. Moreover, VR delivered in HMDs is being widely used for rehabilitation, assessment and even prediction of cognitive impairments in older adults (Optale et al., 2010; Corriveau Lecavalier et al., 2018; Howett et al., 2019). Therefore, more research is needed to evaluate safety aspects of using HMD-delivered VR with older adults having cognitive decline or other age-related health conditions.
Table 3. Simulator sickness questionnaire total scores.
Table 4. Simulator sickness questionnaire subscale scores.
Sex and VR Sickness
An analysis of sex differences was performed with a correlation between the percentage of females in studies and total SSQ scores. Sex breakdown was not supplied in studies when reporting total SSQ scores, therefore, this was the only way that sex could be analyzed and therefore a limitation of this analysis. The results indicated no difference. This is not consistent with research indicating that females are at higher risk of VR sickness (Lawson et al., 2004). Finding evidence in studies that females are more susceptible than males to VR sickness depends on what study is examined with many confounding variables not taken into account (Lawson, 2015). The importance of this topic suggests that more research is needed to better understand the incidence of VR sickness based on sex differences. Age and sex have been stated as being the most common user characteristics likely to predict motion sickness (Golding, 2006) highlighting a need for further research. Other user characteristics including ethnicity; motion sickness susceptibility; fitness; and prior experience of VR may provide a deeper insight into symptomatology of user characteristics and assist to develop a more targeted approach to dealing with VR sickness.
Strengths and Limitations
This is the first study to pool estimates of VR sickness symptoms measured with the SSQ using HMDs with a pooled sample size of 3,016, however, the study is not without limitations. Although the most commonly used measure of VR sickness was used (SSQ), there were also many studies excluded (112) that did not use the SSQ. As the SSQ is self-report participants may under or over-report symptoms. Physiological measures can assist with overcoming this limitation however, a consensus is yet to be reached on the best physiological response for assessing VR sickness (Duzmanska et al., 2018). The scoring system for the SSQ can create some confusion and this was seen in this review with some authors incorrectly calculating total scores. Another limitation of the SSQ is the relevance of symptoms for HMD use. For example, the Virtual Reality Symptom Questionnaire (Ames et al., 2005), increased the focus on oculomotor symptoms, while Kim et al. (2018) removed the symptom of nausea in the Virtual Reality Sickness Questionnaire, due to not contributing to motion sickness compared with other symptoms, both of these studies were HMD specific. For a more detailed discussion of alternative measures see (Hale and Stanney, 2014).
Additionally, all analyses had high heterogeneity demonstrating large variation across the included studies. As well as individual differences of age and sex, susceptibility to VR sickness can also vary between individuals and therefore influence results. Gaming or VR experience is another individual difference that can influence the likelihood of side effects and needs to be both reported and taken into account during analysis of results. The small number of studies including older adults and lack of reporting of sex differences and dropouts are also limitations and areas requiring further research or improved reporting in future VR studies including HMDs. As 22 studies did not report dropout rates, the rate of 15.6% may be inflated if many of these studies did not have dropouts, however, we cannot assume there were no dropouts if they were not reported. This highlights the need to make reporting of dropout rates a standard in VR research.
Finally, another limitation involves the varied nature of the HMDs used across these studies. HMDs can differ in terms of field of view, use of stereo, resolution, framerate, availability of inter-pupillary distance controls/adjustment, and other technical display factors. Modern HMDs from the last 5 years differ fundamentally from the more limited display technology that was available before these recent advances (Kourtesis et al., 2019), and since 35% of papers included in this analysis used these older HMDs, it is difficult to predict how those findings would predict the occurrence of symptoms with use of currently available HMDs. Moving forward, there is an obvious need for more controlled laboratory research with standard reference VR environments that are adjustable in terms of content, movement, user interaction, etc. With such specifically created environments, one would be able to test out the incidence of side effects across different display types with varied hardware capabilities. This will be essential for promoting parametric research that creates a database of known properties for different types of virtual environments delivered across varied hardware types and would serve to produce the baseline normative data needed to enable better research in how to mitigate or eliminate the incidence of these use-limiting side effects.
Conclusion
Previous research has focused on the influence of technological aspects on VR sickness. This review advances this knowledge by examining content as a major contributing factor to VR sickness, which will remain a problem despite future technological advances. Our findings show that content significantly influences VR sickness symptoms. Recent HMD technology can provide a better experience (Kourtesis et al., 2019) and if this is combined with careful selection of content the risk of VR sickness can be reduced and those symptoms that do occur can be easily managed. In this review, we compared our total SSQ scores with the cut-off scores suggested by Kennedy et al. (2003), what these scores mean in relation to HMDs and how these scores relate to the general population remains unclear. Nevertheless, comparing total scores between studies shows that content is a major contributing factor. This review also highlights the need for a further understanding of the influence of user characteristics such as age and sex as there is a lack of studies including older samples, and sex differences that are often not reported. Increasing our understanding of VR sickness could be particularly valuable to researchers and practitioners, as there may be ethical and liability implications in research, training and clinical applications.
Data Availability Statement
The datasets generated for this study are available on request to the corresponding author.
Author Contributions
DS, AS, and TL conception of the work and analyzed and interpreted the results. DS and AS article selection and screening. DS wrote the manuscript. All authors revised the work critically for important intellectual content and have read and approved the manuscript. AS and DS created Figure 2, certain 3D models for this figure sourced from cadnav.com and modified.
Funding
DS was supported by the Australian Government Research Training Program Scholarship. TL and HK were funded by National Health and Medical Research Council (NHMRC) Dementia Research Leadership Fellowships (GNT1136269 & GNT1135676 respectively).
Conflict of Interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
References
Aged Care Virtual Reality (2018). VR Programs for Health Benefits. Retrieved from http://www.agedcarevirtualreality.com/ (accessed December 17, 2018).
Ames, L. S., Wolffsohn, S. J., and McBrien, A. N. (2005). The development of a symptom questionnaire for assessing virtual reality viewing using a head-mounted display. Optom. Vis. Sci. 82, 168–176. doi: 10.1097/01.OPX.0000156307.95086.6
Arcioni, B., Palmisano, S., Apthorp, D., and Kim, J. (2018). Postural stability predicts the likelihood of cybersickness in active HMD-based virtual reality. Displays 58, 3–11. doi: 10.1016/j.displa.2018.07.001
Benoit, M., Guerchouche, R., Petit, P.-D., Chapoulie, E., Manera, V., Chaurasia, G., et al. (2015). Is it possible to use highly realistic virtual reality in the elderly? A feasibility study with image-based rendering. Neuropsychiatr. Dis. Treat. 11, 557–563. doi: 10.2147/NDT.S73179
Bermúdez Rey, M. C., Clark, T. K., Wang, W., Leeder, T., Bian, Y., and Merfeld, D. M. (2016). Vestibular perceptual thresholds increase above the age of 40. Front. Neurol. 7:162. doi: 10.3389/fneur.2016.00162
Bessa, M., Melo, M., Narciso, D., Barbosa, L., and Vasconcelos-Raposo, J. (2016). “Does 3D 360 video enhance user's VR experience? An evaluation study,” in Proceedings of the XVII International Conference on Human Computer (New York, NY), 1–4. doi: 10.1145/2998626.2998669
Borenstein, M., Hedges, L., Higgins, J., and Rothstein, H. (2013). Comprehensive Meta-Analysis Version 3. Englewood, NJ: Biostat.
Borenstein, M., Hedges, L. V., Higgins, J. P., and Rothstein, H. R. (2011). Introduction to Meta-Analysis. John Wiley & Sons.
Bos, J. E., MacKinnon, S. N., and Patterson, A. (2005). Motion sickness symptoms in a ship motion simulator: effects of inside, outside, and no view. Aviat. Space Environ. Med. 76, 1111–1118.
Brooks, J., Lodge, R., and White, D. (2017). Comparison of a head-mounted display and flat screen display during a micro-UAV target detection Task. Proc. Hum. Factors Ergon. Soc. Annu. Meet. 61, 1514–1518. doi: 10.1177/1541931213601863
Budhiraja, P., Miller, M. R., Modi, A. K., and Forsyth, D. (2017). Rotation blurring: use of artificial blurring to reduce cybersickness in virtual reality first person shooters. arXiv [Preprint]. arXiv:1710.02599.
Carnegie, K., and Rhee, T. (2015). Reducing visual discomfort with HMDs using dynamic depth of field. IEEE Comput. Graph. Appl. 35, 34–41. doi: 10.1109/MCG.2015.98
Chance, S. S., Gaunet, F., Beall, A. C., and Loomis, J. M. (1998). Locomotion mode affects the updating of objects encountered euring travel: the contribution of vestibular and proprioceptive inputs to path integration. Presence Teleop. Virt. Environ. 7, 168–178. doi: 10.1162/105474698565659
Cheung, B., and Hofer, K. (2002). Lack of gender difference in motion sickness induced by vestibular Coriolis cross-coupling. J. Vestibul. Res. 12, 191–200.
Christou, C. G., and Aristidou, P. (2017). Steering Versus Teleport Locomotion for Head Mounted Displays. Cham: Springer. doi: 10.1007/978-3-319-60928-7_37
Clemes, S. A., and Howarth, P. A. (2005). The menstrual cycle and susceptibility to virtual simulation sickness. J. Biol. Rhythms. 20, 71–82. doi: 10.1177/0748730404272567
Cobb, S. V. G., Nichols, S., Ramsey, A., and Wilson, J. R. (1999). Virtual Reality-Induced Symptoms and Effects (VRISE). Presence Teleop. Virt. Environ. 8, 169–186. doi: 10.1162/105474699566152
Corriveau Lecavalier, N., Ouellet, É., Boller, B., and Belleville, S. (2018). Use of immersive virtual reality to assess episodic memory: a validation study in older adults. Neuropsychol. Rehabil. 1–19. doi: 10.1080/09602011.2018.1477684
Covidence (2019). Covidence Systematic Review Software. Melbourne, Australia: Veritas Health Innovation. Retrieved from: https://www.covidence.org/home (accessed February 4, 2019).
Deb, S., Carruth, D. W., Sween, R., Strawderman, L., and Garrison, T. M. (2017). Efficacy of virtual reality in pedestrian safety research. Appl. Ergon. 65, 449–460. doi: 10.1016/j.apergo.2017.03.007
Dennison, M. S., and D'Zmura, M. (2017). Cybersickness without the wobble: experimental results speak against postural instability theory. Appl. Ergon. 58, 215–223. doi: 10.1016/j.apergo.2016.06.014
Dennison, M. S., and D'Zmura, M. (2018). Effects of unexpected visual motion on postural sway and motion sickness. Appl. Ergon. 71, 9–16. doi: 10.1016/j.apergo.2018.03.015
Dennison, M. S., Wisti, A. Z., and D'Zmura, M. (2016). Use of physiological signals to predict cybersickness. Displays 44, 42–52. doi: 10.1016/j.displa.2016.07.002
Dorado, J. L., and Figueroa, P. A. (2014). “Ramps are better than stairs to reduce cybersickness in applications based on a HMD and a Gamepad,” in 2014 IEEE Symposium on 3D User Interfaces (3DUI) (Minneapolis, MN), 47–50. doi: 10.1109/3DUI.2014.6798841
Duzmanska, N., Strojny, P., and Strojny, A. (2018). Can simulator sickness be avoided? a review on temporal aspects of simulator sickness. Front. Psychol. 9:2132. doi: 10.3389/fpsyg.2018.02132
Era, P., Sainio, P., Koskinen, S., Haavisto, P., Vaara, M., and Aromaa, A. (2006). Postural balance in a random sample of 7,979 subjects aged 30 years and over. Gerontology 52, 204–213. doi: 10.1159/000093652
Farmani, Y., and Teather, R. J. (2018). “Viewpoint snapping to reduce cybersickness in virtual reality,” in Proceedings of Graphics Interface (Toronto, ON). doi: 10.20380/GI2018.21
Fernandes, A. S., and Feiner, S. K. (2016). “Combating VR sickness through subtle dynamic field-of-view modification,” in 2016 IEEE Symposium on 3D User Interfaces (3DUI) (Greenville, SC). doi: 10.1109/3DUI.2016.7460053
Frommel, J., Sonntag, S., and Weber, M. (2017). “Effects of controller-based locomotion on player experience in a virtual reality exploration game,” in Proceedings of the 12th International Conference on the Foundations of Digital Games (Hyannis, MA), 1–6. doi: 10.1145/3102071.3102082
Fujikake, K., Miyao, M., Watanabe, T., Hasegawa, S., Omori, M., and Takada, H. (2009). “Evaluation of body sway and the relevant dynamics while viewing a three-dimensional movie on a head-mounted display by using stabilograms,” in International Conference on Virtual and Mixed Reality, ed R. Shumaker (Berlin; Heidelberg), 41–50. doi: 10.1007/978-3-642-02771-0_5
Fulvio, J. M., Ji, M., and Rokers, B. (2019). Variability in sensory sensitivity predicts motion sickness in virtual reality. bioRxiv [Preprint]. 488817. doi: 10.1101/488817
Gallagher, M., and Ferrè, E. R. (2018). Cybersickness: a multisensory integration perspective. Multisens. Res. 31, 645–674. doi: 10.1163/22134808-20181293
Gavgani, A. M., Nesbitt, K. V., Blackmore, K. L., and Nalivaiko, E. (2017). Profiling subjective symptoms and autonomic changes associated with cybersickness. Auton. Neurosci. 203, 41–50. doi: 10.1016/j.autneu.2016.12.004
Golding, J. F. (2006). Predicting individual differences in motion sickness susceptibility by questionnaire. Pers. Individ. Dif. 41, 237–248. doi: 10.1016/j.paid.2006.01.012
Guna, J., Geršak, G., Humar, I., Song, J., Drnovšek, J., and Pogačnik, M. (2019). Influence of video content type on users' virtual reality sickness perception and physiological response. Future Gener. Comput. Syst. 91, 263–276. doi: 10.1016/j.future.2018.08.049
Hale, K. S., and Stanney, K. M. (2014). Handbook of Virtual Environments: Design, Implementation, and Applications, 2nd Edn. Boca Raton, FL: CRC Press. doi: 10.1201/b17360
Howarth, P. A., and Costello, P. J. (1997). The occurrence of virtual simulation sickness symptoms when an HMD was used as a personal viewing system. Displays, 18, 107–116. doi: 10.1016/S0141-9382(97)00011-5
Howett, D., Castegnaro, A., Krzywicka, K., Hagman, J., Marchment, D., Henson, R., et al. (2019). Differentiation of mild cognitive impairment using an entorhinal cortex-based test of virtual reality navigation. Brain 142, 1751–1766. doi: 10.1093/brain/awz116
Hutton, C., Ziccardi, S., Medina, J., and Rosenbarg, E. S. (2018). “Please don't puke: early detection of severe motion sickness in VR,” in 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR) (Reutlingen), 579–580. doi: 10.1109/VR.2018.8446382
Iskenderova, A., Weidner, F., and Broll, W. (2017). “Drunk virtual reality gaming: exploring the influence of alcohol on cybersickness,” in Conference: Annual Symposium on Computer-Human Interaction in Play (Amsterdam), 561–572. doi: 10.1145/3116595.3116618
Kang, Y. J., Ku, J., Han, K., Kim, S. I., Yu, T. W., Lee, J. H., et al. (2008). Development and clinical trial of virtual reality-based cognitive assessment in people with stroke: preliminary study. Cyberpsychol. Behav. 11, 329–339. doi: 10.1089/cpb.2007.0116
Karl, I., Berg, G., Ruger, F., and Farber, B. (2013). Driving behavior and simulator sickness while driving the vehicle in the loop: validation of longitudinal driving behavior. IEEE Trans. Intell. Transp. Syst. Mag. 5, 42–57. doi: 10.1109/MITS.2012.2217995
Kennedy, R. S., Drexler, J., and Kennedy, R. C. (2010). Research in visually induced motion sickness. Appl. Ergon. 41, 494–503. doi: 10.1016/j.apergo.2009.11.006
Kennedy, R. S., Drexler, J. M., Compton, D. E., Stanney, K. M., Lanham, D. S., and Harm, D. L. (2003). “Configural scoring of simulator sickness, cybersickness and space adaptation syndrome: similarities and differences,” in Virtual and Adaptive Environments: Applications, Implications, and Human Performance Issues, eds L. J. Hettinger and M. W. Haas (Lawrence Erlbaum Associates Publishers), 247–278. doi: 10.1201/9781410608888.ch12
Kennedy, R. S., Lane, N. E., Berbaum, K. S., and Lilienthal, M. G. (1993). Simulator sickness questionnaire: an enhanced method for quantifying simulator sickness. Int. J. Aviat. Psychol. 3, 203–220. doi: 10.1207/s15327108ijap0303_3
Keshavarz, B., and Hecht, H. (2011). Validating an efficient method to quantify motion sickness. Hum. Factors. 53, 415–426. doi: 10.1177/0018720811403736
Kesztyues, T. I., Mehlitz, M., Schilken, E., Weniger, G., Wolf, S., Piccolo, U., et al. (2000). Preclinical evaluation of a virtual reality neuropsychological test system: occurrence of side effects. Cyberpsychol. Behav. 3, 343–349. doi: 10.1089/10949310050078788
Kim, A., Darakjian, N., and Finley, J. M. (2017). Walking in fully immersive virtual environments: an evaluation of potential adverse effects in older adults and individuals with Parkinson's disease. J. Neuroeng. Rehabil. 14:16. doi: 10.1186/s12984-017-0225-2
Kim, H. K., Park, J., Choi, Y., and Choe, M. (2018). Virtual reality sickness questionnaire (VRSQ): motion sickness measurement index in a virtual reality environment. Appl. Ergon. 69, 66–73. doi: 10.1016/j.apergo.2017.12.016
Kim, J., Kane, D., and Banks, M. S. (2014). The rate of change of vergence–accommodation conflict affects visual discomfort. Vision Res. 105, 159–165. doi: 10.1016/j.visres.2014.10.021
Kim, Y. Y., Kim, H. J., Kim, E. N., Ko, H. D., and Kim, H. T. (2005). Characteristic changes in the physiological components of cybersickness. Psychophysiology 42, 616–625. doi: 10.1111/j.1469-8986.2005.00349.x
Kinsella, A., Mattfeld, R., Muth, E., and Hoover, A. (2016). Frequency, not amplitude, of latency affects subjective sickness in a head-mounted display. Aerosp. Med. Hum. Perform. 87, 604–609. doi: 10.3357/AMHP.4351.2016
Kourtesis, P., Collina, S., Doumas, L. A. A., and MacPherson, S. E. (2019). Technological competence is a pre-condition for effective implementation of virtual reality head mounted displays in human neuroscience: a technological review and meta-analysis. Front. Hum. Neurosci. 13:342. doi: 10.3389/fnhum.2019.00342
Kruse, L., Langbehn, E., and Steinicke, F. (2018). “I can see on my feet while walking: sensitivity to translation gains with visible feet,” in 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR) (Reutlingen), 305–312. doi: 10.1109/VR.2018.8446216
Kuiper, O. X., Bos, J. E., and Diels, C. (2018). Vection does not necessitate visually induced motion sickness. Displays 58, 82–87. doi: 10.1016/j.displa.2018.10.001
LaViola, J. J. J. (2000). A discussion of cybersickness in virtual environments. ACM SIGCHI Bull. 32, 47–56. doi: 10.1145/333329.333344
Lawson, B. D. (2015). “Motion sickness symptomatology and origins,” in Handbook of Virtual Environments, eds K. S. Hale and K. M. Stanney (Boca Raton FL: Taylor & Francis Group), 531–600. doi: 10.1201/b17360-29
Lawson, B. D., Kass, S., Lambert, C., and Smith, S. (2004). Survey and review concerning evidence for gender differences in motion susceptibility. Aviat. Space Environ. Med. 75:105.
Lee, J., Kim, M., and Kim, J. (2017). A study on immersion and VR sickness in walking interaction for immersive virtual reality applications. Symmetry 9:78. doi: 10.3390/sym9050078
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., et al. (2009). The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. PLoS Med. 6:e1000100. doi: 10.1371/journal.pmed.1000100
Ling, Y., Brinkman, W. P., Nefs, H. T., Qu, C., and Heynderickx, I. (2012). Effects of stereoscopic viewing on presence, anxiety, and cybersickness in a virtual reality environment for public speaking. Presence Teleop. Virt. Environ. 21, 254–267. doi: 10.1162/PRES_a_00111
Llorach, G., Evans, A., and Blat, J. (2014). “Simulator sickness and presence using HMDs: comparing use of a game controller and a position estimation system,” in VRST '14 Proceedings of the 20th ACM Symposium on Virtual Reality Software and Technology (Edinburgh), 137–140. doi: 10.1145/2671015.2671120
McCauley, M. E., and Sharkey, T. J. (1992). Cybersickness: perception of self-motion in virtual environments. Presence Teleop. Virt. Environ. 1, 311–318. doi: 10.1162/pres.1992.1.3.311
McGill, M., Ng, A., and Brewster, S. (2017). “I am the passenger: how visual motion cues can influence sickness for in-car VR,” in Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (Denver, CO), 5655–5668. doi: 10.1145/3025453.3026046
Merhi, O., Faugloire, E., Flanagan, M., and Stoffregen, T. A. (2007). Motion sickness, console video games, and head-mounted displays. Hum. Factors. 49, 920–934. doi: 10.1518/001872007X230262
Min, B.-C., Chung, S.-C., Min, Y.-K., and Sakamoto, K. (2004). Psychophysiological evaluation of simulator sickness evoked by a graphic simulator. Appl. Ergon. 35, 549–556. doi: 10.1016/j.apergo.2004.06.002
Misha, S., Xuhai, X., Aske, M., and Pattie, M. (2018). “VMotion: designing a seamless walking experience in VR,” Proceedings of the 2018 Designing Interactive Systems Conference (Hong Kong).
Mittelstaedt, J., Wacker, J., and Stelling, D. (2018). Effects of display type and motion control on cybersickness in a virtual bike simulator. Displays 51, 43–50. doi: 10.1016/j.displa.2018.01.002
Moss, J. D., Austin, J., Salley, J., Coats, J., Williams, K., and Muth, E. R. (2011). The effects of display delay on simulator sickness. Displays 32, 159–168. doi: 10.1016/j.displa.2011.05.010
Moss, J. D., and Muth, E. R. (2011). Characteristics of head-mounted displays and their effects on simulator sickness. Hum. Factors. 53, 308–319. doi: 10.1177/0018720811405196
Moss, J. D., Scisco, J., and Muth, E. (2008). “Simulator sickness during head mounted display (HMD) of real world video captured scenes,” in Proceedings of the Human Factors and Ergonomics Society Annual Meeting (New York, NY), 52, 1631–1634.
Munafo, J., Diedrick, M., and Stoffregen, T. A. (2017). The virtual reality head-mounted display oculus rift induces motion sickness and is sexist in its effects. Exp. Brain Res. 235, 889–901. doi: 10.1007/s00221-016-4846-7
Nesbitt, K., Davis, S., Blackmore, K., and Nalivaiko, E. (2017). Correlating reaction time and nausea measures with traditional measures of cybersickness. Displays 48, 1–8. doi: 10.1016/j.displa.2017.01.002
Neth, C. T., Souman, J. L., Engel, D., Kloos, U., Bülthoff, H. H., and Mohler, B. J. (2011). “Velocity-dependent dynamic curvature gain for redirected walking,” in IEEE Transactions on Visualization and Computer Graphics 18 (Singapore), 1041–1052. doi: 10.1109/VR.2011.5759454
Nichols, S. (1999). Physical ergonomics of virtual environment use. Appl. Ergon. 30, 79–90. doi: 10.1016/S0003-6870(98)00045-3
Optale, G., Urgesi, C., Busato, V., Marin, S., Piron, L., Priftis, K., et al. (2010). Controlling memory impairment in elderly adults using virtual reality memory training: a randomized controlled pilot study. Neurorehabil. Neural Repair. 24, 348–357. doi: 10.1177/1545968309353328
Paillard, A., Quarck, G., Paolino, F., Denise, P., Paolino, M., Golding, J. F., et al. (2013). Motion sickness susceptibility in healthy subjects and vestibular patients: effects of gender, age and trait-anxiety. J. Vestib. Res. 23, 203–209. doi: 10.3233/VES-130501
Papachristos, N. M., Vrellis, I., and Mikropoulos, T. A. (2017). “A comparison between oculus rift and a low-cost smartphone VR headset: immersive user experience and learning,” in 2017 IEEE 17th International Conference on Advanced Learning Technologies (ICALT) (Timisoara), 477–481. doi: 10.1109/ICALT.2017.145
Parijat, P., and Lockhart, T. E. (2011). “Can virtual reality be used as a gait training tool for older adults?” in Proceedings of the Human Factors and Ergonomics Society Annual Meeting, Vol. 55 (Las Vegas, NV), 157–161. doi: 10.1177/1071181311551033
Pettijohn, K. A., Geyer, D., Gomez, J., Becker, W. J., and Biggs, A. T. (2018). Postural instability and simulator seasickness. Aerosp. Med. Hum. Perform. 89, 634–641. doi: 10.3357/AMHP.4998.2018
Pot-Kolder, R., Veling, W., Counotte, J., and van der Gaag, M. (2018). Anxiety partially mediates cybersickness symptoms in immersive virtual reality environments. Cyberpsychol. Behav. Soc. Netw. 21, 187–193. doi: 10.1089/cyber.2017.0082
Pouke, M., Tiiro, A., LaValle, S. M., and Ojala, T. (2018). “Effects of visual realism and moving detail on cybersickness,” in 25th IEEE Conference on Virtual Reality and 3D User Interfaces, (VR) (Reutlingen), 665–666. doi: 10.1109/VR.2018.8446078
Powell, W., Rizzo, A., Sharkey, P., and Merrick, J. (2018). Innovations and challenges in the use of virtual reality technologies for rehabilitation. J. Altern. Med. Res. 10, 3–4.
Ragan, E. D., Scerbo, S., Bacim, F., and Bowman, D. A. (2017). Amplified head rotation in virtual reality and the effects on 3D search, training transfer, and spatial orientation. IEEE Trans. Vis. Comput. Graph. 23, 1880–1895. doi: 10.1109/TVCG.2016.2601607
Ray, B., Jung, J., and Larabi, M. (2018,. 3-7 Sept. 2018). “On the possibility to achieve 6-DoF for 360 video using divergent multi- view content,” in 2018 26th European Signal Processing Conference (EUSIPCO) (Rome), 211–215. doi: 10.23919/EUSIPCO.2018.8553397
Rebenitsch, L., and Owen, C. (2016). Review on cybersickness in applications and visual displays. Virtual Real 20, 101–125. doi: 10.1007/s10055-016-0285-9
Reminiscience (2018). Reminiscence Based Virtual Reality. Retrieved from https://www.reminiscience.co.uk/about-us (accessed December 17, 2018).
Rendever (2018). Virtual Reality for Seniors. Retrieved from http://rendever.com/ (accessed December 17, 2018).
Rizzo, A. S., and Koenig, S. T. (2017). Is clinical virtual reality ready for primetime? Neuropsychology 31, 877–899. doi: 10.1037/neu0000405
Rupp, M. A., Odette, K. L., Kozachuk, J., Michaelis, J. R., Smither, J. A., and McConnell, D. S. (2019). Investigating learning outcomes and subjective experiences in 360-degree videos. Comput. Educ. 128, 256–268. doi: 10.1016/j.compedu.2018.09.015
Saldana, S. J., Marsh, A. P., Rejeski, W. J., Haberl, J. K., Wu, P., Rosenthal, S., et al. (2017). Assessing balance through the use of a low-cost head-mounted display in older adults: a pilot study. Clin. Interv. Aging. 12, 1363–1370. doi: 10.2147/CIA.S141251
Schmitz, P., Hildebrandt, J., Valdez, A. C., Kobbelt, L., and Ziefle, M. (2018). “You spin my head right round: threshold of limited immersion for rotation gains in redirected walking,” in IEEE Trans. Vis. Comput. Graph 24, 1623–1632. doi: 10.1109/TVCG.2018.2793671
Serge, S. R., and Fragomeni, G. (2017). “Assessing the relationship between type of head movement and simulator sickness using an immersive virtual reality head mounted display: a pilot study,” in International Conference on Virtual, Augmented and Mixed Reality, Vol. 10280 (Cham: Springer) 556–566. doi: 10.1007/978-3-319-57987-0_45
Sharples, S., Cobb, S., Moody, A., and Wilson, J. R. (2008). Virtual reality induced symptoms and effects (VRISE): comparison of head mounted display (HMD), desktop and projection display systems. Displays 29, 58–69. doi: 10.1016/j.displa.2007.09.005
Shin, J., An, G., Park, J.-S., Baek, S. J., and Lee, K. (2016). Application of precise indoor position tracking to immersive virtual reality with translational movement support. Multimed. Tools Appl. 75, 12331–12350. doi: 10.1007/s11042-016-3520-1
Singla, A., Fremerey, S., Robitza, W., Lebreton, P., and Raake, A. (2017). “Comparison of subjective quality evaluation for HEVC encoded omnidirectional videos at different bit-rates for UHD and FHD resolution,” in Proceedings of the on Thematic Workshops of ACM Multimedia (Mountain View, CA), 511–519. doi: 10.1145/3126686.3126768
Song, E.-J. (2017). A Study on cyber sickness of virtual reality based on HMD. International Information Institute (Tokyo). Information 20, 5015–5022.
Stanney, K. M., Hale, K. S., Nahmens, I., and Kennedy, R. S. (2003). What to expect from immersive virtual environment exposure: influences of gender, body mass index, and past experience. Hum. Factors. 45, 504–520. doi: 10.1518/hfes.45.3.504.27254
Stanney, K. M., and Kennedy, R. S. (1997). The psychometrics of cybersickness. Commun. ACM. 40, 66–68. doi: 10.1145/257874.257889
Stauffert, J. P., Niebling, F., and Latoschik, M. E. (2018). “Effects of Latency Jitter on Simulator Sickness in a Search Task,” in 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR) (Reutlingen), 121–127. doi: 10.1109/VR.2018.8446195
St. Pierre, M. E., Banerjee, S., Hoover, A. W., and Muth, E. R. (2015). The effects of 0.2 Hz varying latency with 20-100 ms varying amplitude on simulator sickness in a helmet mounted display. Displays 36, 1–8. doi: 10.1016/j.displa.2014.10.005
Terzić, K., and Hansard, M. (2016). Methods for reducing visual discomfort in stereoscopic 3D: a review. Signal Process. Image Commun. 47, 402–416. doi: 10.1016/j.image.2016.08.002
Tyrrell, R., Sarig-Bahat, H., Williams, K., Williams, G., and Treleaven, J. (2018). Simulator sickness in patients with neck pain and vestibular pathology during virtual reality tasks. Virtual Real. 22, 211–219. doi: 10.1007/s10055-017-0324-1
Walch, M., Frommel, J., Rogers, K., Schüssel, F., Hock, P., Dobbelstein, D., et al. (2017). “Evaluating VR driving simulation from a player experience perspective,” in Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems (Denver, CO), 2982–2989. doi: 10.1145/3027063.3053202
Wang, P., Wu, P., Wang, J., Chi, H.-L., and Wang, X. (2018). A critical review of the use of virtual reality in construction engineering education and training. Int. J. Environ. Res. Public Health. 15:1204. doi: 10.3390/ijerph15061204
Weidner, F., Hoesch, A., Poeschl, S., and Broll, W. (2017). “Comparing VR and non-VR driving simulations: an experimental user study,” in 2017 IEEE Virtual Reality (VR) (Los Angeles, CA), 281–282. doi: 10.1109/VR.2017.7892286
White, P. J., Byagowi, A., and Moussavi, Z. (2015). Effect of viewing mode on pathfinding in immersive virtual reality. Conf. Proc. IEEE Eng. Med. Biol. Soc. 2015, 4619–4622. doi: 10.1109/EMBC.2015.7319423
Young, M. K., Gaylor, G. B., Andrus, S. M., and Bodenheimer, B. (2014). “A comparison of two cost-differentiated virtual reality systems for perception and action tasks,” in Proceedings of the ACM Symposium on Applied Perception (Vancouver, BC), 83–90. doi: 10.1145/2628257.2628261
Ziegler, P., Roth, D., Knots, A., Kreuzer, M., and von Mammen, S. (2018). “ Simulator sick but still immersed: a comparison of head-object collision handling and their impact on fun, immersion, and simulator sickness,” in 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR) (Reutlingen), 743–744. doi: 10.1109/VR.2018.8446221
Keywords: cybersickness, simulator sickness, head-mounted display, virtual reality, virtual environment
Citation: Saredakis D, Szpak A, Birckhead B, Keage HAD, Rizzo A and Loetscher T (2020) Factors Associated With Virtual Reality Sickness in Head-Mounted Displays: A Systematic Review and Meta-Analysis. Front. Hum. Neurosci. 14:96. doi: 10.3389/fnhum.2020.00096
Received: 18 November 2019; Accepted: 02 March 2020;
Published: 31 March 2020.
Edited by:
Pietro Cipresso, Italian Auxological Institute (IRCCS), ItalyReviewed by:
Bernhard E. Riecke, Simon Fraser University, CanadaHai-Ning Liang, Xi'an Jiaotong-Liverpool University, China
Frederic Merienne, ParisTech École Nationale Supérieure d'Arts et Métiers, France
Copyright © 2020 Saredakis, Szpak, Birckhead, Keage, Rizzo and Loetscher. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Dimitrios Saredakis, ZGltaXRyaW9zLnNhcmVkYWtpcyYjeDAwMDQwO215bWFpbC51bmlzYS5lZHUuYXU=
†These authors share first authorship