Editorial: Presence and beyond: Evaluating user experience in AR/MR/VR

Editorial on the Research Topic
Presence and beyond: Evaluating user experience in AR/MR/VR

1 Introduction

The call for this Research Topic was intentionally broad: We sought papers that identify or propose constructs that can be used to describe AR/MR/VR, and papers that evaluate the utility of those constructs; we sought papers that discussed measures relating to user experience in AR/MR/VR - including, but not limited to, presence. In the end, we were very happy to publish fifteen articles addressing a variety of these questions - but, notably, not all of them. In the remainder of this editorial, we briefly introduce each of the fifteen articles, loosely grouping them into relevant categories. We then discuss each of the three categories in turn, and close with a call to action for our AR/MR/VR research community to more actively engage with human-computer interaction (HCI) and user experience (UX) researchers.

2 Paper summaries

The subsections that follow reflect loose topic categories that will be revisited in Section 3. That said, several articles resisted easy categorization, including these first two.

Ratan and colleagues examine the stereotype threat effect - that is, the fear of behaving in a manner stereotypically associated with one’s social group - in the context of VR and AR STEM-gaming applications. Their results suggest that VR and AR experiences may produce different levels of stereotype threat (or its opposite, stereotype reactance).

Neidhardt and Zerlik examine the plausibility of an auditory augmented reality environment that includes position-dynamic binaural synthesis. The subjects wore headphones and could move around independently. The results suggest that inexperienced listeners report a plausible illusion of the spatialized sound; however, the same results did not hold for the experienced listeners.

2.1 Theory

Skarbez, Smith, and Whitton reflect upon the reality-virtuality continuum of Milgram and Kishino. They make several arguments regarding the definition and nature of mixed reality, as well as the continuum itself. For example, they argue that virtual reality - in its present realization - should be considered a subset of MR.

Weinrich and colleagues extensively discuss the nature of the presence construct in Mixed Reality. In the process of doing so, they also propose a modified reality-virtuality continuum and offer a suite of research desiderata and research questions regarding reference frames, transportation, and realism in MR.

Latoschik and Wienrich propose a new model describing experiences across the xR spectrum which takes as its essential conditions congruence (an ontological specification of coherence) and plausibility, from which the place and plausibility illusions can be derived.

Jung and Lindeman present a model for describing the quality of a VR experience using three orthogonal dimensions: coherence, immersion, and illusion; they use illusion as an umbrella term for presence and its kin. They go on to argue that user preference is an appropriate metric for evaluating VR experiences.

Hartmann and Hofer propose a psychological parallel processing explanation for users’ experiences in xR environments. Their account claims that sensations such as presence are accompanied by the belief that “this is not really happening,” which they refer to as media awareness.

Vindenes and Wasson present a post-phenomenological framework for understanding VR experiences, which is to say they propose to study VR as a technology that mediates a human user’s relationship with the world.

2.2 Measures

Halbig and Latoschik survey the use of physiological measurements to evaluate virtual reality. They summarize research areas that have used physiological measures and provide tables enumerating the sensors and analysis tools currently available to researchers. We believe this is an excellent and comprehensive resource for researchers.

Hayes, Hughes, and Bailenson report the rigorous initial development of a system of behavioral coding to measure social presence. They validate with a user study and propose directions for future refinement of the system.

2.3 Applications

2.3.1 Social Presence

Miller and Bailenson compare the social presence engendered by virtual humans within the augmented field-of-view and outside it; that is, visible or not visible to the user. The results suggest that users feel less social presence with virtual humans they cannot see.

Sun and Won examine participants’ ability to accurately judge one another’s emotional state in VR. Participants were represented either as photorealistic or abstract (cube) avatars; the results suggest that participants could correctly judge each other’s emotional state regardless of the avatar condition.

(The article by Hayes, Hughes, and Bailenson could have been placed here as well.)

2.3.2 Learning

Bagher and colleagues examine the sense of presence and bodily engagement and their roles in enhancing learners’ experience and performance in the context of interactive virtual learning environments. They identify a positive correlation between knowledge gain and the sense of agency supported by embodied affordances.

Ochs and Sonderegger use an experimental mixed-methods approach to evaluate human performance in a memorization task. While participants who learned in VR reported higher levels of presence, participants who learned on a conventional desktop configuration demonstrated better performance on the memorization task.

Carnell and colleagues report on their experience applying the Kirkpatrick Model of training evaluation to medical communication skills training. The results of their study suggest that human behaviors observed in a virtual environment may provide early indicators of how an individual will behave in a comparable real-world scenario.

3 Themes and commonalities

3.1 Theory

A plurality of our published articles present models or frameworks for the description and analysis of AR/MR/VR experiences. Notably, two articles - by Skarbez, Smith, and Whitton and Weinrich and colleagues—propose to modify or extend Milgram and Kishino’s reality-virtuality continuum, and two more - by Latoschik and Wienrich and Jung and Lindeman—propose new models that incorporate coherence (congruence in Latoschik and Wienrich) as a key component of their models. These recurring themes communicate the enduring power of these concepts, while simultaneously indicating that they may need to be adapted to suit an evolving technological landscape.

3.2 Measures

Historically, researchers have employed questionnaires and measures of task performance, but the papers published herein highlight the utility of other techniques, such as physiological measurement and behavioral coding. Moving forward, the evaluation of AR/MR/VR systems cannot be limited to single measures, and researchers should triangulate using multiple measures informed by the specific goals of the research and objectives that AR/MR/VR systems are set to support. Rather than evaluating system hardware or software, researchers should aim to evaluate their participants’ learning, behavior, and experience.

3.3 Applications

This category includes papers that incorporated user studies primarily focused on social presence and learning applications.

3.3.1 Social presence

AR/MR/VR usage is increasingly social; as such, future research needs to consider not only the individual user’s experience of a system, but perhaps the social and cultural effects associated with that system as well. We believe that looking to our colleagues in the social sciences for inspiration, methods, and measures will be a fruitful endeavor for a field that has historically been led by computing scientists and engineers.

3.3.2 Learning

Learning, knowledge, and skill acquisition have always been key areas of AR/MR/VR research, and the articles in this Research Topic reflect that. These results suggest that while AR/MR/VR technologies are exciting new learning tools, they may not be best suited for every learning task; it is important to bear in mind that effective learning can result from any of a number of methods, many of which have been well-studied in related domains. AR/MR/VR learning applications can often benefit from the adoption of best practices from education literature focused on non-immersive learning solutions. The article by Carnell and colleagues is a good example of this: virtual humans are used not as a substitute for, but as an adjunct to, traditional learning structured around the Kirkpatrick model of evaluation for training and learning interventions.

4 Conclusion

In reviewing the articles included in this research topic, we note that authors admirably addressed presence—and social presence—from a variety of perspectives. This aspect of the research topic, then, was clearly a success. That said, many of the specific questions that we raised in the call were not addressed by any of the received manuscripts; there remains ample opportunity for future work in this area.

It may be meaningful that none of these articles adopted the language of “user experience” (UX), nor did they refer to “human-computer interaction” (HCI). We interpret this to signify an unfortunate (in our opinion) siloing of the AR/MR/VR research community. While many issues arise in the study of immersive technologies that are unique to this field, it is just as certain that this work falls within the larger HCI domain - or the UX domain, to use the language preferred by industry. We believe that a failure to situate our field within - and to engage more deeply with - these communities will limit the growth and impact of AR/MR/VR research.

Author contributions

All authors listed have made a substantial, direct, and intellectual contribution to the work, and approved it for publication.

Acknowledgments

The authors thank Douglas A. Bowman (Virginia Tech, US) for serving as Editor and shepherd for a paper in this collection for which all of the topic editors declared a conflict of interest.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.