Thammathip Piumsomboon
Youngho Lee
Parinya Punpongsanon- 1School of Product Design, Faculty of Engineering, University of Canterbury, Christchurch, New Zealand
- 2Department of Computer Engineering, Mokpo National University, Muan, South Korea
- 3Graduate School of Engineering Science, Osaka University, Suita, Japan
Editorial on the Research Topic
Supernatural enhancements of perception, interaction, and collaboration in mixed reality
1 Introduction
Immersive technologies promise an unprecedented transformation of a global society by spatially interfacing humans with the digital domain. Mixed reality (MR) enables us to merge the real and virtual world, paving the way for a new paradigm of computing and how we live and socialise across multiple worlds. Beyond the ordinaries, MR can help enhance our capabilities beyond the natural world into the realm of the supernatural. This topic has compiled six original research articles encompassing three main themes: supernatural perception for enhanced empathy, supernatural communication for effective collaboration, and supernatural interaction across reality–virtuality. Although there are two articles under each theme, their contributions are by no means limited within the theme but arguably across the board with their own motivation, unique methodology, and interesting findings Figure 1.
FIGURE 1. Six original research articles under three themes of supernatural perception for enhanced empathy, supernatural communication for enhanced collaboration, and supernatural interaction across reality‒virtuality have been compiled in this study (Zhang et al., Li et al., Jing et al., Sasikumar et al., Dong et al., and Piumsomboon et al.)
2 Supernatural perception for enhanced empathy
One of the benefits of MR is its ability to enhance the human perception of themselves, others, and their environment. An application of such an enhancement is for the MR user to be able to see through the eyes of another. Zhang et al. investigated the effects of manipulating the scale of the VR user’s virtual eye height and interpupillary distance to simulate the perspectives of children and persons with disability by extending prior visualisation techniques [e.g., Piumsomboon et al., 2018a; Piumsomboon et al., 2018b]. This is one of the first works to explore augmented perception on design-related tasks, specifically interior and furniture design, to help identify potential design hazards. Their results yielded strong evidence that experiencing other perspectives can significantly impact the perception of risks and the ability to identify hazards and influence the estimated scale of the virtual designs. Furthermore, there was a strong positive correlation between different eye heights and interpupillary distances on the resulting virtual designs for different perspectives.
On the contrary, when MR has not been well designed, it can also hinder the experience. Li et al. examined the scenario where co-located collaborators using see-through head-worn displays employed barehanded referencing in the modern MR systems. However, issues may arise when the geometric model of the hands cannot be obtained, causing difficulty in determining a precise pointing location when the hands are occluded by the virtual object(s). Interestingly, participants compensated by developing several strategies to mitigate the problem. The pointer would point from the top or have their finger penetrate the virtual object. The observer would study the characteristic of the pointer’s pointing gesture or envision the pointer’s perspective and estimate where the pointing should land. This work provoked thoughts on issues that arise from the limitations of the current generation of technologies and how they shaped users’ behaviour beyond the norms and, at times, unnatural or suboptimal.
3 Supernatural communication for enhanced collaboration
MR can help augment collaborations with cues and information to improve communication between collaborators. Jing et al. conducted the first user study on visualisation cues for sharing eye-gaze in a co-located symmetric collaboration with two MR users. Three visualisation cues for bi-directional collaborative gaze were compared: a 2D ring-shaped reticle called cursor donut, a 2D ring and a ray called laser eye, and a 2D ring and a gaze path called trail path, where the two former techniques had been proposed in the past (Lee et al., 2017; Piumsomboon et al., 2019) and the last technique had been first introduced in this work. Through three independent tasks, it was found that visualisation cues significantly improved the rating of joint attention and user intention, making the collaborative task more engaging between the collaborators. Furthermore, the laser eye technique was perceived to be significantly higher in terms of effectiveness as it promoted mutual gaze awareness while requiring low mental effort.
Similarly, Sasikumar et al. explored different visualisation cues in MR collaboration but in a remote setting. Initially, their study focused on comparing the four cues to provide spatial instructions based on prior research (Piumsomboon et al., 2017; Piumsomboon et al., 2018c; Kim et al., 2019; Thanyadit et al., 2022): annotation for drawing in 3D space, hand gestures where only the hand models were shown, avatar was a rigged virtual human, and volumetric playback was represented by 3D point clouds captured using three RGBD cameras from three sides. With a mechanical assembly task, it was found that the volumetric playback cue could improve co-presence and usability while lowering mental workload and frustration. Moreover, when the eye-gaze cue was introduced, the task took less effort and mental load.
4 Supernatural interaction across reality–virtuality
Beyond enhancing an individual’s perception and communication between collaborators, user interaction is crucial in augmenting user experience in MR. Dong et al. elicited and compared two approaches of gesture interaction: surface gestures and motion gestures, for mobile augmented reality (AR). From a survey of existing mobile AR applications, they could not find any guidelines for designing gesture interaction for mobile AR. Adopting the methodology of the original gesture elicitation study (Wobbrock et al., 2009; Piumsomboon et al., 2013), they elicited 504 gestures from 21 participants for 12 general mobile AR tasks. Based on the observed design patterns, they proposed an interaction technique called Touch-Move-Release or TMR that utilises a combination of surface touch and motion gesture of the device for a unique interaction. The validation study found that TMR enhanced engagement and offered a better gaming experience, but it is less precise and more difficult to use than surface gestures.
Piumsomboon et al. were inspired by Slater et al. (2020) on the danger of over-engagement of users in the immersive virtual environment (IVE). They investigated the visualisation and interaction techniques to help disengage users without greatly disrupting their current immersive experience. With the help of 11 experts in the area, they obtained 132 techniques for four scenarios: narrative-driven, social-platform, adventure sandbox, and fast-paced. A set of expert-elicited visualisation and interaction techniques (Ex-Cit XR) was proposed as an exemplary use case for disengaging users for different scenarios. Finally, an online survey was conducted to validate the design. As a result, a novel behavioural manipulation spectrum called SPINED (Suppress-Punish-Inform-Nudge-Entice-Deter) emerged to guide how the systems can strategically escalate to disengage users from an IVE.
To conclude, these articles have demonstrated the impact of supernatural enhancements on mixed reality in perception, interaction, and collaboration in various applications. We hope these works will inspire more research to imagine use cases of immersive technologies beyond the current paradigm and take them further into the supernatural realm overcoming limitations and augmenting human capabilities.
Author contributions
All authors of this Editorial contributed substantially as editors of papers on this Special Topic. TP drafted this editorial with feedback and approval from YL and PP.
Conflict of interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Publisher’s note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors, and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
References
Kim, S., Lee, G., Huang, W., Kim, H., Woo, W., and Billinghurst, M. (2019). “Evaluating the combination of visual communication cues for hmd-based mixed reality remote collaboration,” in Proceedings of the 2019 CHI conference on human factors in computing systems (New York, NY, USA: Association for Computing Machinery), 1–13. CHI ’19. doi:10.1145/3290605.3300403
Lee, G. A., Kim, S., Lee, Y., Dey, A., Piumsomboon, T., Norman, M., et al. (2017). “Improving collaboration in augmented video conference using mutually shared gaze,” in ICAT-EGVE 2017 - international conference on artificial reality and telexistence and eurographics symposium on virtual environments. Editors R. W. Lindeman, G. Bruder, and D. Iwai (The Eurographics Association, Montreal, Canada). doi:10.2312/egve.20171359
Piumsomboon, T., Clark, A., Billinghurst, M., and Cockburn, A. (2013). “User-defined gestures for augmented reality,” in Human-computer interaction – INTERACT 2013. Editors P. Kotzé, G. Marsden, G. Lindgaard, J. Wesson, and M. Winckler (Berlin, Heidelberg: Springer Berlin Heidelberg), 282–299.
Piumsomboon, T., Day, A., Ens, B., Lee, Y., Lee, G., and Billinghurst, M. (2017). “Exploring enhancements for remote mixed reality collaboration,” in SIGGRAPH asia 2017 mobile graphics I& interactive applications (New York, NY, USA: Association for Computing Machinery). SA ’17. doi:10.1145/3132787.3139200
Piumsomboon, T., Dey, A., Ens, B., Lee, G., and Billinghurst, M. (2019). The effects of sharing awareness cues in collaborative mixed reality. Front. Robotics AI 6, 5. doi:10.3389/frobt.2019.00005
Piumsomboon, T., Lee, G. A., and Billinghurst, M. (2018a). “Snow dome: A multi-scale interaction in mixed reality remote collaboration,” in Extended abstracts of the 2018 CHI conference on human factors in computing systems (New York, NY, USA: Association for Computing Machinery). CHI EA ’18, 1–4. doi:10.1145/3170427.3186495
Piumsomboon, T., Lee, G. A., Ens, B., Thomas, B. H., and Billinghurst, M. (2018b). Superman vs giant: A study on spatial perception for a multi-scale mixed reality flying telepresence interface. IEEE Trans. Vis. Comput. Graph. 24, 2974–2982. doi:10.1109/TVCG.2018.2868594
Piumsomboon, T., Lee, G. A., Hart, J. D., Ens, B., Lindeman, R. W., Thomas, B. H., et al. (2018c). “Mini-me: An adaptive avatar for mixed reality remote collaboration,” in Proceedings of the 2018 CHI conference on human factors in computing systems (New York, NY, USA: Association for Computing Machinery), 1–13.
Slater, M., Gonzalez-Liencres, C., Haggard, P., Vinkers, C., Gregory-Clarke, R., Jelley, S., et al. (2020). The ethics of realism in virtual and augmented reality. Front. Virtual Real. 1, 1. doi:10.3389/frvir.2020.00001
Thanyadit, S., Punpongsanon, P., Piumsomboon, T., and Pong, T.-C. (2022). Xr-live: Enhancing asynchronous shared-space demonstrations with spatial-temporal assistive toolsets for effective learning in immersive virtual laboratories. Proc. ACM Hum.-Comput. Interact. 6, 1–23. doi:10.1145/3512983
Keywords: mixed reality (MR), supernatural, enhancement, perception, interaction, collaboration
Citation: Piumsomboon T, Lee Y and Punpongsanon P (2023) Editorial: Supernatural enhancements of perception, interaction, and collaboration in mixed reality. Front. Virtual Real. 4:1132921. doi: 10.3389/frvir.2023.1132921
Received: 28 December 2022; Accepted: 16 January 2023;
Published: 07 February 2023.
Edited and reviewed by:
Antonio E. Uva, Politecnico di Bari, ItalyCopyright © 2023 Piumsomboon, Lee and Punpongsanon. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Thammathip Piumsomboon, tham.piumsomboon@canterbury.ac.nz