Editorial: Beyond audiovisual: novel multisensory stimulation techniques and their applications

Editorial on the Research Topic
Beyond audiovisual: novel multisensory stimulation techniques and their applications

1 Introduction

The field of immersive virtual reality (VR) has rapidly evolved, driven by advancements in 3D graphics and accessible high-performance hardware. It is a multimodal technology based on multisensory integration. While much of the existing VR research has focused on audiovisual cues, humans perceive the world through a variety of senses, including touch, smell, taste, and less commonly considered senses like equilibrioception. Despite their importance, these additional sensory modalities have often been overlooked in VR development. This intentionally broad Research Topic aims to address that gap. It covers novel or underutilized sensory stimulation techniques and their applications in VR. To that end, we sought to collect manuscripts that push the boundaries of traditional VR by incorporating these diverse sensory inputs to enhance the immersive experience.

The present editorial summarizes the ten selected papers on multimodal VR applications that cover themes including navigation and spatial orientation, haptic feedback and touch, gustatory and olfactory stimuli, and therapeutic applications. We highlight significant technological advancements, methodological insights, and practical applications. Collectively, they underscore the significance and potential of multisensory integration in VR. In addition, these developments demonstrate the transformative potential to create more engaging, realistic, and effective experiences. Consequently, they pave the way for enhanced VR applications across education, entertainment, and healthcare.

2 Themes and content of the Research Topic

2.1 Enhancing navigation and spatial orientation

These studies emphasize the importance of combining physical movements and sensory inputs to enhance spatial orientation and user experience.

Adhikari et al. investigated the HeadJoystick, an embodied leaning-based flying interface, and its effects on performance and user experience during a 3D navigational search task in VR. Their findings suggest that leaning-based interfaces can offer a more intuitive and engaging VR navigation experience.

Kirollos and Herdman explored how the brain resolves conflicts between visual and vestibular inputs during self-motion perception using caloric vestibular stimulation (CVS) paired with visual stimuli in VR. They found that visual and vestibular cues are integrated equally by the nervous system to reduce perceptual uncertainties.

Takahashi et al. investigated the feasibility of using percutaneous electrical stimulation (PES) of ankle tendons to induce sensations of anteroposterior and lateral body tilt. Their technique presents a non-invasive, cost-effective method to provide realistic somatosensory feedback.

2.2 Haptic feedback and touch

These studies highlight the role of touch in enhancing presence, embodiment, and performance in VR.

Boban et al. examined the influence of active haptic feedback on the perception of finger movements and the dominance of visual cues. They concluded that active haptic feedback could override visual dominance, leading to improved accuracy in finger movement perception, and thus, enhance the realism and reliability of sensory feedback.

Desnoyers-Stewart et al. investigated the impact of performer-facilitated touch on presence and embodiment in immersive VR performances. The study found that real human touch significantly enhanced participants’ sense of presence and embodiment, making the VR experience more engaging and emotionally impactful.

Sawahata et al. explored the effects of combining auditory and electrostatic force stimuli on visual field guidance in 360° VR. They demonstrated that combining these two modalities improved performance. The results prove that subtle haptic cues are effective in guiding visual attention and enhance user engagement in VR.

2.3 Olfactory and gustatory stimuli

These studies demonstrate how smells and tastes can alter perceptions, reduce stress, and enhance therapeutic outcomes. They showcase the potential of cross-modal correspondences.

Wu et al. explored the impact of ambient colours in VR on taste perception. Their findings suggest that ambient colours can indeed alter taste perceptions in VR. Results provide valuable insights for food-related VR applications and enhancing the overall user experience through cross-modal correspondence.

Lopes and Falk systematically reviewed the effectiveness of multisensory digital nature exposure, including olfactory stimuli, in reducing stress and anxiety. The review highlights the potential of integrating olfactory cues with audio-visual VR to enhance therapeutic outcomes and advocates for more standardized methodologies in future research.

2.4 Therapeutic applications

Freedman et al. presented the use of olfactory stimuli over VR exposure therapy for a combat veteran with PTSD. The integration of olfaction helped the patient recall and reprocess traumatic memories more effectively, consequently reducing the symptoms. This highlights the potential of multisensory VR in enhancing emotional processing and memory reconsolidation in PTSD treatment.

De Jesus Junior et al. tested the feasibility and effectiveness of a 3-week program with VR natural scenes, sounds, and scents. They found significant reductions in PTSD and depressive symptoms, cognitive improvements, and increased heart rate variability.

3 Broader implications

The selected studies introduce several novel hardware and software solutions that push the boundaries of multimodal VR. For instance, Adhikari et al. presented the HeadJoystick, an innovative leaning-based flying interface that enhances navigation by providing intuitive and embodied control for a more natural and engaging user experience. Similarly, Sawahata et al. explored the use of electrostatic force stimuli as a form of haptic feedback in 360° VR environments. This approach not only provides subtle yet effective guidance for visual attention but also demonstrates the feasibility of non-contact haptic feedback, opening new avenues for VR interaction design.

The papers also contribute to significant methodological advancements, offering robust experimental designs and comprehensive reviews that enhance our understanding of multisensory integration in VR. For example, Kirollos and Herdman methodological approach in investigating visual-vestibular integration can serve as a model for future studies aiming to explore the interplay between different sensory modalities in VR. Additionally, Lopes and Falk systematic review of the effects of multisensory digital nature exposure identified key methodological gaps that need to be addressed in future research.

These studies highlight the wide-ranging practical applications of multimodal VR applications, demonstrating the technology’s broad potential in therapeutic, educational, and entertainment contexts. For instance, De Jesus Junior et al. and Freedman et al. studies shed light on the power of olfactory stimuli in VR exposure therapy for PTSD for both affective regulation and memory reconsolidation, offering a powerful tool for mental health practitioners. In terms of education and entertainment, Wu et al. study on cross-modal correspondence between ambient color and taste perception in VR provides insights that can be leveraged to create more engaging and immersive culinary VR applications.

3.1 Impact on VR quality

Papers in this Research Topic consistently show that multisensory VR leads to a stronger sense of presence, embodiment, and emotional engagement, which enhances the overall quality of VR experiences. For example, Desnoyers-Stewart et al. research on performer-facilitated touch in immersive performances highlights how real human touch can enhance the sense of presence and emotional connection in VR. This is particularly crucial in performance arts or therapeutic settings. Additionally, the innovative subtle haptic feedback introduced by Sawahata et al. enhanced task performance and created a more cohesive and immersive VR experience. Such improvements are essential for VR applications in education, where a strong sense of presence and engagement can significantly enhance learning outcomes.

4 Conclusion

This Research Topic has brought together diverse studies that collectively advance our understanding of the multimodal applications of VR to provide multisensory interventions. The findings underscore the importance of considering the full spectrum of human senses when designing immersive VR experiences. They provide valuable insights into making VR more engaging, realistic, and impactful. Importantly, the contributions made by the selected papers offer a solid foundation for continued exploration and innovation in the field. In summary, they promise a future where VR can more effectively replicate and augment real-world experiences across various applications.

Author contributions

JS: Conceptualization, Validation, Writing–original draft, Writing–review and editing. OG: Conceptualization, Writing–original draft, Writing–review and editing. DB: Writing–original draft, Writing–review and editing. MM-G: Writing–original draft, Writing–review and editing. GP: Writing–original draft, Writing–review and editing. SS: Writing–original draft, Writing–review and editing.

Funding

The author(s) declare that financial support was received for the research, authorship, and/or publication of this article. JŚ is supported by XR-PAIN Grant Agreement No. 101070533 and META-TOO Grant Agreement No. 101160266. OG’s contribution was supported from the EU Horizon 2020 research and innovation programme under grant agreement No 101017746, Touchless.

Conflict of interest

Author OG was employed by Ultraleap Ltd.

The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.