Taking an Extended Embodied Perspective of Touch: Connection-Disconnection in iVR

Bringing touch into VR experiences through haptics is considered increasingly important for user engagement and fostering feelings of presence and immersion, yet few qualitative studies have explored users' iVR touch experiences. This paper takes an embodied approach–bringing attention to the tactile-kinaesthetic body–to explore users' wholistic experiences of touch in iVR, moving beyond the cutaneous and tactile elements of “feeling” to elaborate upon themes of movement and kinetics. Our findings show how both touch connections and disconnections emerged though material forms of tactility (the controller, body positioning, tactile expectations) and through “felt proximities” and the tactile-kinaesthetic experience thus shaping the sense of presence. The analysis shows three key factors that influence connection and disconnection, and how connection is re-navigated or sought at moments of experienced disconnection: a sense of control or agency; identity; and bridging between the material and virtual. This extended notion of touch deepens our understanding of its role in feelings of presence by providing insight into a range of factors related to notions of touch – both physical and virtual–that come into play in creating a sense of connection or presence (e.g., histories, expectations), and highlights the potential for iVR interaction to attend to the body beyond the hands in terms of touch.

Introduction

Haptic technologies are radically changing the digital landscape for touch interaction and communication. In immersive virtual reality (iVR) contexts this is significant given that touch is considered a key aspect in generating a sense of presence and immersion (e.g., Srinivasan and Basdogen, 1997). Touch plays a role in forging our “connection” with the world and others, placing importance on the feeling of being “connected” in virtual spaces that may foster a sense of presence and immersion, which are central to achieving engaging and stimulating VR experiences. Designing haptic experiences into iVR spaces is therefore thought to be critical in enhancing a sense of presence and connectedness. Parisi (2018) (p. 323) writes no technology “holds greater potential for realizing the core promise of haptic technology to fully embody users in computer-generated environments than virtual reality.” While various haptic technologies have been developed, they are not yet ready to deliver a realistic physical sensation of touch (Stone, 2019), and the promise of haptic technologies in creating presence (through connection) has not (to date) been realized (Parisi, 2018, 2020). Within iVR an overarching goal then is “to create the illusion of tangibility through mimetic machines, and the greater the fidelity of haptic sensation the greater the user's sense of presence in a virtual space” (emphasis added, Paterson, 2009, p.129).

Touch in mainstream iVR experiences is subject to various constraints including haptic devices, technological maturity, design and genre of the specific experience, the ways bodies are captured and represented (representation of hands, body, controllers). Given these challenges physical touch with objects or the environment are primarily limited, being mediated through controllers and, increasingly, glove-based devices. These devices draw attention to “touch” experience on the hands, raising questions around the degree to which they deliver feelings of connection or moments of disconnection from the bodily touch connections of our everyday experiences and those in VR. Just as Sheets-Johnstone (2018) highlights the lack of attention to the “body below the neck” in relation to communication, so current iVR interaction lacks attention to the body beyond the hands in terms of touch. The notion of touch in the literature embraces a broader more “embodied” idea of touch experience (e.g., Sheets-Johnstone, 1999; Paterson, 2006; Parisi, 2014), where touch is conceived as extending beyond the point of contact (that one might have with the hand on the controller), and brings attention to the whole body-in-movement with its ways of touching and “feeling.” The relevance to expanding the sense of touch through the lens of embodiment (as elaborated on in An Extended Embodied Sense of Touch) is timely given the growing awareness that: (1) users engagement with, and experiences of, virtual realities are shaped by the histories, expectations, subjectivities they bring with them (Hollett et al., 2019; Jewitt et al., 2021); (2) recreating touch is technologically complex, not least because touch consists of more than cutaneous events, it is mobile and distributed throughout the body (Parisi, 2014); and (3) touch is felt deeper than the skin where, through phenomenological investigations into embodiment, complex but intimate relationships between touching and feelings of connectedness (Paterson, 2009) are exposed and where the relationships between touch and movement are pivotal to how we come to feel the world (Sheets-Johnstone, 1999).

This paper draws on this extended “embodied” notion of touch to explore how touch is experienced in iVR and the role it plays in forging a sense of connection. Given that current iVR capacity lacks touch as we experience it in the physical world, this paper explores how touch is made manifest and experienced in iVR and the ways in which it builds a sense of connection, and moments of disconnection. Specifically, our analysis explores notions of the “tactile-kinesthetic body” (Sheets-Johnstone, 1992, 1999, 2018) and “felt proximities” (Paterson, 2009) since they recognize and emphasize how touch experiences were articulated by participants through themes of movement, control (agency), emotional responses, self-identity and body fragmentations that were identified as important in fostering a sense of connection and disconnection. In so doing it offers a qualitative companion to prior (often experimental) work, that in using a different theoretical and methodological approach both confirms finding from this work and extends them to provide deeper insight into the broad range of complex factors (rather than discreet factors) that come into VR touch interaction in creating a sense of connection or presence.

Background

Touch in iVR

When we think about touch we typically focus on tactile elements of “feeling.” Current commercially available iVR systems mostly rely on handheld controllers to interact with the environment, objects and others. Nevertheless, they are thought to add a physical dimension to the sense of connection (Jewitt et al., 2020) in iVR. This sense of touch connection sits in tension between the sensory experience of the haptic device itself in the hand, pressing buttons or rotating a joystick, and the sensorial experience–or convincingness of the illusion of touch–through the combination of haptic, visual and auditory media.

Most contemporary iVR systems provide haptic feedback through vibration, aiming to increase immersion through use of more senses, and presence through the illusion of touching something e.g., visual placement of an object on the table. However, haptics require significant development to reach a sophisticated level, where different kinds of haptics or devices are used for different sensations, and currently there is no generalized device that covers all kinds of touch experiences (Slater and Sanchez-Vives, 2016). Furthermore, the use of vibrotactile feedback may not always make sense to the user depending on the context of the interaction, for example, if I pick up a cup, I have a tactile sensation, but this is not (in the physical world) a vibration. This raises key questions around how people interpret haptic feedback in context, and to what degree the specific kind and form of haptic feedback is important in creating tactile illusions.

While there are several constraints, touch in iVR can “feel real” even if the physical experience is very degraded (Parisi, in Candy, 2019; Jewitt et al., 2021), especially given that iVR engages with the sociocultural and affective elements of touch through powerful multisensory (visual, audio, haptic) and contextual cues. The importance of illusion arises since here touch becomes a kind of “imagined” experience, where the brain is tricked to sense the experience through exploitation of perceptual gaps and provision of appropriate multisensory stimuli (Biocca et al., 2001). A feeling of connection as immediate is evoked, suggesting an ability to “feel” and make more tangible the presence of a thing or other person (Jewitt et al., 2020). Such a sensation is somewhat dependent on the success of illusion, which plays a key role in touch experiences in iVR (Price et al., 2021).

Touch being a dual sense [touching and being touched, Merleau-Ponty (2012)] also involves the digital entities with which users interact in VR. Bailenson and Yee (2008) showed that users differentiate their touch when encountered with digital representation of humans and non-human objects in a virtual environment. This finding being aligned with the idea that objects and their digital representations are inscribed in the sense that they cannot be separated from a code of behaviors and a set of emotional and physical responses (Kozel, 1994), demonstrates the importance of studying how these elements of inscribed objects come into play in the perception of touch by different users.

While it is recognized that current haptic feedback offers poor realistic touch interaction with objects, the touch experience is nonetheless critical as part of our multisensory interaction. Furthermore, touch is complex, distributed throughout the body (including movement pressure, temperature and pain) and multimodal, yet the mediating artifact – the controller, is more “singular” in its tactile capabilities (Parisi, 2014). This raises the need to understand how these touch “differences” work in iVR to facilitate a sense of connection or the times when they create a sense of disconnection. The assumption would be that if the tactile mediator is singular, lacking the complexity and comprehensiveness of the multimodal, then the potential for feelings of “disconnection” in iVR may be relatively high. Yet few studies have examined users experience of touch in iVR experience.

Presence and the Role of Touch

The term “presence” has been used since the early 90's to describe immersion, yet these terms are distinct. Immersion is defined as “the degree which the range of sensory channel is engaged by the virtual simulation” (Kim and Biocca, 2018 p.96), while presence is a “perceptual illusion” (Slater, 2018) where the visuals, sound, touch and smell are achieved through deception. Based on a number of studies, Slater and colleagues propose that the sense of presence has three dimensions: (i) place illusion, which refers to your belief that you are in the VR setting; (ii) plausibility illusion, which refers to the belief that what is happening in the virtual environment is, in fact, taking place; and (iii) embodied illusion or body ownership illusion, refers to when you perceive the virtual body as being your own body (e.g., Slater and Sanchez-Vives, 2016). While these dimensions offer an important foundation for designing for a sense of presence that emphasizes the believability of the VR space and one's presence within it, our focus on an “embodied sense of touch” (which engages with users embodied ways of touching and feeling), offers an alternative perspective that extends our understanding of presence by providing insight into a broader range of factors that come into play in creating a sense of connection/ presence.

Marsh (2003) highlights two important factors in maintaining a sense of presence. Firstly, “transparency of equipment” or in other words, the mediating artifact like the controllers are not the focus of attention in the interaction, they offer a more seamless conduit for interaction, akin to Heidegger's notion of “ready at hand” rather than “present at hand” (Dourish, 2001). Secondly, continuity refers to an experience without technical or experiential breakdowns. Of interest here is where the sense of touch plays a role in fostering or breaking this illusion, one that underpins a sense of being there or staying there (Marsh, 2003). The importance of authenticity and a sense of “real” are also embedded in the sense of presence: “when a player feels the simulated world is perceptually convincing, it looks authentic and real and the player feels that he or she is actually there” (Freina and Ott, 2015). “Providing greater realism to users could be achieved by integrating these sensory cues [touch and force] during the manipulation and interaction of virtual objects in VEs” (Ramsamy et al., 2006 p. 603). According to Paterson (2006) “using a haptic device collapses the distance between the virtual object and its representation on the screen as it becomes directly manipulable” (p. 434). Thus, “taking hold of an object close at hand produces a sense of presence of object through force feedback sensations” (p.705). If we think in terms of “collapse of distance” and “bringing closer,” then such haptic interaction in VR could contribute to a sense of connection – through combined visual and haptic (touch). However, as shown in section Touch in iVR, the integration of touch in commercially available iVR experiences is, as yet, relatively unsophisticated. While several studies have focused on the visual and aural experience in iVR, less is known about users experience of touch in iVR, and its relationship with notions of presence. These users are embodied with ways of touching and ways of “feeling” that are brought with them into VR experiences, and raises the question: does touch or the haptic always bring distance closer or collapse distance to create forms of “connection” or are there moments of disconnection in the experience because of the haptic or touch illusions?

An Extended Embodied Sense of Touch

Recent research challenges the “one size fits all” VR experience and shows that our “feeling histories” - our embodied ways of sensing, feeling and moving within and outside technology - shape the way we experience VR (Hollett et al., 2019). Given this, it is important that the work reported here, which is grounded on the idea that embodied interactions are socio-culturally shaped (Jewitt et al., 2020), examines how different individuals experience and perceive touch in VR. While Hollett et al. (2019) study explored how users with different technological expertise experience VR, they did not focus on an embodied sense of touch.

The importance of haptics/touch in iVR has been established (Touch in iVR), and there has been a recent turn toward materiality and an emphasis of embodiment in game studies as a way of achieving immersion (Parisi, 2014). This notion of embodiment appears mostly within gaming theory and VR in order to bring attention to interface designs that hold potential for fusing the physical to the audiovisual experience (e.g., Gregersen and Grodal, 2009) – but also in terms of the users developing a sense of ownership and control over virtual bodies (e.g., Kilteni et al., 2012). Other notable work has operationalised concepts such as body representation and ownership, tactile memory and attention, to understand the cognitive mechanisms of illusion in VR (Gallace and Spence, 2014). These cognitive approaches to touch and conceptualisations of embodiment provide fruitful avenues from which to unpack the significance of touch and the body to iVRs ability to connect users to the environment (making them feel presence). However, these approaches can also be considered as skewed to the architectures and textures of the VR world in doing so by (usually) partially rendering an image of the user as a static, passive, and unaffected recipient of the virtual rather than an animated part of it. Furthermore, because these approaches rest upon a distinction between mind (e.g., cognitive processes) and body (e.g., tactile interactions and experiences) these avenues for exploring the relation between embodiment and touch in VR are less equipped to accommodate the role of embodied identities, for example, that our findings suggest are efficacious in shaping users experiences of presence in iVR. Our analysis, therefore, extends this embodied sense of touch by drawing on a phenomenological and social orientation to touch (e.g., Sheets-Johnstone, 1992; Paterson, 2009; Jewitt et al., 2020), distancing our approach from cognitive aspects of touch that examine underlying cognitive processes that shape tactile experience (Gallace and Spence, 2014).

To develop an extended embodied sense of touch we engage with literature beyond its current usage in VR. The concept of embodiment stretches across disparate schools of thought, yet the term holds common the rejection of dualities (Csordas, 1990) such as mind-body and virtual-physical realities. Indeed, where the body is located in VR is a thorny question. Bodies move through virtual and physical landscapes simultaneously, touching and being touched by elements in both. The users are dually operating and present in both realities and their bodily experiences are informed by movements and sensory inputs (including touch) in both. Therefore, from moment to moment and from movement to movement they traverse the physical and virtual worlds they inhabit. In doing so one might conclude that they are clearly, but differently, connected and disconnected to both realties – yet by reflecting on these observations through an extended embodied sense of touch that rejects such dualistic frames, we come to recognize that there are not two separate bodies to analyze, but one embodied experience. We therefore cannot skew analytical primacy toward the affordances of the interface instead we understand that VR users' experiences emerge through their touching, moving, and feeling bodies–a locale where both (virtual and physical) worlds meet.

This statement is situated within an extended and phenomenologically orientated approach to embodiment and touch. In this paper, notions of the “tactile-kinesthetic body” (Sheets-Johnstone, 1992, 1999, 2018) and “felt proximities” (Paterson, 2009) speak to key themes within the data. Sheets-Johnstone (1992) highlights the primacy of movement as well as kinetic chains and affective relations in our “sensing” body (1999, 2018), that account for the “qualitative dynamic realities” (2018, p. 5) of touch and movement. These descriptions are useful in looking beyond the cutaneous and tactile elements of “feeling” and elaborate upon themes of movement, feeling and agency. Phenomenological approaches extend a view of touch that moves beyond points of contact where touch “works not as a single sense but as a broad sensory modality that utilizes the combination of a number of receptors at the cutaneous and subcutaneous levels” (Paterson, 2009, p. 130). Here the touching and feeling body is a complex morphology because it is composed as “a field of flesh that includes the somatic senses, proprioception, the vestibular (balance) sense and kinaesthesia in an assemblage that spills out beyond mere skin toward action, expression and extended sensation” (Paterson, 2009, 108). Paterson developed the notion of “felt proximities” to emphasize the relationship between feelings of closeness or “connection” that are relevant to but, in excess of, purely physical sensations of touch (e.g., cutaneous, proprioreceptive and kinaesthetic). These descriptions are useful in accounting for the emotional responses of touching and “being touched” within VR, and to accommodate participants feelings of connection and disconnection that emerged from the physical and illusionary to viscerally felt experiences.

Materials and Methods

Research Design

The study took a qualitative case study design approach since it seeks to explore and better understand the complexity of in situ touch. This sits in contrast to, yet complements, experimental work that focuses on target touch interactions, or discreet variables of touch, such as illusion or ownership. The study consisted of two interconnected parts: participants were invited to interact with two purposefully selected iVR experiences, followed by a post-interaction semi-structured interview on their participant experience.

Participants

Participants were 16 students recruited from the MA “Museums and Galleries in Education” and “Digital Media and Production” at University College London. In line with the experiences selected for this study (Climbing, Natural History Museum), participants had a mix of experience of gaming (10), climbing (11), museums (9), and digital games (5). While game and climbing experiences shaped participants' ability to interpret the VR experience (e.g., visual assessment of what objects were likely to be active, and what might be a “climbing hold”) there was little evidence of related impact on participants' touch experiences, perhaps due to the mediating role of the touch controllers in VR. Participants were notified about the study via email, followed by a researcher visiting relevant classes to inform the students about the study and collect expressions of interest and time availability.

iVR Experiences

The studies used the commercially available Oculus Rift Headset and Touch Controllers, which provide haptic feedback to the hand (via a rumble motor) activated when performing specific touch actions. The two iVR experiences used in the study were selected on the basis of touch being a central feature in the interaction context, with a focus on ‘real-world' touch experiences (handling objects, climbing) and a range of touch actions (e.g., gripping, rotating, stretching). They are single-player experiences (the norm in VR) which do not involve interaction with other social actors, yet engage participants in sensory touch experiences.

Hold the World

The Hold the World (2018) experience draws on museum objects and exhibits present in the Natural History Museum, London. The design allows users to touch and explore museum objects and exhibits (a tactile activity) which are usually not available for tactile forms of exploration in the real world. For this study participants experienced a preselected room (the Conservation Center) where they could explore two specimens: a butterfly (Queen Alexandra's Birdwing Butterfly) (Figure 1A) and a dinosaur (Stegosaurus) (Figure 1B). These objects were chosen as they differ in size, texture and form (the butterfly is small and fragile, whereas the dinosaur is big, with reconstruction at bone level), and as such they invite different ways of touching. In this experience, the user has to open a drawer on a table, take out the object, and place it in circular markers on the virtual table. A virtual David Attenborough describes the object, its history, etymology, and other relevant facts, while the object becomes suspended in mid-air. The user is then invited to pick up the object. Using the controllers they can grasp the object, pull it toward them, manipulate it, and click on specific locations on the object to get more information. They can enlarge and decrease the object's size by pulling both hands apart to “stretch” it and instigate an animation of the butterfly or dinosaur to experience it moving in the iVR space (Figure 1C). While the user presses the triggers on the controller to interact, their hand is represented in the virtual world with an outline sketch of the hand (Figure 1A). Synced to specific touch interactions, users receive (digitally mediated/produced) feedback in the form of vibrotactile stimulation provided by touch controllers, sounds and visual cues of touch from the headset. The experience lasted ~15 min¹.

FIGURE 1

Figure 1. (A) Butterfly with outline hand grasp; (B) Stegosaurus; (C) Dinosaur when animated.

The Climb

The Climb (2016)² is a popular commercial VR game which simulates the physical activity of rock climbing, where touch plays a critical role (sensing and holding the rock, supporting the body). It is advertised by Oculus as bringing “alive the excitement and thrill of rock climbing in incredible virtual reality” where “Players will scale new heights and explore stunning environments.” A headset provides a dynamic visualization of the climbing environment, that allows enables a 360 degree view (Figure 2A). Audio includes a sound of the hand “hitting-grabbing” the grips on the rock or climbing wall, sound of panting while climbing, a voice that goes “Ahhhh” when the grip on the rock is lost and the user falls, and ambient background sound (e.g., birds and bugs). Players chalked their hands throughout the game–by shaking their hand: visual feedback indicated when chalking was needed – the virtual hands reddening and or the stamina/energy bar changing from blue to red (Figure 2B) if the player did not chalk their hands regularly or maintained a grip for too long. One controller is held in each hand, using the trigger with the forefinger to make a grasping/holding action with the virtual hands (Figure 2B). Vibrotactile feedback is provided when users make initial (correct) contact with the climbing wall and again if their “energy” bar (Figure 2B) runs low.

FIGURE 2

Figure 2. (A) Climbing environment through headset; (B) Hand visualization during practice, showing the grips, clip, and energy bar (bottom right).

Participants began with the practice wall to familiarize themselves with the controls and how to interact (i.e., hand-chalking, gaining stamina) (Figure 2B). This was followed by an opportunity to take the Tourist track in ‘Zen Bay' (described as a relaxing climbing route), which offers a visually realistic environment for rock climbing, specifically in terms of height, landscape views and various rock faces. Again, the experience lasted for ~15 min³.

Procedure

A 5-min introduction to the two iVR experiences was given to provide safety information (i.e., to stop if dizzy or uncomfortable), to obtain consent from the participants, and to collect participant information (e.g., level of gaming experience etc.).

Interaction Experience

All participants took part in both experiences. Participants started with the Hold the World experience, followed by The Climb. This sequence was chosen on the basis that participants might have little or no experience in iVR. Hold the World provides an easy entry point to iVR, since participants need only to press the trigger button on the controllers to touch an object, and to point with their head to specific points in order to navigate in space (leave or enter a room).

Participants had a 2–4 min break before engaging with The Climb experience. The Climb requires more complex use of the controllers, for example to grasp or chalk hands. Participants therefore began with the tutorial to familiarize themselves with the experience and try out key actions needed to progress with the game (i.e., how to chalk your hands and how to gain stamina so as not to fall). After the tutorial participants took the “tourist experience: Zen Bay.” The interaction experiences were facilitated by a researcher, who set up the games, ensured that participants did not roam outside the iVR space, and gave guidance for interaction where needed. Throughout the session the researcher responded to participants' technical/usage questions and ensured their safety (e.g., trip risks) but did not prompt or ask questions in an effort to capture the participants “natural” interaction with the environment. Interaction was video recorded using a fixed camera at the front-side of the participant to capture their whole body interaction synced to screen capture software (OBS studio) that recorded the interaction in the virtual experience. A total of just over 8-h of composite video data (16 x average of 30 mins) was collected. Participant experiences lasted from 30 to 40 min.

Post-interaction

All participants were interviewed by another member of the research team, who was not present during their iVR experiences. Interview design for asking/prompting around digital touch is challenging, since people generally find it hard to articulate their touch experiences, and touch may not be at the forefront of their awareness when interacting in general, especially in digital environments which are highly dependent on visual and audio modalities. Given these challenges an open interview format was adopted, using topic guides and probes. In conjunction with this a number of “touch” objects were used as props for probing and stimulating participant engagement with touch in the interview. These included a chalk bag and climbing clip (related to The Climb), and a plastic toy dinosaur and rubber butterfly sourced from the NHM (related to Hold the World) (Figure 3). Participants were also invited to move and touch with the objects to demonstrate or re-enact their VR touch experiences. This experiential method is designed to generate a sensorial empathetic research encounter with a participant's sensory world and focus on participants' experiential experiences with perception at its center, a focus on memory, imagination, and affect (Leder Mackley and Pink, 2013). The interviews explored if and how participants felt when they were touching an object, their tactile sensations, their relationship to the virtual hands represented, how other modalities shaped their touch experience, and what if anything was “missing” for them, and how they knew how to touch. The interviews provided in-depth data on participants' reflections on touch to supplement the observations of touch interaction. Interviews were video recorded using a single fixed camera. The average duration of the interviews was 35 min. A total of just over 9 h of interview data was collected.

FIGURE 3

Figure 3. “Touch” objects used as sensory props in interviews: plastic dinosaur, rubber butterfly, climbing clip and chalk bag.

Data Collection and Analysis

Data comprised video data of all interactions in iVR with synced screen capture and post-interaction interviews. The two recordings (video and screen capture) of each participant interaction were combined using Lightworks, a non-linear editing system for editing and mastering digital video to produce a time-stamped composite video, which enabled us to see the link between participant actions and reactions with their in-VR actions. This captured participants interaction from differing perspectives and facilitated a multimodal and sensorial analysis, an approach that moves beyond language (or text) to draw on multisensory and bodily forms of communication. This over-arching analytical frame brings the body, technology and environment into dialogue through a simultaneous concern with the semiotic, material, sensory and experiential dimensions of touch. Touch in VR is emergent and in flux (Jewitt, 2017), often examined as discreet features in lab-based settings highlighting the benefits of exploring the intersection of the semiotic and the sensorial-experiential (Jewitt and Leder Mackley, 2018). We viewed the video data of each participant's VR experience guided by the concepts of the touching and feeling body a complex composition of materiality and tactility, the somatic senses, proprioception, the vestibular (balance) sense and kinaesthesia, to generate a descriptive account of their multimodal interaction including bodily reactions to touching in VR environment (e.g., changing bodily position or posture), kinaesthetic and felt experience, materiality, and tactile expectations. While the video observations provided key analytical openings and directions, the interview data was essential to make meaning of the participants' experiences. Thus, we undertook thematic analysis across the two different data sets, working iteratively across the data. Themes were first developed from the observational data, second explored and expanded through the interview data, and third used to revisit and review the video observational data: this iterative analytical process was valuable as it brought participants' semiotic actions, their reflections, and experiential experiences into an intimate analytical dialogue, and revealed key themes that focus on articulations of presence and connection, felt proximities and the tactile-kinaesthetic.

Findings and Discussion

The analysis provides an in depth understanding of users experience of touch in iVR, illustrating how users interpret haptic feedback in context, how the different forms of feedback are important in creating tactile illusions, the role of inscribed objects in the perception of touch by different users, and how touch forged feelings of connection and disconnection in virtual spaces. This extended embodied sense of touch in iVR is unpacked with reference to two overarching themes. The first theme focuses on articulations of presence and connection through material tactilities; body anchoring, posturing and positioning; and tactile expectations leading to moments of disconnection. The second theme focuses on felt proximities and the tactile-kinaesthetic in relation to a sense of (dis)connection through feeling (dis)connected to virtual bodies; visceral feelings of (dis)connection; fragmentation in bodies-in-movement, and exertions of bodies-in-movement. Findings are presented under these headings, followed by a discussion which draws out what these findings say in terms of touch connection and disconnection and their relationship to notions of presence.

Articulations of Presence

Material Tactilities (Bridging Between Realities)

Material tactilities or a sense of material touch helped to bridge a feeling of contact with “things” in VR. Some participants reported the significance of having this sense of touching an object through the material touching of the controller: “Touching is important between me and the object” [P10]. Many participants commented on this physical engagement with the controller being key to creating a felt touch in iVR. The sensation of holding the controllers led to a feeling that participants were grasping, holding or touching the object (Figure 4). This initial physical touch (notably the pressure of their grip) was reported as being essential to creating a connective bridge between the physical and virtual. In other words, the controller was a part of helping to generate a feeling of touching.

“I felt touch maybe because of the weight of the controller. If I grip something I feel like the ball of air was there under my controller. It felt like my hands were opening and closing. I actually forgot that I was holding the controllers. It was so natural.” [P7]

FIGURE 4

Figure 4. Hand and arm positioning when holding, touching and manipulating objects.

Despite being aware of the difference between the controller and the object, other participants still felt that “touch” sensation. For example, P1 claimed:

“I was aware that I wasn't touching the tree or the butterfly – but you get the touch sensation from that [the controller]. you can almost imagine that its that object that you are holding, but I'm also aware that its not. It hasn't got the texture of a butterfly….It felt like I was holding it [the butterfly] in my hands. I don't know why, maybe because you hold the hand device. That was the sensation as well. It did actually feel very much that there was a link between you two.” [P1]

While the physicality of the controller provided some degree of a tactile sense of touching, tactile sensory experiences related to the visual objects can result from the “filling of the perceptual gap,” widely shown in visual perception (Gestalt psychology) but also in tactile space (Kaneko et al., 2018), where meaningful interpretations are made of sensory experience. P6 mentions several times in her interview that the sensed correlation or connection between the real and the iVR environment helped to generate this sense of touch. The notion of feeling “natural” or “real” was important in participants forgetting that the touch sensation was generated through the controllers, and supported the perceptual work they needed to do, to bridge the gap between real and virtual touch. Moving beyond the controller itself, linked sound was also raised by one participant as being central to a “touch” sense of connection

“Sound made feel like touching more in climbing – when I grasped there was a sound, a connection” [P14].

This links to notions of “sonic tactility” (Cranny-Francis, 2009) and foregrounds the multi-sensory aspects implicit in material/tactile encounters that develop this notion of “bridging.” The connective bridges between physical and virtual worlds therefore consisted of more than material/tactile encounters as participants explored the virtual landscape – they were meeting points where a full range of the digital sensorium (visual, audial, narrative etc.) served to contextualize touch interactions and foster a sense of connection within the VR experiences.

For one of the participants, however, the physicality of the controllers were an “interference,” which made their hand feel more like a tool and touch too functional, their comment highlighting a desire for unmediated touch:

I felt like clasping, grabbing, rotating, moving rather than touching. Didn't really feel like touching. I could use the controller as a device to click or grab but it feels like an alien…Unreal. It doesn't quite feel human. It just feels very clunky. Your hand is not your hand it's just a thing that presses objects. it didn't feel like me but version of me that I was controlling. I was controlling objects, not feeling them. [P2]

Such a feeling of “interference” suggests that “transparency” of the controllers (the equipment) and the illusion of non-mediation is unsuccessful here, negatively affecting the participant's sense of presence or staying there (Lombard et al., 2000; Marsh, 2003).

At times participants were unable to make any kind of touch connection, leading some to experience a sense of a lack of control over their interaction and the environment itself, that led to a different form of (unwanted) connection. P15 commented in relation to the dinosaur in the museum experience,

“when it became very big and started moving I could not control (its) movement. I was moving with the dinosaur. I was following the dinosaur even though I was not actually following it. ….Because I knew that I am just standing but somehow I feel like I am moving. especially when the dinosaur came toward me I knew that I could not move my head. We were somehow connected together. we both were in that world… Even though I was looking right, the dinosaur was still there. it was like a machine was controlling me. And making me do that. wherever I look I am just gonna see the same thing. I cannot hide. And I did not like that”. [P15]

Touch mediated through the controller appeared to create “links” and forge a relationship between the participant and the object when touch was perceived by the user. The controller can, thus, serve to create touch connections by being a “stand in” for the object itself, or through participants working to make this link through imagination of the “real” and where the controller plays a compensating role for a lack of physical tactile feedback. However, when there is no touch possibility in the iVR environment participants felt a lack of control, and used their body and the surrounding environment to regain the sense of touch or connection to control the environment.

Seeking Connections: Anchoring, Posturing and Positioning

The need for sensing a bodily connectedness with the real physical space emerged for several participants [e.g., P1, P2, P11, P12, P15] all of whom had climbing or gaming experience. This was evidenced through their body posture and positioning throughout their engagement in the iVR experiences.

Participants positioned themselves, moved and touched their bodies as well as the immediate physical environment around them (including the researcher) as a way to ground their virtual experiences in their bodily/physical surroundings. The majority of these participants kept their feet firmly planted on the floor, keeping their lower bodies purposefully static, and moving only their arms and head (Figure 5). While the experiences themselves demanded the use of the arms and hands rather than walking around within the iVR space, it was nevertheless notable the degree to which participants maintained firm contact between their feet and the floor. P1 commented at the end of the interaction experience “my feet are planted on the floor,” and commented in the interview that “When the lab is there and disappears into the background that feels like dizzy,” suggesting the need to feel grounded. Perhaps as a result of unpredictable visual movements (e.g., hand moving through chair in the museum experience – no solid connection or anchor) other participants also lost physical control or experienced unbalanced proprioception when things were no longer touchable in iVR. One participant [P12] sought a touch on the arm from the researcher during the museum, seeking physical connection with another.

FIGURE 5

Figure 5. Feet planted firmly as upper body leans as she moves upper body away from the animated dinosaur.

These actions used the body and the physical properties of touch to gain control over the iVR experience, reflected in the metaphor of an “anchor”:

“When the dinosaur became big and started moving I could not control its movements. It was very confusing and disoriented… I have to have someone holding my hand… Just like as an anchor”(P15).

This bodily touch and grounding seemed to provide a way of staying connected with the safety of the physical world in order to be able to handle the tension of the VR environment. Although P11 noted that climbing “felt safe,” they kept their feet firmly on the ground, suggesting the role of this bodily stance in maintaining or supporting a feeling of connection and safety: “I felt safe – still had feet on the ground” (P11). These participants' actions realized a felt continuity of the function of touch as steadying, grounding and reassuring across physical and virtual experiences.

A few participants moved more freely with their whole bodies, without showing the need to maintain their feet firmly in one position, and did not seem to seek this form of connection. For example, P14 “is quite active: she does not remain rooted to the spot during The Climb, but rather walks around looking up, down and all around” [observation transcript P14]. Nevertheless, the majority sought some form of grounding or anchoring to the physical space. This illustrates a potential function of touch in providing a feeling of body security in iVR space, enabling interaction with the virtual experience. Without this grounding, the risk of feeling disorientated and “dizzy” increases, inhibiting users' ability to engage and interact in a iVR space. It also highlights the agentic strategies the embodied users employ to curate their experiences and to regain control over their sense of presence and to deliberately disconnect themselves from “unwanted connections.”

Tactile Expectations (Moments of Disconnection)

As alluded to in section Articulations of Presence, the notion of “reality” in relation to touch action was important. Specifically, how real the touch experience felt in terms of participants' expectations. Participants were wanting or expecting particular kinds of touch, which appeared to map to their touch experiences in the real world. A sense of “reality” facilitates a sense of connection: our “brains” have expectations about objects that are familiar from the real world and expect those objects to behave similarly in VR. “When these sensor-coupled stimuli match the brain's expectations of what the next moment will bring, then the brain will tend to treat the simulated reality as real” (Gonzalez-Franco and Lanier, 2017, p.2). Moments when that simulated reality was broken led to feelings of disconnection that were not sought out or initiated by the users as a way of reconnecting with the “outside” environment, as detailed in section Felt Proximities and the Tactile-Kinaesthetic.

Participants talked about the experience being “more realistic” when they used their hand to pull an object toward them rather than the object automatically moving toward them (as happens in many digital environments). This pull action implicitly involves touch, both in the grasp required to pull and in the muscle activity: a form of touch action mapped to real world interaction (Price et al., 2021). VR objects that shape the touch action to be similar to real world behavior in the same context are more likely perceived as “real.” For example, the way P9 opens and closes the drawer or places the dinosaur on the table is very similar to “real” behaviors related to these actions, since the object behaviors maintain or generate the illusion of resistance when they are placed on the table. The Climb was mentioned as being “more realistic” than Hold the World [e.g., P4]. This sense of reality emerges through participants' physical and emotional experience (discussed in more detail in Felt Proximities: Visceral Feelings of (dis)Connection and Exertions of Bodies-in-Movement), but was also seen in their bodily movement in the space observed during the experience itself, suggesting the central role of movement in our “sensing” body (Sheets-Johnstone, 1999, 2018). For example, P8 uses their whole body, bends down to squatting when climbing grips go lower, and reaches very high when the grips are upwards. Leans body over to peer for grips and to see the landscape: seems bodily immersed [observation transcript for P8]. P8 later reported a high sense of reality in The Climb: “feels like you're really in that place – totally difference atmosphere that with a computer. The scenery felt real.”

In contrast, objects that displayed behaviors that did not map to participants tactile expectations led to a sense of disconnection. In Hold the World the museum objects 'floated in the air' when explored in close up, while other objects (e.g., chair, table) had no solidity in relation to participant touch actions. While floating objects led to more diverse manipulative and interaction behaviors (e.g., rotating, peering underneath), they also elicited a tension between the “experienced as real” and “experienced as not real” resulting in a sense of disconnection. Participants expressed disappointment that they were unable to physically experience other objects, for example they were unable to put their hands on or lean on the table or stand or sit on the chair. Instead, their hand action moved through the visualization of these objects. In the Climb the safety clip was 'unresponsive' to touch. Being unable to grasp the safety clip resulted in feelings of disconnection from the experience, and even fear. For example, P7 noted in the interview,

“I couldn't touch the clip in the game. It represents safety, but you couldn't grab it in the game – so that was stressful.”

Moments of disconnection were also noted when participants could perform actions beyond their normal physical capability. This was notable in The Climb as reaching through the virtual body enabled reaching of a rock hold situated higher in the iVR space than one could normally reach in a physical space [e.g., P4].

Participant expectations of how things should feel and how they should respond to their touch were a perspective through which the “illusion” of VR was perceived, explored and even tested. For example, one participant banged the butterfly (a fragile object) hard onto the table to see if it would break. The sense of connection was disrupted by a break in the illusion or being “experienced as real” or when expected touch interactions were not realizable. The analysis suggests that these occasions were highly linked to felt proximities and elicit visceral and emotive responses (engagement, fear, disappointment) (see Felt Proximities and the Tactile-kinaesthetic).

Felt Proximities and the Tactile-Kinaesthetic

While movement and “feelings” are present in the themes presented above, this section attends to findings related more directly to the tactile-kinaesthetic and “felt proximities” of touch. Four key themes emerged: feeling (dis)connected to virtual bodies, felt proximities, fragmentations of bodies-in-movement and exertions of bodies-in-movement.

Feeling (Dis)connected to Virtual Bodies

There was evidence of moments where participants became aware of a felt sense of their own body in the virtual experience, demonstrating how their experience in the iVR environment shaped their sense of awareness of “me-ness” within the virtual space, and speaks closely to the body ownership illusion (Kilteni et al., 2012). This relationship to self was evident in both observation from the video interaction sessions and from the post-interaction interview, and primarily emerged in relation to the virtual hands in both experiences, as well as the voice while falling in The Climb.

The virtual body was represented by a visualization of the hands in both iVR experiences (Figure 6), but in Hold the World hands were described as “just outlines” or “lines.” Nonetheless the hands were a key mediator of identity to self and provided a point of connection and extension of the physical sensation of touch (i.e., interaction with the controllers, body and environment) for some participants. Despite the reduced representation of the virtual hands (reduced graphic realism), being able to see “their hands” when interacting in iVR promoted and supported (and to some extent, instructed) participants to touch. In some cases participants attributed them as being their own. For example, P5 commented: they are “just like real…. I didn't notice the hand was in the virtual world.” This was especially effective when participants could map their physical hand to “their” virtual hand through its appearance, touch actions and sound (feeling the sound near to their physical body). Several studies investigating the importance of the graphic realism of the hands in VR demonstrate how it modulates ownership (e.g., Lin and Jörg, 2016). Interestingly, participants have been shown to experience body ownership with virtual counterparts that differ from their own, for example, in skin color (Farmer et al., 2012) and body size (Normand et al., 2011). Our study suggests how these visual representations of the fingers or hands shaped touch forms of engagement.

FIGURE 6

Figure 6. Looking at hands in iVR linked to their own.

In The Climb identification with the hands supported a sense of touch:

VR hands were quite similar to physical hands in terms of the energy limits…Virtual energy present in the climbing experience conveyed a sense of using energy – I felt the energy. It helped to feel touch. [P6: The Climb]

There was no difference between VR hands and my physical one. They just came together. Hands during climbing look like physical hands but in Museum experience there is no hand, it's just a line, outline of the hand. When VR and physical hands came together - physically it felt easier to climb. [P5: The Climb]

More strikingly, they elicited a sense of physiological change. For example, P13 stated that when their hand became red in The Climb (indicating the need to chalk them), they started to worry about the state or health of their own hand! The virtual hands supported a sense of connection in other ways, for example, using the outline of the virtual hand in Hold the World to look at the size of the butterfly – mapping this to the hand size – making a connection between the hand (albeit virtual) and the object. Furthermore, this sense of the hands (particularly in The Climb) being real also brought a feeling of “being in control” [P16] fostering a sense of connection.

However, participants' sense of touch and presence was disrupted to different extents when they could not map the representation of the virtual hand to their physical hand. Several participants encountered a sense of disconnection from their experience with their “virtual” hands, typically in The Climb. During interaction several participants looked closely at their virtual hands at the beginning of the practice session. This was often elicited when the controllers were positioned in the “wrong” hand (P8, P9, P10, P12, P13). Thus, the right hand looked like the left and vice versa. For example, P9: holds hands in front of face and looks from one hand to the other, back and forth. Some participants showed confusion around “their” hands, while other immediately noticed the disjunction and swapped the controllers.

In interview several participants commented on a feeling of disconnection in terms of look and feel of the virtual hands (P2, P9, P12, P13, P14, P16).

I could use the [VR] hand as device to click or grab but its alien. It doesn't quite feel human. When you are in real space you don't think how to use your hands, how to specifically do sort of things. You just do. It didn't feel like me but a version of me that I was controlling. It didn't feel natural. I was controlling objects, not feeling them. [P2]

Particularly in The Climb, the virtual hands were described as plastic, a bit slimy, not looking like real skin. As a result, some participants did not identify with them: they are “not like mine,” bigger like a man's hands; they “didn't feel like my hands” as you “do not move your hands or fingers in the way that the virtual hand behaves.”

Hands – didn't look like real skin – looked plastic, a bit slimy. and a distant sound [when the hands moved to grasp onto a rock].It was strange because the voice was a woman's voice, when you fall you hear a scream. I was hearing that voice. Was it my voice? It was a bit strange. First I thought it [the person on screen] was me and then I realized it wasn't. But it was me. But I was a woman. Then I realized I am playing a character. The hands kinda seemed masculine I guess [P2]

While these perceptions derive from the “look” of the hands, they impacted on the “feel” of the interaction, and the degree to which body ownership was achieved. This participant explicitly noted how the disconnect between the virtual hands and their own identity led them to feel like they were playing a different role - a different character - in the narrative, thus resulting in them having a different relationship to the experience than if they themselves felt present in the experience. Thus, not only was there a disconnect, but that disconnect led to a remaking of the experience as “playing a character,” and in this sense the participant finds a new way to connect to the VR experience.

In The Climb hands are depicted as opening and closing as they grasp the rock, yet the physical hands remain gripped around the controller using individual fingers to press appropriate buttons on the controller (Figure 7). Thus, functionally the grip actions were described as being different on the virtual hands, since “whatever you do with your hands, if you click you will connect to the rock” [P9]. In addition, P9 noted that chalking the hands was hard to achieve in a timely manner since you “cant feel that your hands are sore or need chalk” (emphasis added), and it was harder to take this information from a visual cue than a tactile one. Despite this, the analysis illustrates the importance of identifying with virtual body elements that relate to their own touch interaction (cf Kilteni et al., 2012) to develop a sense of connection.

FIGURE 7

Figure 7. Gripping controller, while virtual hand is reaching with palm open prior to grasping the rock.

Felt Proximities: Visceral Feelings of (Dis)connection

Visceral sensations of emotion were noted across both experiences. The point of contact or sense of touch elicited a sense of higher engagement with the narrative of the experience, touch thus providing a point of emotional connection, or establishing a relationship with the objects. The possibilities of touching and being touched by the virtual landscape (and the characters/objects presented in it), even when the touch encounters were degraded and unrealistic, framed felt proximities, or a sense of closeness and connection to the virtual (Figure 8). For example, P5 noted in relation to the museum experience that:

“I get closer to the objects emotionally. It is not something separate from my life. Because I can touch it, I can establish a relationship through the point of contact.”

FIGURE 8

Figure 8. Engaging the body to take a closer look – leaning in/peering underneath – at the dinosaur and butterfly.

This connection made her feel more engaged with the narrative through the ‘in the moment connection', which is different from a more removed stance when reading a label in the museum.

Engagement with the materiality of the objects has been linked to establishing social, cognitive connections (Chatterjee and Noble, 2013), and emotionally meaningful experiences with those objects that can provide access to representations of social, cultural and historical meanings (Jewitt and Price, 2019).

Another example occurred while interacting with the dinosaur in the museum experience. At one point in the experience, the dinosaur becomes animated and moves around. At this point it appears to move toward the participant, with its spiked tail swinging, it turns and the tail appears to come close to the participant's face. Several participants made a deliberate step backwards away from the dinosaur, while leaning their body backwards on their feet [P9, P11, P12, P13, P15]. This movement suggests a visceral sensation of the presence and nearness of the dinosaur, and awareness of their own bodies' spatial relationship to it. One participant raises her arms as if trying to protect her face [P9] (Figure 9); another exclaimed - Oh no! - in horror when the animated dinosaur swung its tail; yet another described getting “goosebumps” when the dinosaur started moving around [P11]. This response was evident in the interaction as well as being confirmed in interview:

P5 moved her body slightly when the dinosaur became animated. Up until then she has stood very still (except for small hand movements). Starts stepping backwards and sideways moving away from the animated image when the tail is swinging, as if to avoid it touching her.[observation transcript]

FIGURE 9

Figure 9. Shielding face from dinosaur's swinging tail; cringing on falling from rock face.

In the follow up interview she states:

I wanted stay back from the moving dinosaur. I was a little bit scared, I wanted to move away.

In The Climb several participants talked about feelings of nervousness. This was typically related to emotions of fear of falling: the height when looking down, the changing color of the hands in relation to loss of energy (leading to falling), and the panting sound effects.

“Once I got really immersed in it and when I got higher I didn't want to quit. And my fear actually spiked up because if I fall I actually fall” [P6].

Observation of interaction revealed moments where participants visibly cringed when they (virtually) fell from the rock face (Figure 9). The speed and rhythm of touch was also described as creating a sense of urgency and danger [P1]. These “felt proximities” were evident through observing participants' bodies and movement: “emotions move us to move in ways that are dynamically congruent with the dynamics of the feelings we are experiencing” (Sheets-Johnstone, 2018 p. 86). The sense of connection with someone or something is understood to work on an emotional/intellectual level (Jewitt et al., 2020). While not all participants reported feeling strong emotions, such as fear, emotional engagement suggests a heightened sense of connection, fostering a sense of presence. Touch as a “felt proximity” was found to be closely linked to emotion, thus bringing about a sense of connectedness and presence.

Fragmentations of Bodies-in-Movement

While participants moved, stretched, twisted in the iVR experiences, particularly in The Climb, and spoke of their bodily engagement, they simultaneously commented on the absence and partiality of their physical and virtual body. They spoke in terms of “missing” and “lack” (e.g., of body weight, legs, feet) and the fragmentation of their body:

I found the hands [climbing] strange, I think the fact that you don't have the arm makes it funny. you don't know how long your arm is. [P4]

“[climbing] is very physical, and you really feel the body in that. The body and brain are together doing things. That experience doesn't feel like that at all” [P2]

The body is not represented beyond the hands in the two iVR experiences and physical movement was largely focused on the arms, shoulders and upper torso (Figure 10). The hands are central to both virtual experiences (and indeed most VR experiences) in terms of their mode of interaction and representation. Indeed, the importance of perceiving the hands/arms as being connected to a body in VR has been shown to increase a sense of embodiment (Perez-Marcos et al., 2012). The findings here show how participants' conceptualization of virtual touch is also situated within a sense of an integrated bodily sense of touch – not just the hands.

FIGURE 10

Figure 10. Body positioning for low and high reaching in The Climb.

Several participants experienced this “fragmentation of their body” in the interaction. While P4 commented on Hold the World as having “a lack of bodily presence – you were like floating, disconnected from your body,” a sense of body fragmentation was most notable in “The Climb,” an activity that, in a physical context, involves the whole body. In the iVR experience participants experienced a predominant awareness of their upper body. Making a virtual climb was dependent on reaching with their arms and grasping with their hands (the controllers) to grip rock holds, with occasional body leaning or stretching to reach more distant grip holds. Observation of interaction showed the majority of participants moving their arms between waist and head height, using rapid smallish horizontal movements within a width space just beyond their shoulder breadth, and upward movements followed by pulling down movements. This typical (relatively limited) range of arm movements was sufficient to achieve the actions needed in the iVR experience to reach grips and pull the virtual body up or across the virtual rock face.

The dominant awareness brought to the upper body was noted in the interviews, for example:

“it felt like having half a body” [P2]

“I felt the upper body more” [P6]

“Where are my legs? I can't feel my legs [P14]”

“you don't use feet as part of the experience – normally you have to consider where you're putting your feet as well as your hands” [P7]

A few participant actions suggested more “effortful” engagement in their climbing actions. For example, P12 pulls his arms down slowly as he climbs (looking more effortful than others who moved quickly); P13 pushes off with 2 hands, followed by more obvious pulling motions with one hand methodically after another, and stretching regularly; and P14 displayed strong, almost exaggerated arm movements.

Overall, the virtual climb emphasized arms and hands in interaction, while in a physical environment climbing is more about the legs. This highlights the technical focus on devices that engage hands and arms in interacting in these particular iVR spaces – here the controller providing the central mediator of interaction. This body fragmentation elicited explicit feelings of disconnection, where the “rest of the body” was “missed” [P4]. This highlights a tension in design for body-engagement in generating a sense of “presence” that is framed by the particular narrative of experience. In the Climb, not being “fully present” as a complete body became problematic in terms of feeling connected to the experience when some participants missed parts of their body.

Exertions of Bodies-in-Movement

The iVR experiences brought the participants awareness to their bodies, felt muscular tension, bodily signs of physical exertion, sweating, and tiredness were a part of their overall sense of virtual touch experience (Figure 11):

I thought that I won't feel physicality but then I saw how my body was literally putting effort into that…My muscles were tense. I've finished sweaty. [P4]

FIGURE 11

Figure 11. Bodily movements and positioning showing exertion in The Climb.

These physiological body responses and sensations were reported in the post-interviews. Typically this was linked to “The Climb” which involved more body movement (even if primarily the arms) – physical effort - across the experience, and is psychologically or experientially linked to a more physical experience than observing objects in a museum. Participants noted how the experience was “physically demanding” [P10], where “you felt the need to use your muscles” [P4] and “experience tiredness” [P11, P23]: the “measure of tiredness was amazing” [P11]. Some particularly noted how tired their muscles felt, “even though they weren't needed” [P4].

“Obviously, it's not the same amount of energy as in a real-time scenario but it was quite interesting to see how my muscles were still compressed while doing the movements” [P4]

Several participants described noticing their bodies tensing, their muscles were tense, and that they became sweaty, or finished the experience sweating, both generally and on their hands.

In contrast, some participants did not experience such physiological sensations [P2, P9, P10, P16] or made links to tactile experiences that are present in real world climbing contexts.

“[climbing] is very physical, and you really feel the body in that. The body and brain are together doing things. That experience doesn't feel like that at all” [P2]

Others commented that they did not get physically tired, or sore muscles from stretching or pulling their weight up the rock face. They found climbing much easier in iVR, partly since the “roughness of the rock in real life makes climbing tougher…. You feel your harness all the time, and your shoes are very tight” [P16], and partly because the activity demanded use of only the upper body.

Overall, these physiological responses were unexpected and brought participants' awareness to specific aspects of their body's physicality. The physicality of the climbing experience, such as muscle sensations, in the interaction, enhanced their feelings of connection between physical body and the iVR experience [P4].

Key Factors Shaping Connection and Disconnection

The analysis shows how both touch connections and disconnections emerged though material forms of tactility (the controller, body positioning, tactile expectations) and through “felt proximities” and the tactile-kinaesthetic experience. Three key factors emerged that influence connection and disconnection: a sense of control or agency; identity; and bridging between the material and virtual.

Control/Agency

The sense of touch emerged as important in participants feelings of agency and control within the VR environment and influenced the degree participants felt connected or disconnected in their experience. A sense of control emerged in relation to touch in both the physical and the virtual space. The “anchoring” of the body to the physical space illustrates a potentially key function of touch in providing a feeling of body “connection” or security in the iVR space, and that provides an important foundation for enabling positive interaction with the virtual experience. Without this grounding, the risk of feeling disorientated and “dizzy” increases, inhibiting users' ability to engage and connect with the iVR experience. Furthermore, when an “expected” touch interaction in the iVR environment was missing, participants felt a lack of control over their actions, often leading to a sense of disconnection. This occurred when object behaviors did not map to tactile expectations: participants could not “touch” certain objects in the iVR space, like being unable to lean on the table in Hold the World or the safety clip in The Climb. It also occurred in relation to the sensed body as a whole, or the fragmentation of the body and the lack of feeling of the body, for example, when the arms and hands were foregrounded in the touch experience (but not the feet and legs) in “The Climb.” At times like these participants used their bodies and the surrounding environment to regain the sense of touch or connection to control the environment. This highlights the agentic strategies embodied users employ to curate their experiences and to regain control over their sense of presence. In designing touch for iVR consideration needs to be given to the sense of agency or control that touch provides and the implications of this at moments where touch connections are missing or lost.

Identity

The sense of identity of the tactile self that participants experienced was important in eliciting feelings of connection. Where there were mismatches in identification with self or body movement, a sense of disconnection was experienced. In line with previous work (e.g., Lin and Jörg, 2016) the findings showed how ownership, or identification of self, through the hands was central to providing a sense of connection of self being embodied within the environment. Building on this the findings also showed that when participants attributed the virtual hands as their own, their touch experience was enhanced, and contributed to their wider bodily kinaesthetic experiences. However, when this mapping was disrupted through the look and the “feel” of the hands, then a sense of disconnection from the experience occurred. If the virtual body (or body part) is not realistic enough or does not follow the same movements or same body semantics of a participant, there is no sense of being “embodied” (Slater and Sanchez-Vives, 2016) leading to disconnection from the virtual environment, and less or no feeling of presence. The analysis illustrates the importance of an iVR design that fosters users ability to identify with the virtual body elements that relate to their own touch interaction (cf Kilteni et al., 2012). Despite previous work (e.g., Farmer et al., 2012), the size, gender, ethnicity of the hands impacted not only on the “embodiment illusion, but also on the work of imagining and drawing on previous experience needed by the user to feel “touch connection” – a key factor in achieving a sense of presence. The notion of identity has an important relationship with “felt proximities” too, since an embodied notion of self in the experience is more likely to elicit visceral sensations of emotion in relation to the narrative and provide “in the moment connection” (Kilteni et al., 2012). While this may bring a stronger sense of connection, the sense of bodily touch was conversely disrupted, or broken, in narratives that required the use of the whole body beyond the hands and arms. Specifically, in The Climb, this led to a noted absence and conscious awareness of the fragmentation of their physical and virtual body.

Our findings suggest that the role of embodied identities are efficacious in shaping users experiences of presence in iVR. Designing for identity is a key consideration, since VR can change not only where you are but who you are (Kilteni et al., 2012). If the experience design aims to engage users as themselves, not only is a coherent mapping between self-image and virtual image important, but also the coherent sense of body in movement in relation to the specific experience. For instance, “The Climb” lacked use of legs and feet, an instrumental part of a typical climbing experience. Despite this, the findings also suggest how users actively sought to make connections in terms of identification of self in the VR space, as self or as another imagined character.

Bridging Between Material and Virtual Presence

Leading on from the point above, the findings showed how participants actively worked to bridge their touch experiences or re-navigate them in order to develop a sense of connection. This active role of the embodied users was instrumental in the bridging between their material and virtual touch presence. The sense of connection and disconnection for some (e.g., P2) switched between a physiological sense of connection (muscular and felt) and a more “felt” sense of connection or disconnection through identity (“didn't feel like me”) and imagination (P9 dream experience).

Material tactilities, or a sense of material touch, also helped to bridge a feeling of contact – or connection–with “things” in VR. The physicality of the controller provided some degree of this tactile sense of touching, but in combination with participants' imagination – working as a bridge - led to more nuanced tactile sensory experiences specifically related to the visual objects. However, the emotion/visceral sensory experience – or felt proximities - also played a role in bridging between material and virtual touch presence. The connective bridges between physical and virtual worlds therefore consisted of more than material/tactile encounters as participants explored the virtual landscape – they were meeting points where a full range of the digital sensorium (visual, audial, narrative etc.) and movement (Sheets-Johnstone, 1999) served to contextualize touch interactions and foster a sense of connection within the VR experiences.

Bringing attention to the tactile-kinaesthetic body spoke to an embodied sense of touch that is sensitive to the whole body-in-movement (see body fragmentations and exertions). Insights of an embodied sense of touch as developed by Sheets-Johnstone and Paterson spoke to the participants experiences through accounting for the agentic, felt, dynamic and affective dimensions of embodied touch. This analysis offers the state-of-the-art in terms of user experience of touch in VR, extending our understanding of presence in iVR that moves beyond illusion by providing insight into a range of factors related to notions of touch – both physical and virtual - that come into play in creating a sense of connection or presence (e.g., histories, expectations). While the tactile mediator (the controller) is singular, lacking the complexity and comprehensiveness of the multimodal (Parisi, 2014), the analysis in this paper highlights a more complex and broader range of factors that shape a sense of connection and disconnection. The embodied sense of touch came through in participants interactions and reported experience, and analysis of this showed how those touch experiences shape the feeling of connection or disconnection. For example, a sense of the “collapse of distance” and “bringing closer” to the experience was mediated through haptic interaction (material tactilities), “felt proximities” (emotion, exertion), and identity. The analysis also showed how participants played an active role in developing a sense of connection, an aspect that came through strongly as being important for participants in the experience. When feelings of disconnection emerged, they sought to bridge gaps in their material expectations, they sought control through embodied stances within the physical space and they sought to maintain their identity or take on the identity of another when a disconnect of body occurred.

Conclusion

Creating connection is important in maintaining a sense of presence in iVR experiences. Focusing on users' experiences of touch in iVR this qualitative study makes a contribution to iVR by providing detailed empirical analysis of how both touch connections and disconnections emerged though the material forms of tactility, “felt proximities,” and tactile-kinaesthetic experiences shape the sense of presence, and the ways in which these three key factors are brought into play at moments of experienced disconnection, thus shaping a sense of presence. It also makes a conceptual contribution to the field through the use of an extended notion of touch, which deepens our understanding of its role in feelings of presence by providing insight into a range of physical and virtual factors related to notions of touch in creating a sense of connection or presence. Taking an embodied perspective of touch -accounting for our socio-culturally shaped “feeling histories,” materiality, notions of the “tactile-kinaesthetic body” (Sheets-Johnstone, 1992, 1999, 2018) and “felt proximities” (Paterson, 2009), moves beyond the cutaneous and tactile elements of “feeling” to also elaborate upon themes of movement and kinetics. VR users' experiences emerge through their touching, moving, and feeling bodies - a locale where both (virtual and physical) worlds meet. While prior work shows how users previous experience, expectations and ways of feeling influence a sense of presence in VR (e.g., Farmer et al., 2012; Lin and Jörg, 2016; Gonzalez-Franco and Lanier, 2017), these are lab-based studies examining discreet factors of touch. Our qualitative study elaborates on those findings within a wider user interaction context, where these different factors all come into play together. In so doing it extends existing work to provide deeper insight into the broad range of factors that come into play in creating a sense of connection/ presence (e.g., how histories, expectations come into play), shows the complexity of perceived touch and touch experience in VR, and illustrates how connection is re-navigated or sought at moments of experienced disconnection. In doing so, this paper highlights the potential for iVR interaction to attend to the body beyond the hands in terms of touch and considerations in the design of tactile experiences in VR.

Data Availability Statement

The datasets presented in this article are not readily available because the data is all video data, which is personally identifiable, since it is not anonymised. Further enquiries should be directed to Carey Jewitt, as PI of the project and owner of the data sets: c.jewitt@ucl.ac.uk.

Ethics Statement

The studies involving human participants were reviewed and approved by UCL Institute of Education research ethics committee. The patients/participants provided their written informed consent to participate in this study. Written informed consent was obtained from the individual(s) for the publication of any potentially identifiable images or data included in this article.

Author Contributions

SP: conceptualisation, methodology, data collection, formal analysis, and writing – original draft preparation. CJ: conceptualisation, methodology, data collection, writing – review and editing, and funding acquisition. DC: data curation, formal analysis, and writing – review. NB: data collection and writing – review and editing. NY: conceptualisation, methodology, data collection, and writing – review. All authors contributed to the article and approved the submitted version.

Funding

This research was undertaken as a part of the InTouch project, a European Research Council Consolidator Award (Award Number: 681489).

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Acknowledgments

We thank all the participants who took part in the study, Ashley Lee for her invaluable help in collecting the data; and Abel Drew for his support with the study.

Footnotes

1. ^https://www.youtube.com/watch?v=kFasefT-zrY&t=83s

2. ^https://www.youtube.com/watch?v=er1aUDbyUXo

3. ^https://www.youtube.com/watch?v=er1aUDbyUXo

References

Bailenson, J. N., and Yee, N. (2008). Virtual interpersonal touch: haptic interaction and copresence in collaborative virtual environments. Multimed. Tools Appl. 37, 5–14. doi: 10.1007/s11042-007-0171-2

CrossRef Full Text | Google Scholar

Biocca, F., Kim, J., and Choi, Y. (2001). Visual touch in virtual environments: an exploratory study of presence, multimodal interfaces, and cross-modal sensory illusions. Presence 10, 247–265. doi: 10.1162/105474601300343595

CrossRef Full Text | Google Scholar

Candy, B. (2019). Keeping Virtual Reality Environments Harassment Free. Available online at: https://onezero.medium.com/keeping-virtual-reality-environments-harassment-free-3b9d4e5d3416 (accessed April 12, 2021).

Google Scholar

Chatterjee, H. J., and Noble, G. (2013). Museums, Health and Wellbeing. Ashgate: Farnham.

Google Scholar

Cranny-Francis, A. (2009). Touching film: the embodied practice and politics of film viewing and filmmaking. Senses Soc. 4, 163–178 doi: 10.2752/174589309X425111

CrossRef Full Text | Google Scholar

Csordas, T. J. (1990). Embodiment as a paradigm for anthropology. Ethos 18, 5–47. doi: 10.1525/eth.1990.18.1.02a00010

CrossRef Full Text | Google Scholar

Dourish, P. (2001). Where the Action Is: The Foundations of Embodied Interaction. Cambridge, MA: MIT Press. doi: 10.7551/mitpress/7221.001.0001

CrossRef Full Text | Google Scholar

Farmer, H., Tajadura-Jiménez, A., and Tsakiris, M. (2012). Beyond the colour of my skin: how skin colour affects the sense of body-ownership. Conscious. Cogn. 21, 1242–1256. doi: 10.1016/j.concog.2012.04.011

PubMed Abstract | CrossRef Full Text | Google Scholar

Freina, L., and Ott, M. (2015). “A literature review on immersive virtual reality in education: state of the art and perspectives,” in eLearning and Software for Education (Bucharest), p. 133–141.

Google Scholar

Gallace, A., and Spence, C. (2014). In Touch With the Future: The Sense of Touch From Cognitive Neuroscience to Virtual Reality. Oxford: Oxford University Press.

Google Scholar

Gonzalez-Franco, M., and Lanier, J. (2017). Model of illusions and virtual reality. Front. Psychol. 8:1125. doi: 10.3389/fpsyg.2017.01125

PubMed Abstract | CrossRef Full Text | Google Scholar

Gregersen, A., and Grodal, T. (2009). “Embodiment and interface,” in The Video Game Theory Reader 2, eds B. Perron, and M. J. P. Wolf (New York, NY; Abingdon: Routledge).

Google Scholar

Hollett, T., Luo, S., Turcotte, N., Ramsay, C., Stubbs, C., and Zidik, Z. (2019). Moments of friction in virtual reality: how feeling histories impact experience. E-Learn. Digit. Media 17, 56–77. doi: 10.1177/2042753019876043

CrossRef Full Text | Google Scholar

Jewitt, C. (2017). “Towards a multimodal social semiotic agenda for touch,” in Advancing Multimodal and Critical Discourse Studies: Interdisciplinary Research, eds S. Zhao, E. Djonov, A. Bjorkvall, and M. Boerils (Routledge: London). p. 79–93. doi: 10.4324/9781315521015-6

CrossRef Full Text | Google Scholar

Jewitt, C., Chubinidze, D., Price, S., Yiannoutsou, N., and Barker, N. (2021). Making sense of digitally remediated touch in virtual reality experiences. Discourse Context Media 41:100483. doi: 10.1016/j.dcm.2021.100483

CrossRef Full Text | Google Scholar

Jewitt, C., and Leder Mackley, K. (2018). Methodological dialogues across multimodality and sensory ethnography. Qual. Res. 19, 90–110. doi: 10.1177/1468794118796992

CrossRef Full Text | Google Scholar

Jewitt, C., and Price, S. (2019). Family touch practices and learning experiences in the museum. Senses Soc. 14, 221–235. doi: 10.1080/17458927.2019.1619316

CrossRef Full Text | Google Scholar

Jewitt, C., Price, S., Leder-Mackley, K., Yiannoutsou, N., and Atkinson, D. (2020). Interdisciplinary Insights for Digital Touch Communication. Springer Briefs in Human Computer Interaction. doi: 10.1007/978-3-030-24564-1

CrossRef Full Text | Google Scholar

Kaneko, S., Kajimoto, H., and Hayward, V. (2018). “A case of perceptual completion in spatio-temporal tactile space,” in Haptics: Science, Technology, and Applications. EuroHaptics 2018. Lecture Notes in Computer Science, Vol. 10893, eds D. Prattichizzo, H. Shinoda, H. Tan, E. Ruffaldi, and A. Frisoli (Cham: Springer). doi: 10.1007/978-3-319-93445-7_5

CrossRef Full Text | Google Scholar

Kilteni, K., Groten, R., and Slater, M. (2012). The sense of embodiment in virtual reality. Presence 21, 373–387. doi: 10.1162/PRES_a_00124

CrossRef Full Text | Google Scholar

Kim, G., and Biocca, F. (2018). “Immersion in virtual reality can increase exercise motivation and physical performance,” in International Conference on Virtual, Augmented and Mixed Reality (Cham : Springer), p. 94–102. doi: 10.1007/978-3-319-91584-5_8

CrossRef Full Text | Google Scholar

Kozel, S. (1994). Spacemaking: Experiences of a Virtual Body. Available online at: http://www.art.net/~dtz/kozel.html (accessed April 12, 2021) (more extended version available In Dance Theatre Journal 11, p. 12–13, 31, 46–47).

Google Scholar

Leder Mackley, K., and Pink, S. (2013). From emplaced knowing to interdisciplinary knowledge: sensory ethnography in energy research. Senses Soc. 8:335–353 doi: 10.2752/174589313X13712175020596

CrossRef Full Text | Google Scholar

Lin, L., and Jörg, S. (2016). “Need a hand? How appearance affects the virtual hand illusion,” in Proceedings of the ACM Symposium on Applied Perception, SAP (Anaheim, CA), 69–76. doi: 10.1145/2931002.2931006

CrossRef Full Text | Google Scholar

Lombard, M., Ditton, T. B., Crane, D., Davis, B., Gil-Egui, G., Horvath, K., et al. (2000). Measuring presence: a literature-based approach to the development of a standardized paper-and-pencil instrument. Presence 2000: The Third International Workshop on Presence (Delft).

Google Scholar

Marsh, T. (2003). “Staying there: an activity-based approach to narrative design and evaluation as an antidote to virtual corpsing,” in Being There: Concepts, Effects and Measurements of User Presence in Synthetic Environments, eds G. Riva, F. Davide, W. A. IJsselsteijn (Amsterdam: IOS Press), 85–96.

Google Scholar

Merleau-Ponty, M. (2012). Phenomenology of Perception (D.A. Landes, Trans.). London; New York, NY: Routlege. doi: 10.4324/9780203720714

CrossRef Full Text | Google Scholar

Normand, J.-M., Giannopoulos, E., Spanlang, B., and Slater, M. (2011). Multisensory stimulation can induce an illusion of larger belly size in immersive virtual reality. PLoS ONE 6:e16128. doi: 10.1371/journal.pone.0016128

PubMed Abstract | CrossRef Full Text | Google Scholar

Parisi, D. (2014). Reach in and feel something: on the strategic reconstruction of touch in virtual space. Animation 9, 228–244. doi: 10.1177/1746847714527195

CrossRef Full Text | Google Scholar

Parisi, D. (2018). Fingerbombing, or “Touching Is Good”: The Cultural Construction of Technologized Touch in Eds Mika Elo and Miika Luoto Figures of Touch: Sense, Technics, Body. Helsinki: University of Helsinki Press.

Google Scholar

Parisi, D. (2020). Expanding the Universe of Haptics. Blog post August 12th2020. https://medium.com/lofelt/expanding-the-universe-of-haptics-8c6189636e2f

Google Scholar

Paterson, M. (2006). Feel the presence: technologies of touch and distance. Environ. Plan. D Soc. Space 24, 691–708 doi: 10.1068/d394t

CrossRef Full Text | Google Scholar

Paterson, M. (2009). Haptic geographies: ethnography, haptic knowledges and sensuous dispositions. Prog. Hum. Geogr. 33, 766–788. doi: 10.1177/0309132509103155

CrossRef Full Text | Google Scholar

Perez-Marcos, D., Sanchez-Vives, M. V., and Slater, M. (2012). Is my hand connected to my body? The impact of body continuity and arm alignment on the virtual hand illusion. Cogn. Neurodyn. 6, 295–305. doi: 10.1007/s11571-011-9178-5

PubMed Abstract | CrossRef Full Text | Google Scholar

Price, S., Jewitt, C., and Yiannoutsou, N. (2021). Conceptualising touch in VR. Virtual Real. doi: 10.1007/s10055-020-00494-y. [Epub ahead of print].

CrossRef Full Text | Google Scholar

Ramsamy, P., Haffegee, A., Jamieson, R., and Alexandrov, V. (2006). “Using haptics to improve immersion in virtual environments,” in Computational Science – ICCS 2006: 6th International Conference, Reading, UK, May 28-31, 2006. Proceedings, Part II, eds V. N. Alexandrov, G. D. van Albada, P. M. A. Sloot, and J. Dongarra (Berlin, Heidelberg: Springer Berlin Heidelberg), 603–609. doi: 10.1007/11758525_81

CrossRef Full Text | Google Scholar

Sheets-Johnstone, M. (1992). Giving the Body its Due. Albany, NY: State University of New York Press.

Google Scholar

Sheets-Johnstone, M. (1999). The primacy of movement. Advances in consciousness research series. Philadelphia: John Benjamins. doi: 10.1075/aicr.14

CrossRef Full Text | Google Scholar

Sheets-Johnstone, M. (2018). “If the body is part of our discourse, why not let it speak? Five critical perspectives,” in Surprise: An Emotion?, eds N. Depraz, and A. Steinbock (Springer), 83–95. doi: 10.1007/978-3-319-98657-9_5

CrossRef Full Text | Google Scholar

Slater, M. (2018). Immersion and the illusion of presence in virtual reality. Br. J. Psychol. 109:431. doi: 10.1111/bjop.12305

PubMed Abstract | CrossRef Full Text | Google Scholar

Slater, M., and Sanchez-Vives, M. V. (2016). Enhancing our lives with immersive virtual reality. Front. Robot. AI 3:74. doi: 10.3389/frobt.2016.00074

CrossRef Full Text | Google Scholar

Srinivasan, M., and Basdogen, C. (1997). Haptics in virtual environments: taxonomy, research status and challenges. Comput. Graph. 21, 393–404 doi: 10.1016/S0097-8493(97)00030-7

CrossRef Full Text | Google Scholar

Stone, B. (March 2019). Haptics for VR AR - Where Are We. Really?. Available online at: https://www.linkedin.com/pulse/haptics-vr-ar-where-we-really-bob-stone/ (accessed April 12, 2021).

Google Scholar

The Hold the World (2018). Hold the World. Available online at: https://www.youtube.com/watch?v=kFasefT-zrY&t=83s>; https://www.youtube.com/watch?v=er1aUDbyUXo

Google Scholar