Challenges of controlling the rotation of virtual objects with variable grip using force-feedback gloves

Some virtual reality (VR) applications require true-to-life object manipulation, such as for training or teleoperation. We investigate an interaction technique that replicates the variable grip strength applied to a held object when using force-feedback gloves in VR. We map the exerted finger pressure to the rotational freedom of the virtual object. With a firm grip, the object’s orientation is fixed to the hand. With a loose grip, the user can allow the object to rotate freely within the hand. A user study (N = 21) showed how challenging it was for participants to control the object’s rotation with our prototype employing the SenseGlove DK1. Despite high action fidelity, the grip variability led to poorer performance and increased task load compared to the default fixed rotation. We suspect low haptic fidelity as an explanation as only kinesthetic forces but no cutaneous cues are rendered. We discuss the system design limitations and how to overcome them in future haptic interfaces for physics-based multi-finger object manipulation.

1 Introduction

When we encounter objects in virtual reality (VR), we intuitively attempt to handle the virtual objects as if they were real. We must soon realize that our manual abilities in VR are comparatively limited. However, it is essential in some VR applications to reproduce reality with high fidelity. For example, the simulation of interactions must be as realistic as possible for training motor skills, such as in medicine, manufacturing, or engineering, but also for robot and surgery teleoperation, digital twins, digital marketing, or mixed reality. These use cases require a precise one-to-one mapping of the users’ actions to the virtual hand enabling true-to-life object manipulation. To simulate the astonishing dexterity of the human hands, the control over an object must go beyond having it glued to the virtual hand as a single-point effector. With current VR systems, when we hold a virtual object and move or turn our hand, the movement and rotation are often transferred directly to the object. The object’s possible orientations are thus limited by how far we can revolve our hand. Beyond this extrinsic movement of turning the entire hand, we would control a real object with intrinsic movements, i.e., use our fingers to adjust how we hold it within our hand (Elliott and Connolly, 1984).

We can impact a held object through the pressure we apply to it. With our grip, we determine how our skin and the object’s surface interact. For example, when holding an object with only gentle pressure, we can let it slip through our fingers. This has been virtually recreated with a haptic device that renders the tangential forces of the sliding object to the user’s fingertips (Kim et al., 2022). Previous work also explored the possibilities of applying pressure as means of input. Using squeezing actions on haptic devices has been investigated in abstract ways, e.g., when holding a smartphone (Yabe et al., 2017; Quinn et al., 2019), and with pinch gestures in mixed reality applications (Schmitz et al., 2022), but also for basic object manipulation (Achibet et al., 2014). Previous work also addressed the experience of object elasticity in VR by exerting pressure on the deformable object with controllers (Tsai et al., 2019), via a proxy (Ryu et al., 2020), or with force-feedback gloves (Coquillart et al., 2004).

In this work, we expand on the interaction technique proposed by Bonfert et al. (2019), which considers the grip applied to a held object using controller-based systems. By varying the grip strength, the object can be held loosely, allowing it to rotate freely between the fingers, or it can be grasped firmly, which transfers the hand’s rotation directly to the object. This grip variability affords control over additional rotational degrees of freedom. Without it, users must release and grasp the object again in the desired orientation, called clutching (Zhai et al., 1996). The affordances of grip variability influence our everyday tasks in various situations—often subconsciously—e.g., when moving a full glass, using a screwdriver, screwing in a light bulb, or fidgeting with a pen. Imagine moving a glass of water from a higher shelf to a table without gravity, ensuring the opening is always level to the top. The control over the grip strength allows a flexible grasp with automatic vertical alignment so that nothing is spilled. In other situations, we rely on an object’s inertia to change its rotation. For example, we would intuitively swing a book in an upright orientation by leveraging its momentum with a loose grip before putting it on a shelf. Virtually reproducing these natural grip-based hand manipulations could benefit use cases that require high interaction fidelity. Beyond realistic interactions, a variable grip can compensate for inadequate object orientations due to the initial grasp, e.g., to finely adjust the angle between the hand and a virtual hammer or knife.

An evaluation of the interaction technique showed advantages in terms of user satisfaction, intuitiveness, and realism with a slight decrease in the task load (Bonfert et al., 2019). This study used a controller-based system with HTC Vive controllers for input. Users set the grip strength with the trigger or grip buttons depending on the experiment condition. The abstract button actions must be mapped to the intended virtual actions of controlling an object, which was found to increase mental demand (Bonfert et al., 2019).

The interaction technique was implemented for the Valve Index controllers in a follow-up study by Pedersen et al. (2023). The user can grab virtual objects with these controllers by enclosing and pressing the controller’s handle. The pressure applied to the handle was interpreted as direct input for the grip strength applied to the object. This removed the need for mapping, which increases input fidelity. In line with previous findings (Bonfert et al., 2019), an evaluation showed that the interactions with variable grip were considered the most realistic, slightly slower, and with no difference in the TLX ratings. Only the higher usability could not be confirmed in this study (Pedersen et al., 2023).

1.1 Finger-based manipulation with force feedback

For applying pressure to something virtual, rendering a resistance force from the object’s surface is helpful. There is a rich history of research on haptic devices providing kinesthetic forces to the user’s hand and fingers (Massie and Salisbury, 1994; Hirota and Hirose, 1995; Bouzit et al., 2002; Minamizawa et al., 2007; Endo et al., 2011), typically for translational manipulations, shape exploration, or weight simulation. A recent survey on glove-shaped haptic devices that render force feedback details different designs and their characteristics (Wang et al., 2019).

VR systems with force-feedback gloves provide precise hand poses with input actions from the individual fingers. This direct form of grasping exactly resembles the hand movements of handling real objects. Additionally, the user is provided with haptic feedback that simulates the object’s resistance when pressing against its surface. The user can vary the finger pressure to adjust the grip strength as illustrated in Figure 1. Thus, the proposed interaction technique combines isomorphic control over object selection and movement with isometric control over rotational freedom. The higher input and feedback fidelity of using force-feedback gloves might enable a more natural control over a held object. Therefore, we developed a system that enables the user to control the rotation of virtual objects with variable grip through a dorsal-based SenseGlove DK1 with force feedback, as shown in Figure 1. In a quantitative user study (N = 21), we evaluated the users’ experiences with the system regarding placement accuracy, the time needed, the number of grasps, self-reports on task load, perceived object control, and aspects of presence. In a pick-and-place task, users moved 36 objects to a target area with a given position and orientation.

FIGURE 1

FIGURE 1. A user changes the orientation of a virtual soup can. Left: The SenseGlove DK1 is attached to the user’s fingers providing force feedback along the can’s shape. Right: (A) The user applies a firm grip. Thus the rotation of the can is fixed. (B) When the user loosens the grip by reducing the finger pressure, the can swings downwards due to gravity. (C) The can stays in a level orientation. Created with Unity Editor®. Unity is a trademark or registered trademark of Unity Technologies.

Although we anticipated higher perceived realism and user satisfaction with similar performance, we found disadvantages of providing grip variability. The participants performed poorly and reported a high task load even in the baseline condition. We discuss the experiment outcome and provide possible explanations why the interaction technique fell short of expectations. We derive opportunities for future research that builds on our experiences. Overall, we contribute a system design that advances the interaction technique of grip variability to glove-based interaction, as well as the results and lessons learned from an initial prototype evaluation for finger pressure-based object manipulations in VR.

2 System design

We built an interaction system for our user study using the SenseGlove DK1 VR gloves (SenseGlove, 2023). This exoskeleton is grounded on the back of the hand and connected with Velcro straps to the fingertips. It weighs 300 g per glove. The DK1 can restrict each finger’s inward movement to simulate the contact force of an object’s surface. It can only inhibit finger movement but not alter its position. Cables run from the glove’s base up to each finger segment’s tip and through the segment’s joints. Applying a brake force on the cable prevents the finger from moving further inwards (flexing). The hand position is captured with an HTC Vive Tracker. The glove precisely tracks the fingers’ pose and movements. In order to create the 3D representation of the glove, the angles of the four linkages of each segment are measured at a rate of 120 Hz and a resolution of 0.35°. Based on this, the SenseGlove software calculates the brakes’ resistances. For each finger, a maximum force of 40 N can be applied at the fingertip which can be updated at a rate of up to 200 Hz and a resolution of 100 steps of force (SenseGlove, 2019). The resistance increases when a collider on the fingertip approaches and interpenetrates an item. A maximum penetration depth and a force value for the maximum depth can be configured for the item. Based on that, a force value is calculated depending on how deep the collider is interpenetrating the item.

For the experiment, we used a Valve Index headset. The virtual scene was a hobby workshop with a workbench. The research question is on interactions with high simulation fidelity in terms of physical realism, so we aimed for photo-realistic objects. It was built in Unity 2020.3.23f1 using the SteamVR Unity Plugin v2.7.3 (SDK 1.14.15) and the SenseGlove Unity Plugin v2.3.1. We modified the plugin’s grasping interaction for the different experimental conditions. When the user reaches for an object, it is firmly attached to the hand as soon as enough fingers touch the surface and press against it. As the SenseGlove interaction system necessarily attaches the held object in fixed relation to the hand, we spawn a copy of the object at grasp. The copy is fully visible and seems to the user like the object that is actually being held and manipulated. Directly after grasping, it is aligned with the original object and will always follow its position. But when using a loose grip, the copy’s orientation deviates from the original orientation due to gravity. It can rotate in any direction and neither collides with the original object nor the hand, thus potentially penetrating the hand model. The potential hand penetrations reduce simulation fidelity, but this design decision allowed more rotational freedom increasing the interaction technique’s utility. Depending on how the object was grasped initially, the behavior is also better predictable when enabling interpenetration. As the object’s copy is visible to the user, the original instance of the object is invisible while it is grasped. The original shape still provides force feedback to the fingers. Hence, the resistance forces do not fluctuate while the object’s copy is rotating, as the held object always remains in its original orientation relative to the hand.

For controlling the object’s freedom of rotation, we interpreted the glove’s resistance force as pressure against the object’s surface. The softer the grip, the more freely the object can rotate within the hand. A loose object rotates with three degrees of freedom around an anchor point in the middle between the fingertips of the thumb and index finger, marked with an x in Figure 2. While an axis as the center of rotation would be more realistic in most cases, given the grasp with an opposing thumb, it would also restrict the versatility and predictability of the interaction technique. Therefore, the object’s rotation was not constrained in any direction when grasped softly. More pressure restricted the rotational freedom until fully locked to the hand’s rotation using a firm grip.

FIGURE 2

FIGURE 2. Forces and constraints during the interaction: forces in reality $\overrightarrow{f_{\mathrm{r}}}$ from the fingers pressing against the glove (cyan), virtual gravitational force $\overrightarrow{g_{\mathrm{v}}}$ (magenta), the swinging motion of the object due to gravity (green), virtual anchor point x from the configurable joint around which the object rotates (green), and the virtual counter force $\overrightarrow{deceleratio{n}_{\mathrm{v}}}$ from the slerp drive component slowing down the swinging motion. Created with Unity Editor®. Unity is a trademark or registered trademark of Unity Technologies.

We implemented this mechanism in Unity with a Configurable Joint attached to the held object and connected to the copy of the object, which feels rather rusty or slack depending on the Slerp Drive–Position Spring parameter. The lower the position spring value is, the less the copy object resists gravity; hence, the more it can deviate from the original object’s orientation. This is visualized in Figure 2. The object’s elasticity was not considered for the calculations. Consequently, flexible objects (such as the milk carton in the user study) did not soften or yield when grasped firmly.

After pilot testing, we found that the perceived control over the grip input was not as precise as hoped. When trying to grip it as gently as possible, the object was at risk of falling. Therefore, we simplified the interaction design to binary grip states: above 80% of finger pressure, the rotation was locked; below that, rotation is possible with slight resistance. This was visualized to the user with a bar above the hand that was filled with higher pressure. Beyond the threshold, it changed from a green to a red area indicating the firm grip, as shown in Figure 3. It was visible when holding an object in the condition with variable grip. Further, for varying the grip strength, we initially considered the pressure input of all fingers. After this turned out unreliable in informal testing, we only used the thumb and index finger pressure to control the grip. Nevertheless, all fingers still received force feedback.

FIGURE 3

FIGURE 3. The indicator above the user’s hand visualizes the grip strength applied to the object. The user grasps the object firmly when the bar is filled to the red area. Created with Unity Editor®. Unity is a trademark or registered trademark of Unity Technologies.

3 Study design

We evaluated our proposed interaction technique in a user study with 21 participants. The pick-and-place tasks in the experiment are inspired by the study by Bonfert et al. (2019), but the study design has been adjusted. The experiment had a within-subject design comparing the two conditions fixed grip and variable grip. In the condition with a fixed grip, the users experienced a standard interaction design of attaching an object firmly to the hand when grasping it. The object’s rotation is directly linked to the rotation of the hand. To change the angle between the object and the hand, the user must release the object and grab it again. In the condition with variable grip, the user experiences our new interaction technique that allows dynamic control over the object’s rotation. Adjusting the strength with which the fingers press against the virtual object either fixes the rotation as in the other condition or releases the object to rotate freely. With a loose grip, the object’s center of mass would rotate downwards following the gravitational pull, independent of the hand’s rotational movement. All participants tested both conditions in counterbalanced order. Although including two additional conditions with the controller-based implementations from Bonfert et al. (2019) would have been interesting for comparison, we refrained from prolonging the study further due to the already required time, object manipulations, and filled questionnaires.

3.1 Tasks

The participants were asked to pick up various objects and move them to a target in two tasks per condition. The objects needed to be placed in the correct location, with a deviation of less than 3 cm, and with the correct orientation, deviating less than 20°. In task A, six cans had to be moved to a target. The poses (position and orientation) of both the start and the target were identical for all six cans. This resulted in repeated measures with similar trajectories. After a can was correctly placed within the thresholds, it would disappear after 0.5 s, and the next one would appear. In task B, three types of objects were moved: cans, books, and milk cartons, as shown in Figure 4. Four instances of every object type had to be moved from different starting poses to the identical target pose, as illustrated in Figure 5. The wide range of starting poses required the users to vary trajectories and rotations between the objects. While the starting poses were identical for all participants, the order of the objects was randomized. However, each participant had the same order for both conditions. The order of the tasks was counterbalanced. In total, 756 object manipulations were measured ((6 Task-A objects+12 Task-B objects)*2 conditions * 21 participants). After cleaning the data from outliers (3*IQR) and system errors, 654 valid cases could be used for analysis. Due to the repeated-measures design, 287 pairs yield valid data in both conditions.

FIGURE 4

FIGURE 4. The three object types used in task B of the experiment: a can (diameter of 11 cm, 10 cm high), a book (20 cm × 14 cm x 4 cm), and a milk carton (base of 7 cm × 8 cm, 16 cm high). Created with Unity Editor®. Unity is a trademark or registered trademark of Unity Technologies.

FIGURE 5

FIGURE 5. The steps of moving an item to its target in task B. There is one item to be moved at a time. Each of the twelve items will appear in an individual orientation and point, although these poses are predefined and equal between the different trials. A reference that does not physically interfere is placed behind the target area so participants can derive the desired orientation. Created with Unity Editor®. Unity is a trademark or registered trademark of Unity Technologies.

3.2 Procedure

After giving informed consent and filling in a demographic questionnaire, the participants were introduced to the VR equipment, including the headset and force-feedback glove. Only one glove was used to prevent bi-manual interactions. For increased comparability, all participants used the right glove and were right-handed. They entered a virtual workshop environment with tables and shelves where the tutorial and experiment took place. Tooltips with instructions and explanations guided them through the process. Before each condition, the participants could test the interaction mode in a tutorial until they felt confident and demonstrated proficiency in both interaction modes with all three items. Then, they performed both tasks and filled in the questionnaires, followed by the same process in the other condition. The questionnaires included the Presence Questionnaire by Witmer and Singer (1998), the raw NASA TLX by Hart and Staveland (1988), and five custom questions. The following custom items were rated on a scale of 1–7, with the labels in brackets.

1. Compared to the real world, how PRECISELY could you place objects in the demanded location and orientation? (much less precisely to much more precisely)

2. Compared to the real world, how FAST could you place objects in the demanded location and orientation? (much slower to much faster)

3. I was aware of how tight I was gripping the items. (strongly disagree to strongly agree)

4. I could develop a sense of how tight I was gripping the items. (strongly disagree to strongly agree)

5. I could move and rotate the items as I expected. (strongly disagree to strongly agree)

They were integrated into the virtual environment and operated with an HTC Vive controller given to the participants in their free hand. Following the recommendations by Alexandrovsky et al. (2020), the in-VR questionnaires avoided interrupting the VR experience and might improve data quality. After the measurements, the participants could return to the tutorial and keep testing the technology. In the end, a semi-structured interview over approximately 5 minutes was conducted. The experiment, questionnaires, and interviews were held in English. If preferred by the participant, the interview was held in German. In total, the experiment took approximately 45 min on average.

3.3 Sample

The sample was recruited on the university campus with email, leaflets, and word-of-mouth advertisements. There was no financial compensation for participating. 21 people participated in the experiment, of which six self-identified as female and 15 as male. Their age ranged from 15 to 61, averaging 27.3 years. All participants were right-handed and used the right glove for the experiment. Nine participants had never used VR before the experiment, and only two used VR at least every month. Four people reported feeling moderately or very experienced with VR, seven little, and ten not at all. Ten participants had used VR controllers before, of which eight used them for object manipulation. Only four have already used some form of hand tracking, such as gloves or optical finger tracking, of which three used it for handling objects. Thus, it was the first time for 18 participants to use a glove to move around objects.

3.4 Data analysis

All statistical tests are calculated with an alpha level of .05, Bonferroni-Holm-corrected, and two-sided assuming any difference between the conditions. The distribution of the metric data from the performance measurements deviates from normality according to the Shapiro-Wilk-Test (p = [0.001 .. 0.061]). Therefore, we used the non-parametric Wilcoxon signed-rank test to find group differences. It was also used for the ordinal data collected in the questionnaires. The effect sizes are reported as matched pairs rank biserial (r_rb) and can be interpreted as a correlation coefficient. We checked the internal consistency with reliability analysis because the custom questionnaire items are not validated as standardized scales. The tests yielded Cronbach’s Alpha values of α = .71, indicating that the single items describe the same underlying concept. The cleaned quantitative data with test reports are available in a repository on OSF (see section 6). The qualitative data from the semi-structured interviews were analyzed unsystematically to identify relevant insights that might explain the quantitative findings.

4 Results

Overall, the participants performed better in the fixed grip condition. The distributions of the performance data are shown in Figure 6. The translational accuracy of placing the objects on the targets was higher with a fixed grip (Median_f = 8 mm offset ± SD_f = 6 mm) than with a variable grip (Mdn_v = 9 ± 7 mm). Hence, the distance from an object to the target’s center was 1 mm smaller, which is statistically significant with a small effect size (Z = − 2.4, p = .015, r_rb = − .17). The rotational accuracy was also higher with a fixed grip (Mdn_f = 4.2° ± 5° deviation, Mdn_v = 6.1° ± 5.6°). The difference of 1.9° in orientation accuracy is significant with a small effect size (Z = − 3.2, p < .005, r_rb = − .22). Further, the participants needed less time to successfully place an object when using a fixed grip (Mdn_f = 3.8 ± 2.3 s, Mdn_v = 5.2 ± 3.6 s). This 1.4 s difference is significant with a large effect size (Z = − 3.4, p < .005, r_rb = − .57). Similarly, the participants needed fewer attempts with a fixed grip (Mdn_f = 1 ± 1.1 grasps per object, Mdn_v = 2 ± 1.5 grasps), which is a significant difference with a medium effect size (Z = − 4.6, p < .005, r_rb = − .43).

FIGURE 6

FIGURE 6. Violin plots for the performance data on participants’ average grasping duration, rotational offset, translational offset, and grasping attempts per condition.

The interaction technique with a fixed grip was generally rated better in the questionnaires. The overall score of the Presence Questionnaire (Witmer and Singer, 1998) and also its subscores showed no group differences (p > .805), except for the subscale interface quality. Here, the participants rated how much the interface interfered with the task with a significant advantage of using a fixed grip (Mdn_f = 12 ± 2.4, Mdn_v = 10 ± 2) showing a large effect size (Z = 2.9, p = .018, r_rb = .77). While the raw score of the NASA TLX (Hart and Staveland, 1988) indicates a lower workload of using the fixed grip (Mdn_f = 41.7 ± 9, Mdn_v = 43.3 ± 6.4), this difference is not significant after Bonferroni-Holm correction (p = .092). However, there are significant differences with large effect sizes for the items mental demand (Z = − 3.1, p = .014, r_rb = − .89), performance (Z = − 3.0, p = .018, r_rb = − .76), and frustration (Z = − 2.9, p = .02, r_rb = − .78), each with higher demands for variable grip. The distributions of the most insightful TLX items are shown in Figure 7. Only one of the custom questionnaire items was rated significantly differently. Participants gave a higher rating on whether they “could move and rotate the items” as they expected (Mdn_f = 6 ± 1.2, Mdn_v = 5 ± 1.5) with a large effect size (Z = 2.9, p = .018, r_rb = .79).

FIGURE 7

FIGURE 7. Selected violin plots for the NASA Task Load Index raw score and the items mental demand and effort per condition.

5 Discussion

The study results showed a poorer performance with the grip variability in a virtual pick-and-place task using a force-feedback glove. The object placement was slightly less accurate, but the study participants needed 40% longer and considerably more grasping attempts to move an object. This is in line with previous research on controller-based systems in which varying the grip of a held object was found to take more time and sometimes additional grabs (Bonfert et al., 2019; Pedersen et al., 2023). In contrast to the outcome of this experiment, grip variability with controllers was reported to be more intuitive, easier to control, and more satisfactory (Bonfert et al., 2019), as well as more realistic (Bonfert et al., 2019; Pedersen et al., 2023), compared to the condition with a fixed grip. However, when using a force-feedback glove with a variable grip, participants reported higher mental load and frustration, inferior perceived performance, and less agency over the object behavior.

There are several possible explanations for these unexpected results. One reason for the poor performance and ratings of the proposed interaction technique might be the general difficulties in handling objects with the SenseGlove DK1 as an early development prototype. Considering the simple task, we measured relatively high TLX scores and slow placement even in the baseline condition with a simple pick-and-place task. According to interview feedback, many users did not feel confident using the glove.

Its force feedback mechanism posed a severe limitation in the experiment as the rendered forces depend on the angle at which a finger presses against the surface. A perfectly perpendicular angle results in appropriate resistance, but the more oblique the finger touches the surface, the higher the chance of no or unexpected feedback. While this is already confusing with the fixed-grip baseline, it is even more confusing with a variable grip. A change in pressure against the surface controls the change between a firm and a loose grip. If the resistance from the surface is unexpected and inadequate, it is impossible to utilize it for intuitive control over the grip strength. As another mechanical restriction of the SenseGlove DK1, the applied pressure can only be approximated from the glove’s generated resistance. Readings from pressure sensors at the fingertips might provide more accurate data and allow finer control over the grip strength.

Another reason might be our sample with many novice users. 18 out of 21 participants have never used hand tracking for object manipulation before. Already challenged by the glove’s behavior, they mostly restricted their actions to the simpler fixed grip, which resulted in a more predictable outcome, even if it required more manual movement and inconvenient hand poses. Similar to an interaction technique with continuous variable grip tested by Bonfert et al. (2019), in which the grip strength was set with a controller’s trigger button, the grip adjustment with the glove was too delicate. Due to the steep learning curve, participants assumed they could have performed better with considerably more practice and might then find it helpful.

From our observations of the experiments and statements in the interviews, we suspect one more reason for the slower object handling times of the variable grip. The unsteady grasp of the variable grip mode caused more items to be accidentally dropped by the participants, which naturally resulted in longer handling times on average because the items had to be grabbed again. Nonetheless, once the users established a secure grasp, moving the item along the required trajectory and placing it down was not observed to be slower with the variable grip.

An additional limitation of our implementation was the visual indicator of the currently applied grip strength. Participants described it as rather distracting than helpful. Alternative cues might be considered in future work, but ideally, the only visual indicator necessary is the observed object behavior. Ideally, users should be able to feel the object’s state only from haptic feedback in a system reproducing grip-based manipulation of real objects with high interaction fidelity.

An improved physics simulation with the reliable rendering of kinesthetic forces might enable more confident control over holding the object. However, even when rendered perfectly, it might be insufficient for users to intuitively control the applied grip strength. Surface-finger interaction involves other physical forces imperative for high haptic fidelity (Muender et al., 2022). When handling real objects, humans are excellent at maintaining a balance between a grip strong enough to prevent slipping yet not excessively powerful (Westling and Johansson, 1984). However, to achieve this, we interpret the frictional sensation at the fingertips (Cadoret and Smith, 1996), which our prototype does not render. When gripping an object, the frictional condition is informed by tangential, not kinesthetic forces (Augurelle et al., 2003) such as skin stretches. If cutaneous cues are missing, as studies have shown using local anesthesia, people also drop real objects or use an overly powerful grip because their mental model of the held object’s physical properties is insufficiently informed (Westling and Johansson, 1984; Augurelle et al., 2003).

Overall, the presented study has not shown that mapping the finger pressure with force-feedback gloves to the grip strength of handling virtual objects would, in principle, be an undesirable solution. We have not yet demonstrated the potential we anticipate when using hardware with more accurate actuation, sophisticated force vector estimation of individual fingers, and additional cutaneous feedback.

5.1 Future work

Therefore, supporting tactile cues beyond kinesthetic forces must be considered for multi-finger object manipulation when controlling grip strength. Shear forces, friction, slip, or contact forces have been shown to improve reproducing real-world haptic experiences (Girard et al., 2016; Whitmire et al., 2018; Salazar et al., 2020; Kim et al., 2022). Also, the impact of their combination with kinesthetic feedback from gloves would be insightful. The ideal prototype for exploring the benefits of natural grip-based object manipulation would render detailed tactile cues, rely on complex physics calculations comprising the individual pressure of all fingers, afford dynamic and continuous grip, and avoid interpenetration of the object and the hand.

While in this study, we explored using the interaction technique for objects with physically realistic behavior, it could also be applied with no gravitational forces in use cases that demand flexible control over the rotational degrees of freedom without clutching. By deactivating gravity for the held object, its orientation can be finely adjusted in any direction without the vertical pull of gravity. This could be useful in educational training, e.g., for surgery or assembly tasks. In addition to the performance indicators, user satisfaction and comfort of hand poses could be operationalized more informatively.

Aside from interactions with force-feedback gloves, improvements in optical hand tracking and rich mixed-reality applications offer intriguing opportunities for similar interaction techniques. For this, there is a need for complex physics-based simulations that infer object behavior from the position and properties of the individual fingers (Höll et al., 2018). As the contact points between fingers and surface change according to how the object moves within the hand, the adaption of the grasping pose must be determined dynamically, as previous work explored for freehand grasping (Dalia Blaga et al., 2021).

6 Conclusion

We present a system that allows the user to control the rotation of a held virtual object by adjusting the pressure of the fingers on the object’s surface, hence, varying the strength of the grip. The prototype was realized with the dorsal-based SenseGlove DK1 providing force feedback to the individual fingers. Thus, in contrast to previous work, the system directly maps how strongly the user presses against the glove’s resistance to how firmly the object is being held. Although we expected a more intuitive and efficient execution of a pick-and-place task, the evaluation shows that users need more time and more attempts, experience a higher task load, and perceive less agency over the object. Technical limitations and theoretical considerations provide explanations and outline possible next steps in research on the dexterous multi-finger manipulation of virtual objects.

Data availability statement

The quantitative dataset with the statistical analysis can be found in the following OSF repository: https://osf.io/d64va.

Ethics statement

Ethical approval was not required for the studies involving humans because the local ethics board only issues approval if funding agencies demand it. The studies were conducted in accordance with the local legislation and institutional requirements. The participants provided their written informed consent to participate in this study.

Author contributions

MB conceived the research idea, wrote the first draft of the manuscript, contributed to the figures, provided feedback on the prototype, and supervised the project. MH implemented the system, conducted the user study, and contributed to the figures and manuscript writing. MB and MH designed the interaction technique, developed the study design, conducted the literature review, and performed the data analysis. RM contributed to the discussion of the research idea and the manuscript review. All authors contributed to the article and approved the submitted version.

Funding

This research was partially funded by Klaus Tschira Stiftung by financing the scholarship of MB, and the German Federal Ministry of Education and Research (BMBF) as part of the project “Hydrogen for Bremen’s Industrial Transformation.” Staats- und Universitätsbibliothek Bremen covered the open-access publication fees.

Acknowledgments

We are grateful to Steeven Villa and the Human-Centered Ubiquitous Media group at the Ludwig-Maximilians-Universität München for providing us with the gloves for this research. We thank Tanja Döring and Thomas Münder for their feedback on the prototype and study design.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

Achibet, M., Marchal, M., Argelaguet, F., and Lécuyer, A. (2014). “The virtual mitten: A novel interaction paradigm for visuo-haptic manipulation of objects using grip force,” in 2014 IEEE Symposium on 3D User Interfaces (3DUI), Minneapolis, MN, USA, 29-30 March 2014 (IEEE), 59–66. doi:10.1109/3DUI.2014.6798843

CrossRef Full Text | Google Scholar

Alexandrovsky, D., Putze, S., Bonfert, M., Höffner, S., Michelmann, P., Wenig, D., et al. (2020). “Examining design choices of questionnaires in vr user studies,” in Proceedings of the 2020 CHI conference on human factors in computing systems (New York, NY, USA: Association for Computing Machinery), 1–21. doi:10.1145/3313831.3376260

CrossRef Full Text | Google Scholar

Augurelle, A. S., Smith, A. M., Lejeune, T., and Thonnard, J. L. (2003). Importance of cutaneous feedback in maintaining a secure grip during manipulation of hand-held objects. hand-held objects 89, 665–671. doi:10.1152/jn.00249.2002

PubMed Abstract | CrossRef Full Text | Google Scholar

Bonfert, M., Porzel, R., and Malaka, R. (2019). “Get a grip! Introducing variable grip for controller-based VR systems,” in 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), Osaka, Japan, 23-27 March 2019 (IEEE), 604–612. doi:10.1109/VR.2019.8797824

CrossRef Full Text | Google Scholar

Bouzit, M., Popescu, G., Burdea, G., and Boian, R. (2002). “The rutgers master II-nd force feedback glove,” in Haptic interfaces for virtual environment and teleoperator systems (Los Alamitos, CA: IEEE Computer Society Press), 145–152. doi:10.1109/HAPTIC.2002.998952

CrossRef Full Text | Google Scholar

Cadoret, G., and Smith, A. M. (1996). Friction, not texture, dictates grip forces used during object manipulation. dictates grip forces used Dur. object Manip. 75, 1963–1969. doi:10.1152/jn.1996.75.5.1963

PubMed Abstract | CrossRef Full Text | Google Scholar

Coquillart, S., Göbel, M., Maciel, A., Sarni, S., Buchwalder, O., Boulic, R., et al. (2004). “Multi-finger haptic rendering of deformable objects,” in Proceedings of the tenth eurographics conference on virtual environments (New York, NY, USA: Association for Computing Machinery). doi:10.2312/EGVE/EGVE04/105-112

CrossRef Full Text | Google Scholar

Dalia Blaga, A., Frutos-Pascual, M., Creed, C., and Williams, I. (2021). “Freehand grasping: an analysis of grasping for docking tasks in virtual reality,” in 2021 IEEE Virtual Reality and 3D User Interfaces (VR), Lisboa, Portugal, 27 March 2021 - 01 April 2021 (IEEE), 749–758. doi:10.1109/VR50410.2021.00102

CrossRef Full Text | Google Scholar

Elliott, J. M., and Connolly, K. J. (1984). A classification of manipulative hand movements. Dev. Med. Child Neurology 26, 283–296. doi:10.1111/j.1469-8749.1984.tb04445.x

PubMed Abstract | CrossRef Full Text | Google Scholar

Endo, T., Kawasaki, H., Mouri, T., Ishigure, Y., Shimomura, H., Matsumura, M., et al. (2011). Five-fingered haptic interface robot: hiro III. IEEE Trans. haptics 4, 14–27. doi:10.1109/TOH.2010.62

PubMed Abstract | CrossRef Full Text | Google Scholar

Girard, A., Marchal, M., Gosselin, F., Chabrier, A., Louveau, F., and Lécuyer, A. (2016). HapTip: displaying haptic shear forces at the fingertips for multi-finger interaction in virtual environments. Front. ICT 3, 6. doi:10.3389/fict.2016.00006

CrossRef Full Text | Google Scholar

Hart, S. G., and Staveland, L. E. (1988). Development of NASA-TLX (task load index): results of empirical and theoretical research. Adv. Psychol. 52, 139–183. doi:10.1016/S0166-4115(08)62386-9

CrossRef Full Text | Google Scholar

Hirota, K., and Hirose, M. (1995). Providing force feedback in virtual environments. IEEE Comput. Graph. Appl. 15, 22–30. doi:10.1109/38.403824

CrossRef Full Text | Google Scholar

Höll, M., Oberweger, M., Arth, C., and Lepetit, V. (2018). “Efficient physics-based implementation for realistic hand-object interaction in virtual reality,” in 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), Tuebingen/Reutlingen, Germany, 18-22 March 2018 (IEEE), 175–182. doi:10.1109/VR.2018.8448284

CrossRef Full Text | Google Scholar

Kim, M. J., Ryu, N., Chang, W., Pahud, M., Sinclair, M., and Bianchi, A. (2022). “SpinOcchio: understanding haptic-visual congruency of skin-slip in VR with a dynamic grip controller,” in CHI conference on human factors in computing systems (New York, NY, USA: Association for Computing Machinery). doi:10.1145/3491102.3517724

CrossRef Full Text | Google Scholar

Massie, T. H., and Salisbury, J. K. (1994). The phantom haptic interface: A device for probing virtual objects. Haptic Interfaces Virtual Environ. Teleoperator Syst. 55 (1), 295–300.

Google Scholar

Minamizawa, K., Fukamachi, S., Kajimoto, H., Kawakami, N., and Tachi, S. (2007). “Gravity grabber: wearable haptic display to present virtual mass sensation,” in ACM SIGGRAPH 2007 emerging technologies (New York, NY, USA: Association for Computing Machinery), 8. doi:10.1145/1278280.1278289

CrossRef Full Text | Google Scholar

Muender, T., Bonfert, M., Reinschluessel, A., Malaka, R., and Döring, T. (2022). “Haptic fidelity framework: defining the factors of realistic haptic feedback for virtual reality,” in Proceedings of the 2022 CHI conference on human factors in computing systems (CHI ’22) (New Orleans, LA, USA: Association for Computing Machinery). doi:10.1145/3491102.3501953

CrossRef Full Text | Google Scholar

Pedersen, A. K., Isaksen, H., Riedel, M. B., Jørgensen, M., Paisa, R., and Nilsson, N. C. (2023). “Get a variable grip: A comparison of three gripping techniques for controller-based virtual reality,” in 2nd IEEE VR 2023 Workshop on 3D Reconstruction, Digital Twinning, and Simulation for Virtual Experiences, Shanghai, China, 25-29 March 2023 (IEEE).

CrossRef Full Text | Google Scholar

Quinn, P., Lee, S. C., Barnhart, M., and Zhai, S. (2019). “Active edge: designing squeeze gestures for the google pixel 2,” in Proceedings of the 2019 CHI conference on human factors in computing systems (New York, NY, USA: Association for Computing Machinery). doi:10.1145/3290605.3300504

CrossRef Full Text | Google Scholar

Ryu, N., Lee, W., Kim, M. J., and Bianchi, A. (2020). “ElaStick: A handheld variable stiffness display for rendering dynamic haptic response of flexible object,” in Proceedings of the 33rd annual ACM symposium on user interface software and technology (New York, NY, USA: Association for Computing Machinery), 1035–1045.

Google Scholar

Salazar, S. V., Pacchierotti, C., Tinguy, X. d., Maciel, A., and Marchal, M. (2020). Altering the stiffness, friction, and shape perception of tangible objects in virtual reality using wearable haptics. IEEE Trans. Haptics 13, 167–174. doi:10.1109/TOH.2020.2967389

PubMed Abstract | CrossRef Full Text | Google Scholar

Schmitz, M., Günther, S., Schön, D., and Müller, F. (2022). “Squeezy-feely: investigating lateral thumb-index pinching as an input modality,” in CHI conference on human factors in computing systems (New York, NY, USA: Association for Computing Machinery). doi:10.1145/3491102.3501981

CrossRef Full Text | Google Scholar

SenseGlove (2019). Getting started with the SenseGlove DK1. Avaliable At: https://www.senseglove.com/wp-content/uploads/2020/08/2020Manual1_3v1_non-booklet.pdf.

Google Scholar

SenseGlove (2023). VR and haptic technology gloves. Avaliable At: https://www.senseglove.com/.

Google Scholar

Tsai, H. R., Rekimoto, J., and Chen, B. Y. (2019). “ElasticVR: providing multilevel continuously-changing resistive force and instant impact using elasticity for VR,” in Proceedings of the 2019 CHI conference on human factors in computing systems (New York, NY, USA: Association for Computing Machinery). doi:10.1145/3290605.3300450

CrossRef Full Text | Google Scholar

Wang, D., Song, M., Naqash, A., Zheng, Y., Xu, W., and Zhang, Y. (2019). Toward whole-hand kinesthetic feedback: A survey of force feedback gloves. IEEE Trans. Haptics 12, 189–204. doi:10.1109/TOH.2018.2879812

PubMed Abstract | CrossRef Full Text | Google Scholar

Westling, G., and Johansson, R. S. (1984). Factors influencing the force control during precision grip. Exp. Brain Res. 53, 277–284. doi:10.1007/BF00238156

PubMed Abstract | CrossRef Full Text | Google Scholar

Whitmire, E., Benko, H., Holz, C., Ofek, E., and Sinclair, M. (2018). “Haptic revolver: touch, shear, texture, and shape rendering on a reconfigurable virtual reality controller,” in Proceedings of the 2018 CHI conference on human factors in computing systems (New York, NY, USA: Association for Computing Machinery), 1–12. doi:10.1145/3173574.3173660

CrossRef Full Text | Google Scholar

Witmer, B. G., and Singer, M. J. (1998). Measuring presence in virtual environments: A presence questionnaire. Presence teleoper. Virtual Environ. 7, 225–240. doi:10.1162/105474698565686

CrossRef Full Text | Google Scholar

Yabe, S. I., Kishino, H., Kimura, T., and Nojima, T. (2017). “Pseudo-haptic feedback on softness induced by squeezing action,” in 2017 IEEE World Haptics Conference (WHC), Munich, Germany, 06-09 June 2017 (IEEE), 557–562. doi:10.1109/WHC.2017.7989962

CrossRef Full Text | Google Scholar

Zhai, S., Milgram, P., and Buxton, W. (1996). “The influence of muscle groups on performance of multiple degree-of-freedom input,” in Proceedings of the SIGCHI conference on human factors in computing systems (New York, NY, USA: Association for Computing Machinery), 308–315. doi:10.1145/238386.238534

CrossRef Full Text | Google Scholar