Introducing wearable haptics for rendering velocity feedback in VR serious games for neuro-rehabilitation of children

Rehabilitation in virtual reality offers advantages in terms of flexibility and parametrization of exercises, repeatability, and continuous data recording and analysis of the progress of the patient, also promoting high engagement and cognitive challenges. Still, most of the proposed virtual settings provide a high quality, immersive visual and audio feedback, without involving the sense of touch. In this paper, we show the design, implementation, and first evaluation of a gaming scenario for upper limb rehabilitation of children with cerebral palsy. In particular, we took care to introduce haptic feedback as a useful source of sensory information for the proposed task, considering—at the same time—the strict constraints for haptic wearable devices to comply with patient’s comfort, residual motor abilities, and with the embedded tracking features of the latest VR technologies. To show the potential of haptics in a rehabilitation setup, the proposed device and rendering method have been used to improve the velocity control of upper limb movements during the VR exercise, given its importance as a motor recovery metric. Eight healthy participants were enrolled, and results showed that haptic feedback can lead to lower speed tracking errors and higher movement smoothness, making the proposed setup suitable to be used in a rehabilitation context as a way to promote movement fluidity during exercises.

1 Introduction

Rehabilitation in virtual reality offers several potential advantages when compared to conventional treatments. In neuro-rehabilitation, motor exercises focusing on targeted motor functions can be implemented in virtual settings and proposed to the patient in the shape of serious games (Deutsch and McCoy, 2017). Flexibility of the virtual setting and of the parameters of the exercise (i.e., reachable workspace, required range of the joint angles, and intensity of the task) is useful in adapting to the treatment to different patients’ needs. Also, repeatability of the exercise and continuous data recording of quantitative metrics through the virtual setup are key features to obtain a more robust and fine analysis of the outcomes of the therapy: repetition of goal-oriented tasks has shown to be important in neurological rehabilitation (Langhorne et al., 2011; Veerbeek et al., 2014). A recent comprehensive review of adaptive VR-based training in rehabilitation is found in a work by Zahabi and Abdul Razak (2020).

In a previous work by Bortone et al. (2020), our group summarized all the studies where VR-based interventions were promoted for upper limb therapy and had been conducted in children with neuromotor impairments (cerebral palsy, CP, or developmental dyspraxia, DD). It emerged that most of the study relied on intervention with the low-cost, off-the-shelf VR system [using Wii Sports Resort in a work by Chiu and Kuo (2015), Xbox 360 with Microsoft Kinect in a work by Zoccolillo et al. (2015), and PlayStation with Move and/or EyeToy in another work by Rand et al. (2008)] without considering the specific clinical outcome of the patients. In general, very few studies reported significant results with respect to conventional therapy and/or control. In the same aforementioned study, our group adopted a two-period crossover design to determine the differences between the VR-based therapy and conventional treatment. They showed that the VR-assisted therapy with a VERA system was overall feasible for introduction in a prolonged rehabilitation trial. Choi et al. (2021) assessed 80 children with brain injury including cerebral palsy aged 3–16 years as part of a multicenter, single-blind, randomized controlled trial. Both groups significantly improved after treatment compared to baseline; however, the virtual reality group showed more significant improvements in upper limb dexterity functions, the performance of activities of daily living, and forearm supination by kinematic analysis.

Of particular importance is the engagement of the patient in performing the proposed exercises. Neuro-rehabilitation is a learning process, and the active mental participation of the patient is a required element to promote brain plasticity and, thus, boost the motor recovery (King et al., 2020). Virtual reality (VR) has shown to be an effective means to achieve these goals, offering to the clinical personnel the ability to control and modulate the proposed exercises to challenge the user into an immersive and focused environment (Laver et al., 2017). To this aspect, the immersive virtual setting and the gamification of the exercises acquire a huge potential, especially in children neuro-rehabilitation, where motivation and maintained attention to accomplish the exercise are not obvious.

1.1 Immersive Virtual reality in neuro-rehabilitation of children

The literature regarding clinical studies using a virtual setting for children neuro-rehabilitation shows only few examples of immersive VR systems used in prolonged clinical trials. As examples, studies presented by Kuriakose and Lahiri (2015), Wade et al. (2016), Simões et al. (2018), and Choi et al. (2021) used a desktop monitor setting for presentation of the virtual environments. Other studies, such as those presented by Bortone et al. (2020), Bossenbroek et al. (2020), Booth et al. (2019), and Kerns et al. (2017) used a head-mounted display (HMD) or treadmill plus cave settings to display an immersive, VR environment.

A challenge in implementing VR settings for clinical studies can be addressed to the complexity and usability of technological systems that have to be assembled at the level of research prototypes, usually on the basis of separated off-the-shelf products.

Novel technologies, however, carry opportunities for increasing both usability and effectiveness of the aforementioned systems. Immersive virtual reality systems, in particular head-mounted displays (HMDs), have seen in the past decade an impressive progress in terms of quality of the visual feedback, involving precise head tracking, a wider field of view, and a higher resolution of rendering. Furthermore, in the very recent years, some off-the-shelf HMDs started to offer advanced features that considerably impact usability: they now include wireless stand-alone operation, simple donning of the system and fast calibration of the workspace, and embedded hand tracking. Such recent technology advances might significantly change the use of VR in clinical applications, improving the feasibility of immersive VR setups operated by non-technical personnel in prolonged clinical trials.

1.2 Haptic feedback in virtual rehabilitation

Regarding immersive VR, an aspect still largely unmet by proposed studies in rehabilitation is the introduction of haptic feedback. The use of exoskeletons in rehabilitation, also associated with virtual exercises, has been a largely investigated field of research (Patton et al., 2006; Qiu et al., 2009; Martin-Niedecken and Götz, 2017). A recent review on haptic training and strategies in rehabilitation is presented by Basalp et al. (2021). Here, the haptic interaction is at the level of robotic motor assistance, rather than haptic rendering, usually with grounded or desktop robotic devices. In cases where motor assistance is not strictly needed, lighter wearable haptic devices can provide informative haptic feedback to the user without limiting workspace and mobility of the patient and can therefore better preserve their residual dexterity in more immersive and interactive VR setups. The addition of haptic feedback is still expected to play a positive role in the treatment, due to the more congruent and informative multi-sensory feedback specifically related to the motor task (Bortone et al., 2018) and to the increased engagement within the virtual scenario (Martin-Niedecken and Götz, 2017). Coherence of multi-sensory feedback including the sense of touch is important in general for immersion and embodiment in VR (Kilteni et al., 2012); moreover, tactile feedback is also directly associated with the targeted motor tasks proposed in typical neuro-rehabilitation exercises. A review published by Demain et al. (2013) highlights that, although the presence of haptic feedback is expected to have the potential to improve sensory motor integration in neurological conditions, most of the proposed devices rely on kinesthetic haptic feedback (i.e., robotic assistance), and only a few, on vibrotactile feedback.

Regarding light and wearable haptic devices for tactile feedback, the literature is particularly rich. Different devices have been developed in the past decades, ranging from thimbles to exoskeleton architecture, proposing different solutions each targeted at specific cutaneous feedback and cues, such as contact orientation (Chinello et al., 2015), area of contact (Fani et al., 2017), stretch and indentation (Leonardis et al., 2016), thermal (Gallo et al., 2015), and multi-modal feedback (Gabardi et al., 2018). In a work by Pacchierotti et al. (2017), the rich variety of wearable haptic devices for tactile feedback proposed in the literature is reviewed. The design of wearable devices is the combined result of different constraints and features in terms of encumbrance, lightweight, richness, and quality of the provided haptic rendering.

Application of these devices to the rehabilitation setting poses additional requirements in terms of wearability: the already strict dimensional constraints for fingertip haptic devices become more critical if used by patients with limited motor capabilities, in particular by children with reduced hand dimensions. To comply with these requirements is also in line with the trend of the latest VR technologies described previously: high wearability of the setup enhances comfort, dexterity, and overall usability of the system.

On the basis of previous studies we presented (Bortone et al., 2018, 2020), this work presents novel design solutions and final implementation of a VR neuro-rehabilitation scenario for children with cerebral palsy (CP), aiming at improving immersion, engagement and usability of the system in the clinical setting by means of the effective use of the latest VR technologies. The scenario is developed within the framework of the research project TELOS (telos-project.eu).

Importantly, the design of the virtual serious game includes informative haptic rendering: we present here a method to provide informative tactile feedback, aimed at improving velocity-related kinesiological metrics used in rehabilitation. Selection of the velocity-related metrics (above others involved in the proposed path-tracking task) is a trade-off between capabilities of lightweight wearable devices and expected effectiveness of the haptic information for the treatment. Precision and smoothness of the trajectory (or the number of velocity peaks) are typical metrics used to evaluate the quality of the movement in similar rehabilitation exercises. Smooth and well-coordinated movements are features of a healthy human motor behavior; hence, smoothness and constant traveling velocity are metrics conventionally adopted in a number of rehabilitation studies [e.g., (Bosecker et al., 2010; Gutierrez et al., 2020)] and also specifically for children with CP (Krebs et al., 2012; Bortone et al., 2020). To the best of our knowledge, velocity-related metrics were not provided before as a real-time feedback for the user, whereas position-error feedback is usually the conventional visual information used in rehabilitation virtual scenarios Stroppa et al. (2017). Also, the haptic feedback is rendered by custom developed, lightweight haptic thimbles, compatible with the markerless hand-tracking system embedded in the HMD.

Summarizing the contribution, in this study, we analyzed the influence of fingertip haptic feedback during path tracking tasks, in terms of smoothness and speed tracking error. With reference to our previous work, the system has been reshaped to comply with the latest hand-tracking technologies in VR platforms, aiming at making the overall system more compact, lightweight, and usable. In Section 2, we present the structure of the virtual serious game; in Section 3, we show the design of the haptic feedback in terms of rendering and device integration; in Section 4, we evaluate the role of the designed haptic feedback in a group of eight healthy participants. Discussion and final conclusions are given in Section 5.

2 Materials and methods

In this section, we describe the different modules developed to compose the proposed rehabilitation system: the virtual scenario implementing the rehabilitation exercise in the form of a serious game, the haptic rendering strategy, and the haptic devices we designed for immersive VR applications.

2.1 Virtual serious game design and structure

2.1.1 The virtual scenario

We designed the virtual scenario by immersing the child in the role of a wizard apprentice. The child is required to learn how to trace magic symbols in order to cast the most powerful spells. The proposed VR scenario, thus, implements an upper limb rehabilitation exercise based on path tracking: path tracking is a typical rehabilitation paradigm recruiting visual and proprioceptive coordination of the upper limb. The VR setting depicts a magical training area for wizards with mountains and trees all around and a big magical rune on the floor, surrounding the area where spells are allowed. The game requires the subject to draw symbols in the air with their index finger as a wand, to generate spells, and launch them against the enemy. The more precise the symbol drawing is, comparing it to the 2D sample shown in front of the subject, the more powerful the spell is. This feedback is not meant to influence either the game flow or the effectiveness of feedback that concerns the aim of this study. The parametrization of the exercise can involve complexity of the proposed trajectory, as well as dimension and location with respect to the workspace upper limb. Also, the addition of cognitive elements is envisaged with the aim of pushing a high engagement in the game over longer periods of treatment. As an example, enemies are weak against certain spell elements; thus, it requires much less effort to defeat them using the correct spell. However, in a first phase, the correct spell is hinted to the child, and in a second phase, the child should associate each enemy with the right magic elements (i.e., fire, water, wind, etc.) and spell.

2.1.2 The game mechanics

The sequence of actions required to cast and launch a spell is as follows: a) impose the index finger on the example symbol starting point, shown as a red circle, and wait for a flame to appear; b) keeping the index finger pose unaltered, follow the path shown and stop when the end is reached; c) leave the index finger pose and open the hand to generate the spell on the palm; d) with a slight and linear hand movement, launch the spell when the guiding purple ray trace points at the enemy. An example of these steps is shown in Figure 1. The two feedback provided during the drawing action in this application are haptic feedback, given by the external haptic device, and a visual feedback (colored circle), given by the virtual environment: both follow the speed tracking error. In the case of a more cognitive challenging task, dynamic time warping (DTW) is used to process position tracking data, different from a more conventional position tracking method that would, instead, constrain the game to always use a reference trajectory and, hence, a hinted symbol. After a 2D spatial projection, sub-sampling and scaling of the path were tracked by the index finger position; a DTW algorithm is used to classify the traced symbol in a set of known symbols and to compute a matching value (Figure 2). Nevertheless, this algorithm does not influence either the feedback or the pattern drawing: its effectiveness only concerns real applications of this scenario in rehabilitation. As a specification, the symbol set in Figure 2, in the output selection column, is just an example set: in this study, only the “O” letter was used. In addition to movement precision, in the presented study, we promoted the achievement of smooth and constant velocity movements, supported by haptic rendering described in the following section.

FIGURE 1

FIGURE 1. (A) Presented VR scenario with haptic feedback, implemented in the clinical setting of the TELOS project. The child is immersed in the role of a wizard apprentice. (B) Drawing of the magic symbols, consisting in a path tracking exercise. (C) Casting of the spell toward the target.

FIGURE 2

FIGURE 2. Symbol drawing exercise and structure of data processing: it supports both hinted and free spell tracing, allowing for introduction of cognitive elements in the game.

2.2 The haptic design

2.2.1 The haptic thimbles

The system implements two lightweight and thin haptic thimbles (Figure 1A) we developed and preliminary presented in a previous study by Camardella et al. (2022). We focused the mechanical design on the rendering of high-frequency components and fast transients. This reduces power requirements and size of the actuators at the cost of limited rendering capabilities. The actuator is a 1 degree of freedom miniaturized and custom-made electromagnetic voice coil. The moving part of the actuator is placed in contact with the finger pad through a soft and compliant structure of the whole thimble: this requires no calibration of the device and a overall high level of adaptation to different finger sizes. The design of the frontal part increases the compliance of the suspended voice coil and of the two lateral brackets adapting to different finger sizes. The thimble is fabricated in soft resin (Photocentric UV DLP Flexible) by stereo-lithography 3D printing. Total mass of the device is just 7 g. The shape of the device is thin and compact and reduces inter-finger interferences, resulting in compliance with the hand-tracking system embedded in the Oculus Quest 2 headset.

2.2.2 Haptic rendering

The haptic rendering strategy comes from the idea of enriching the interaction with the virtual environment, providing also an informative sensory feedback able to guide the user in the rehabilitation exercise.

In the scope of the proposed path tracking task, we aimed at rendering a haptic feedback informative of the finger speed. Information was mapped into the haptic signal: the absolute finger speed—the norm of the index phalanx speed—and the error between the finger speed and a reference one. The idea was to provide information on the absolute speed by modulating the rate of vibration transients with the absolute speed value and information on the error from the reference speed by changing the type of “texture” associated with each transient. This haptic rendering strategy comes from the simulation of real texture perception: if the finger explores a surface with coarse haptic features (i.e., grating texture), transients are spatially located; hence, rate is proportional to velocity as in the developed feedback. The resulting rendering approach resembles haptic rendering methods proposed in the literature to simulate artificial textures Wiertlewski et al. (2011). Coarse, discrete texture transients are rendered with spatial reference rather than time reference.

The information mapping schema is presented in Figure 3. The absolute speed of the finger—which is computed using the Oculus embedded tracking data, and differentiated and low-pass filtered with a 10-Hz cut-off frequency Butterworth filter—modulates the frequency of a square wave with an amplitude equal to one, where the duration of the “ON” state is fixed to 0.1 sec whereas the duration of the “OFF” state is equal to Δx/v−0.1 sec, where Δx is a spatial constant (in cm) and v is the actual speed (in cm/s). Then, the square wave multiplies white noise generating a sequence of the white-noise window as the input source of the texture generators.

FIGURE 3

FIGURE 3. Developed haptic rendering strategy, encoding information on the absolute velocity (rate of transients) and distance from the reference velocity (modulation of the type of texture).

The texture of the spells was computed with an autoregressive (AR) model structure. Other previous works made use of AR models (Romano et al., 2010; Romano and Kuchenbecker, 2011), especially under the assumption of strong stationary conditions. An AR model is an all-pole infinite impulse response (IIR) filter in which the system’s next output is modeled as a linear combination of its previous outputs. The AR model order, p, specifies the number of previous output values used. The AR model structure is as follows:

$A\left(p\right)y\left(t\right)=e\left(t\right)+u\left(t\right),(1)$

where A(p) is the array of AR coefficients, and y(t), e(t), and u(t) are, respectively, the output, white noise disturbance value, and residual at discrete time t. The discrete time transfer function of an AR model is as follows:

$H\left(z\right)=\frac{1}{1-{\sum }_{k=1}^p{A}_k{z}^{-k}}.(2)$

In our work, the coefficients of each AR model are taken from a work by Culbertson et al. (2014) that are computed by minimizing the residual error variance between the recorded acceleration of a sliding device on the given surface and the model of the samples using the Levinson–Durbin algorithm.

The chosen texture for rendering was a plastic material to provide a smooth feedback in order to inform that the actual speed is close to the reference—we use the Plastic Mesh 2 Model 18 of Culbertson et al. (2014)—whereas a more crisp material was used to inform that the actual speed is far from the reference—we used Ceramic Model 12 of Culbertson et al. (2014).

The final output is a linear combination of the two textures. When the error between the actual speed and the reference is above the error threshold, only texture 2 is rendered. In the opposite case, when the speed error is equal to zero, only texture 1 is rendered. The device computes and actuates the feedback at 1 kHz.

2.3 Experiment design

2.3.1 Participants

An experimental session with healthy adults has been performed and presented here in order to evaluate the effectiveness of the proposed haptic feedback before integration in the clinical setting. Eight adult participants (30.25 ± 5.04 years old, two females and 6 males) were enrolled.

2.3.2 Setup

The setup included the Oculus Quest 2 VR system, the two aforementioned haptic thimbles, and a host PC for the data collection routine. Each participant sat on a chair in the middle of a clear game area, previously drawn in the Oculus Quest VR system. The participant was asked to wear the VR headset and run six different experimental conditions with a 5-min rest phase between each of them. Each experimental condition required the participant to draw a symbol (a circle with marked start drawing points, as shown in Figure 1), performing a pointing hand pose, for a total of six repetitions. The latency between a real hand movement and its representation in the virtual environment is in the range of 0.10–0.12 s.

2.3.3 Experiment conditions

The following conditions were investigated: with and without speed reference, and with either haptic or visual feedback related to the tracking velocity. In the case of experiments with the speed reference, two speed values have been presented: slow (i.e. 8 cm/s) and a fast (i.e., 14 cm/s) value; these speed values have been empirically chosen to be both feasible for all the subjects without having any fatigue effect and different enough to better highlight potential differences. In the no-reference condition, participants were instructed to hold a self-chosen constant speed. Regarding haptic and visual feedback, haptic feedback was rendered, as described in Section 2.2.2. Visual velocity feedback was rendered as a red circle turning to green when the measured velocity was close to the reference, using the same linear modulation parameters used for the haptic feedback. All the conditions have been randomized to avoid potential biases in the results. In each experiment, the index fingertip position and speed has been recorded, together with game variables that allow for data segmentation. The movement data capture rate is 30 Hz.

2.3.4 Performance indexes and statistics

The dataset was built using the recorded hand speed as a feature, isolating the symbol drawing action timeframe only. In the case of experiments with the speed reference, the movement smoothness (quantified with the one-dimensional jerk index or JI) and the speed tracking error (quantified in cm/s) were computed; otherwise, only the smoothness was computed. The smoothness was calculated using the formula introduced in a work by Jarrassé et al. (2010). It is a jerk metric (the average rate of change of acceleration during a movement), and it is defined as follows:

$S=\frac{1}{t_{\mathit{end}}-t_{\mathit{init}}}{\int }_{t_{\mathit{init}}}^{t_{\mathit{end}}}\left[{\left(\frac{d^{3}x}{d{t}^3}\right)}^2+{\left(\frac{d^{3}y}{d{t}^3}\right)}^2+{\left(\frac{d^{3}z}{d{t}^3}\right)}^2\right]dt,(3)$

where [x, y, z] are the recorded endpoint coordinates. Large values indicate that many corrections are made during the movement. The tracking error was calculated as the root mean squared error between the user hand speed and the reference speed, sample by sample. For each participant, performance indexes were averaged among all the trials in the same experiment, and statistical tests were performed to detect potential significant differences among conditions. Given the non-normality of the analyzed dataset, non-parametric tests have been performed on both reference-based experiment data and without reference experiment data. For the reference-based experiment data, the Friedman test was used to detect differences among all the conditions, and then, the post hoc Wilcoxon signed-rank test, with the corrected alpha value, was performed on the following comparisons: smoothness with slow and fast speed reference between haptic and visual feedback and speed tracking error with slow and fast speed reference between haptic and visual feedback. For the experiment data without reference, the Wilcoxon signed-rank test was performed on the following comparisons: smoothness without speed reference between haptic and visual feedback and average speed without a speed reference between haptic and visual feedback.

3 Results

Experimental results are shown in graphs of Figure 4; details of the satistical analysis are described in the following paragraphs.

FIGURE 4

FIGURE 4. Experimental results measured in healthy participants regarding smoothness and velocity error in the path tracking serious game. Conditions varied for the presence or absence of a velocity reference and for the enabled haptic or visual feedback. The asterisks show the significance level of the comparison (* for p < 0.05).

3.1 With speed reference

Concerning the experiments with the speed reference, the Friedman test showed significant differences across conditions on both speed tracking error (p < .01, χ² = 19.20) and smoothness (p < .01, χ² = 21.89). Results have shown a significant difference on the movement smoothness between haptic (JI = 9.89 ± 3.74) and visual (JI = 13.86 ± 2.80) feedback, in the case of a slow speed (p = .023) differently from the fast speed (haptic JI = 28.93 ± 10.52; visual JI = 26.87 ± 10.12 and p = .312).

For the mean tracking speed error, no significant differences have been found neither in slow (haptic 2.68 ± 0.46 cm/s, visual 3.07 ± 0.81 cm/s, and p = .312) nor fast (haptic 4.57 ± 0.96 cm/s, visual 4.65 ± 0.64 cm/s, and p = .547) speeds.

3.2 Without speed reference

Concerning the experiments without the speed reference, results did not show any significant differences neither in the average speed (haptic 13.16 ± 4.05 cm/s, visual 10.65 ± 2.27 cm/s, and p = .195) nor in the smoothness (haptic JI = 24.38 ± 12.21 cm/s, visual JI = 18.59 ± 6.90 cm/s, and p = .385).

4 Discussion and conclusion

In this paper, we presented the design and implementation of a virtual reality serious game developed for rehabilitation of children with cerebral palsy. The introduction of a haptic feedback involved different aspects of the system design, including the game structure, the development of dedicated haptic rendering strategies, and hardware design of the wearable devices.

Preliminary experiments conducted in healthy participants showed that the perception of absolute speed, or the capability to follow a constant, self-selected reference speed, was not significantly changed by the introduction of a haptic feedback. Probably, information provided by proprioception and visual feedback was dominant and more precise in this condition than the rendered haptic feedback. On the other hand, when subjects were asked to follow a given reference speed, haptic feedback improved the performance in terms of a higher smoothness and lower speed tracking error, even if only the smoothness resulted to be significantly different. This difference was evident only during slow movements, probably caused by the higher amount of time subjects had to adapt using the additional information coming from the haptic feedback. The usage of two different speed references was exploratory, but a further investigation could address which value may be optimal in terms of patients’ performance.

By implementing the experimental setup, it emerged that a limit of the proposed system is in the estimation of the tracking speed through the hand-tracking data provided by the Oculus Quest. Although the virtual representation of hands appears very convincing and adequately smooth, speed estimation data resulted noisy and required filtering. This is due to the tracking data derivation process that emphasizes both the movement regularity information and the tracking measurement noise. This might have affected the performance of the modulated feedback, especially in the no-reference condition, where proprioception can result in a more reliable source of information for the central nervous system. Another reason can be found in the discretized nature of the absolute velocity information we rendered (rate of discrete transients) that might not have a sufficient resolution in time to increase performance with respect to proprioception. An alternative method to obtain more precise velocity estimation would be the use of external tracking sensors (i.e., optical markers); however, this would strongly diminish advantages of the proposed technology in terms of usability in the clinical setting: the use of the embedded tracking system greatly simplifies the setup in clinical settings, allowing operation by non-technical, clinical personnel. It can also be considered that hand tracking is a novel feature of the latest VR devices, and improvements in the technology can be expected in the upcoming years. Together with the tracking speed quality, the latency between a real and a virtual-rendered movement strongly affects users’ performance, pretty much following the concept of FPS (frames per second) in VR settings. A hand pose update latency higher than 200–300 ms may cause incoherence in users’ movement prediction (i.e., eye movement) (Vercher and Gauthier, 1992), leading to a lower smoothness and, thus, to unreliable measures.

In previous research, we already proposed the use of wearable haptic devices for tactile feedback in immersive VR for children neuro-rehabilitation (Bortone et al., 2018, 2020). Here, the system was reshaped in order to comply with the latest VR technologies and to fully take advantage of certain features that are precious for the rehabilitation setting. The lightweight, wireless setup of the latest HMD increases comfort and dexterity, with obvious expected advantages in a rehabilitation setting based on motor tasks. Moreover, the vision-based tracking system (for both user’s head and hands) embedded in the HMDs is a huge step toward usability of the setting by non-technical personnel. To comply with that, fingertip wearable haptics should not compromise with the gained dexterity and had to avoid interference with the tracking system. Hence, the developed thin haptic devices were based on a different approach (rendering of high-dynamic transients and high-frequency components) with respect to the previous design used (Bortone et al., 2018).

The novel available VR technologies allow also for more rich virtual environments and higher quality of the visual feedback, with expected benefits for the immersion and engagement of the user within the serious game. A fantasy setting with a fantasy background for the player (the player is in the role of a wizard apprentice) was conceived to possibly increase the interest of the child for the scenario and the child’s motivation to succeed.

Overall, the experience showed feasibility of haptic feedback integration in a novel, immersive VR scenario for rehabilitation. The achievement of a compact, wireless, and highly wearable VR setup including haptic feedback is important in the perspective of the next clinical evaluation and pilot rehabilitation studies. This becomes even more critical, considering prolonged clinical studies, in which reliability of the system and usability for the patient and the clinical personnel are key features. The perception of haptic feedback and immersive virtual environments by the targeted patient group is a feature that still needs to be investigated, especially when considering potential benefits that they could provide into the rehabilitation therapy. The current status of the system is already suitable for the aforementioned group since its design, on both the haptic device and the game scenario, was made for this purpose. Nevertheless, multi-sensory feedback need to be carefully chosen and tuned, to boost the focus and engagement of the patient and not to create distractions that would negatively affect the outcome of the rehabilitation. In this case, a trade-off needs to be found, and future studies will better analyze this aspect.

Data availability statement

The raw data supporting the conclusion of this article will be made available by the authors, without undue reservation.

Ethics statement

The study was reviewed and approved by the Board of Ethics of Scuola Superiore Sant’Anna of Pisa, Italy, approval number 15-2021 https://www.santannapisa.it/en/university/board-ethics. The participants provided their written informed consent to participate in this study.

Author contributions

CC contributed to the development of the virtual serious game, to its evaluation in the user study, to the data collection, analysis, and writing. DC contributed to the development of the haptic rendering strategies, to the experiments, data analysis, and writing. DL contributed to the design and development of the haptic devices, to the implementation of the virtual serious games, and to writing and revision. IB contributed to the design of the rehabilitation exercise and to revision. AF contributed to revision of the paper. The work was funded by DL, IB, and AF.

Funding

This work was supported by the Project “TELOS—Tailored neurorehabilitation therapy via multi-domain data analytics and adaptive serious games for children with cerebral palSy,” which is funded under the call “Bando Ricerca Salute 2018” of the Tuscany region, Italy (CUP J52F20001040002). The publication was created with the co-financing of the European Union—FSE-REACT-EU, PON Research and Innovation 2014-2020 DM1062/2021.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors, and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

Basalp, E., Wolf, P., and Marchal-Crespo, L. (2021). Haptic training: Which types facilitate (re) learning of which motor task and for whom answers by a review. IEEE Trans. haptics 14, 722–739. doi:10.1109/toh.2021.3104518

PubMed Abstract | CrossRef Full Text | Google Scholar

Booth, A. T., Van Der Krogt, M. M., Harlaar, J., Dominici, N., and Buizer, A. I. (2019). Muscle synergies in response to biofeedback-driven gait adaptations in children with cerebral palsy. Front. physiology 10, 1208. doi:10.3389/fphys.2019.01208

PubMed Abstract | CrossRef Full Text | Google Scholar

Bortone, I., Barsotti, M., Leonardis, D., Crecchi, A., Tozzini, A., Bonfiglio, L., et al. (2020). Immersive virtual environments and wearable haptic devices in rehabilitation of children with neuromotor impairments: A single-blind randomized controlled crossover pilot study. J. neuroengineering rehabilitation 17, 144–214. doi:10.1186/s12984-020-00771-6

PubMed Abstract | CrossRef Full Text | Google Scholar

Bortone, I., Leonardis, D., Mastronicola, N., Crecchi, A., Bonfiglio, L., Procopio, C., et al. (2018). Wearable haptics and immersive virtual reality rehabilitation training in children with neuromotor impairments. IEEE Trans. Neural Syst. Rehabilitation Eng. 26, 1469–1478. doi:10.1109/tnsre.2018.2846814

PubMed Abstract | CrossRef Full Text | Google Scholar

Bosecker, C., Dipietro, L., Volpe, B., and Igo Krebs, H. (2010). Kinematic robot-based evaluation scales and clinical counterparts to measure upper limb motor performance in patients with chronic stroke. Neurorehabilitation neural repair 24, 62–69. doi:10.1177/1545968309343214

PubMed Abstract | CrossRef Full Text | Google Scholar

Bossenbroek, R., Wols, A., Weerdmeester, J., Lichtwarck-Aschoff, A., Granic, I., van Rooij, M. M., et al. (2020). Efficacy of a virtual reality biofeedback game (deep) to reduce anxiety and disruptive classroom behavior: Single-case study. JMIR Ment. health 7, e16066. doi:10.2196/16066

PubMed Abstract | CrossRef Full Text | Google Scholar

Camardella, C., Gabardi, M., Frisoli, A., and Leonardis, D. (2022). Wearable haptics in a modern vr rehabilitation system: Design comparison for usability and engagement.” in International Conference on Human Haptic Sensing and Touch Enabled Computer Applications. Springer, 274.

CrossRef Full Text | Google Scholar

Chinello, F., Malvezzi, M., Pacchierotti, C., and Prattichizzo, D. (2015). “Design and development of a 3RRS wearable fingertip cutaneous device.” in 2015 IEEE International Conference on Advanced Intelligent Mechatronics (AIM), Busan, Korea, July 7–11, 2015 (IEEE), 293–298.

CrossRef Full Text | Google Scholar

Chiu, H.-C., and Kuo, P.-W. (2015). Effects of virtual reality in children with cerebral palsy: A systematic review. 40, 136

Google Scholar

Choi, J. Y., Yi, S.-H., Ao, L., Tang, X., Xu, X., Shim, D., et al. (2021). Virtual reality rehabilitation in children with brain injury: A randomized controlled trial. Dev. Med. Child Neurology 63, 480–487. doi:10.1111/dmcn.14762

PubMed Abstract | CrossRef Full Text | Google Scholar

Culbertson, H., Delgado, J. J. L., and Kuchenbecker, K. J. (2014). “One hundred data-driven haptic texture models and open-source methods for rendering on 3D objects,” in 2014 IEEE Haptics Symposium, Houston, TX, February 23–24, 2014 (IEEE), 319–325.

CrossRef Full Text | Google Scholar

Demain, S., Metcalf, C. D., Merrett, G. V., Zheng, D., and Cunningham, S. (2013). A narrative review on haptic devices: Relating the physiology and psychophysical properties of the hand to devices for rehabilitation in central nervous system disorders. Disabil. Rehabilitation Assistive Technol. 8, 181–189. doi:10.3109/17483107.2012.697532

PubMed Abstract | CrossRef Full Text | Google Scholar

Deutsch, J., and McCoy, S. W. (2017). Virtual reality and serious games in neurorehabilitation of children and adults: Prevention, plasticity and participation. Pediatr. Phys. Ther. official Publ. Sect. Pediatr. Am. Phys. Ther. Assoc. 29, S23–S36. doi:10.1097/pep.0000000000000387

PubMed Abstract | CrossRef Full Text | Google Scholar

Fani, S., Ciotti, S., Battaglia, E., Moscatelli, A., and Bianchi, M. (2017). W-Fyd: A wearable fabric-based display for haptic multi-cue delivery and tactile augmented reality. IEEE Trans. haptics 11, 304–316. doi:10.1109/toh.2017.2708717

PubMed Abstract | CrossRef Full Text | Google Scholar

Gabardi, M., Chiaradia, D., Leonardis, D., Solazzi, M., and Frisoli, A. (2018). A high performance thermal control for simulation of different materials in a fingertip haptic device.” in International Conference on Human Haptic Sensing and Touch Enabled Computer Applications, Pisa, Italy, June 13–16, 2018 (Cham: Springer), 313–325.

CrossRef Full Text | Google Scholar

Gallo, S., Rognini, G., Santos-Carreras, L., Vouga, T., Blanke, O., and Bleuler, H. (2015). Encoded and crossmodal thermal stimulation through a fingertip-sized haptic display. Front. Robotics AI 2, 25. doi:10.3389/frobt.2015.00025

CrossRef Full Text | Google Scholar

Gutierrez, A., Sepulveda-Munoz, D., Gil-Agudo, A., and de los Reyes Guzman, A. (2020). Serious game platform with haptic feedback and emg monitoring for upper limb rehabilitation and smoothness quantification on spinal cord injury patients. Appl. Sci. 10, 963. doi:10.3390/app10030963

CrossRef Full Text | Google Scholar

Jarrassé, N., Tagliabue, M., Robertson, J. V. G., Maiza, A., Crocher, V., Roby-Brami, A., et al. (2010). A methodology to quantify alterations in human upper limb movement during co-manipulation with an exoskeleton. IEEE Trans. Neural Syst. Rehabilitation Eng. 18, 389–397. doi:10.1109/TNSRE.2010.2056388

PubMed Abstract | CrossRef Full Text | Google Scholar

Kerns, K. A., Macoun, S., MacSween, J., Pei, J., and Hutchison, M. (2017). Attention and working memory training: A feasibility study in children with neurodevelopmental disorders. Appl. Neuropsychol. Child. 6, 120–137. doi:10.1080/21622965.2015.1109513

PubMed Abstract | CrossRef Full Text | Google Scholar

Kilteni, K., Groten, R., and Slater, M. (2012). The sense of embodiment in virtual reality. Presence Teleoperators Virtual Environ. 21, 373–387. doi:10.1162/pres_a_00124

CrossRef Full Text | Google Scholar

King, G., Chiarello, L. A., Ideishi, R., D’Arrigo, R., Smart, E., Ziviani, J., et al. (2020). The nature, value, and experience of engagement in pediatric rehabilitation: Perspectives of youth, caregivers, and service providers. Dev. Neurorehabilitation 23, 18–30. doi:10.1080/17518423.2019.1604580

PubMed Abstract | CrossRef Full Text | Google Scholar

Krebs, H. I., Fasoli, S. E., Dipietro, L., Fragala-Pinkham, M., Hughes, R., Stein, J., et al. (2012). Motor learning characterizes habilitation of children with hemiplegic cerebral palsy. Neurorehabilitation neural repair 26, 855–860. doi:10.1177/1545968311433427

PubMed Abstract | CrossRef Full Text | Google Scholar

Kuriakose, S., and Lahiri, U. (2015). Understanding the psycho-physiological implications of interaction with a virtual reality-based system in adolescents with autism: A feasibility study. IEEE Trans. Neural Syst. Rehabilitation Eng. 23, 665–675. doi:10.1109/tnsre.2015.2393891

PubMed Abstract | CrossRef Full Text | Google Scholar

Langhorne, P., Bernhardt, J., and Kwakkel, G. (2011). Stroke rehabilitation. Lancet 377, 1693–1702. doi:10.1016/s0140-6736(11)60325-5

PubMed Abstract | CrossRef Full Text | Google Scholar

Laver, K. E., Lange, B., George, S., Deutsch, J. E., Saposnik, G., and Crotty, M. (2017). Virtual reality for stroke rehabilitation. Cochrane database Syst. Rev. 2018, CD008349. doi:10.1002/14651858.cd008349.pub4

PubMed Abstract | CrossRef Full Text | Google Scholar

Leonardis, D., Solazzi, M., Bortone, I., and Frisoli, A. (2016). A 3-rsr haptic wearable device for rendering fingertip contact forces. IEEE Trans. haptics 10, 305–316. doi:10.1109/toh.2016.2640291

PubMed Abstract | CrossRef Full Text | Google Scholar

Martin-Niedecken, A. L., and Götz, U. (2017). “Go with the dual flow: Evaluating the psychophysiological adaptive fitness game environment “plunder planet”.” in Joint International Conference on serious games, Valencia, Spain, November 23–24, 2017 (Cham: Springer), 32–43.

CrossRef Full Text | Google Scholar

Pacchierotti, C., Sinclair, S., Solazzi, M., Frisoli, A., Hayward, V., and Prattichizzo, D. (2017). Wearable haptic systems for the fingertip and the hand: Taxonomy, review, and perspectives. IEEE Trans. haptics 10, 580–600. doi:10.1109/toh.2017.2689006

PubMed Abstract | CrossRef Full Text | Google Scholar

Patton, J., Dawe, G., Scharver, C., Mussa-Ivaldi, F., and Kenyon, R. (2006). Robotics and virtual reality: A perfect marriage for motor control research and rehabilitation. Assist. Technol. 18, 181–195. doi:10.1080/10400435.2006.10131917

PubMed Abstract | CrossRef Full Text | Google Scholar

Qiu, Q., Ramirez, D. A., Saleh, S., Fluet, G. G., Parikh, H. D., Kelly, D., et al. (2009). The New Jersey institute of technology robot-assisted virtual rehabilitation (njit-ravr) system for children with cerebral palsy: A feasibility study. J. neuroengineering rehabilitation 6, 40–10. doi:10.1186/1743-0003-6-40

PubMed Abstract | CrossRef Full Text | Google Scholar

Rand, D., Kizony, R., and Weiss, P. T. L. (2008). The sony playstation ii eyetoy: Low-cost virtual reality for use in rehabilitation. J. neurologic Phys. Ther. 32, 155–163. doi:10.1097/npt.0b013e31818ee779

PubMed Abstract | CrossRef Full Text | Google Scholar

Romano, J. M., and Kuchenbecker, K. J. (2011). Creating realistic virtual textures from contact acceleration data. IEEE Trans. haptics 5, 109–119. doi:10.1109/toh.2011.38

PubMed Abstract | CrossRef Full Text | Google Scholar

Romano, J. M., Yoshioka, T., and Kuchenbecker, K. J. (2010). “Automatic filter design for synthesis of haptic textures from recorded acceleration data.” in 2010 IEEE International Conference on Robotics and Automation, Anchorage, Alaska, May 3–8, 2010 (IEEE), 1815–1821. doi:10.1109/ROBOT.2010.5509853

CrossRef Full Text | Google Scholar

Simões, M., Bernardes, M., Barros, F., Castelo-Branco, M., et al. (2018). Virtual travel training for autism spectrum disorder: Proof-of-concept interventional study. JMIR serious games 6, e5. doi:10.2196/games.8428

PubMed Abstract | CrossRef Full Text | Google Scholar

Stroppa, F., Loconsole, C., Marcheschi, S., and Frisoli, A. (2017). “A robot-assisted neuro-rehabilitation system for post-stroke patients’ motor skill evaluation with ALEx exoskeleton.” in Converging Clinical and Engineering Research on Neurorehabilitation II, Segovia, Spain, October 18–21, 2016 (Cham: Springer), Based on proceedings of the 2016 International Conference on Neurorehabilitation, 501–505.

CrossRef Full Text | Google Scholar

Veerbeek, J. M., van Wegen, E., van Peppen, R., van der Wees, P. J., Hendriks, E., Rietberg, M., et al. (2014). What is the evidence for physical therapy poststroke? A systematic review and meta-analysis. PloS one 9, e87987. doi:10.1371/journal.pone.0087987

PubMed Abstract | CrossRef Full Text | Google Scholar

Vercher, J.-L., and Gauthier, G. (1992). Oculo-manual coordination control: Ocular and manual tracking of visual targets with delayed visual feedback of the hand motion. Exp. Brain Res. 90, 599–609. doi:10.1007/bf00230944

PubMed Abstract | CrossRef Full Text | Google Scholar

Wade, J., Zhang, L., Bian, D., Fan, J., Swanson, A., Weitlauf, A., et al. (2016). A gaze-contingent adaptive virtual reality driving environment for intervention in individuals with autism spectrum disorders. ACM Trans. Interact. Intelligent Syst. (TiiS) 6, 1–23. doi:10.1145/2892636

CrossRef Full Text | Google Scholar

Wiertlewski, M., Lozada, J., and Hayward, V. (2011). The spatial spectrum of tangential skin displacement can encode tactual texture. IEEE Trans. Robotics 27, 461–472. doi:10.1109/tro.2011.2132830

CrossRef Full Text | Google Scholar

Zahabi, M., and Abdul Razak, A. M. (2020). Adaptive virtual reality-based training: A systematic literature review and framework. Virtual Real. 24, 725–752. doi:10.1007/s10055-020-00434-w

CrossRef Full Text | Google Scholar

Zoccolillo, L., Morelli, D., Cincotti, F., Muzzioli, L., Gobbetti, T., Paolucci, S., et al. (2015). Video-game based therapy performed by children with cerebral palsy: A cross-over randomized controlled trial and a cross-sectional quantitative measure of physical activity. Eur. J. Phys. Rehabil. Med. 51, 669.

PubMed Abstract | Google Scholar