Fusion of dual modalities of non-invasive sensory feedback for object profiling with prosthetic hands

Introduction: Either non-invasive somatotopic or substitute sensory feedback is capable of conveying a single modality of sensory information from prosthetic hands to amputees. However, the neurocognitive ability of amputees to integrate multi-modality sensory information for functional discrimination is unclear. The purpose of this study was to assess the fusion of non-invasive somatotopic tactile and substitute aperture feedbacks for profile perception of multiple physical features during grasping objects.

Methods: Two left transradial amputees with somatotopic evoked tactile sensation (ETS) of five fingers participated in the study. The tactile information of prosthetic hand was provided to amputees by the ETS feedback elicited on the stump projected finger map. Hand aperture information was conveyed to amputees with substitute electrotactile stimulation on the forearm or upper arm. Two types of sensory feedback were integrated to a commercial prosthetic hand. The efficacy of somatotopic ETS feedback on object length identification task was assessed with or without substitute aperture stimulation. The object size identification task was utilized to assess how ETS stimulation at the stump may affect aperture perception with stimulation on the ipsilateral upper arm or forearm. Finally, the task of identifying combined length and size was conducted to evaluate the ability of amputees to integrate the dual modalities of sensory feedback for perceiving profile features.

Results: The study revealed that amputee subjects can effectively integrate the ETS feedback with electrotactile substitutive feedback for object profile discrimination. Specifically, ETS was robust to provide object length information with electrotactile stimulation at either the forearm or upper arm. However, electrotactile stimulation at the upper arm for aperture perception was less susceptible to the interference of ETS stimulation than at the forearm.

Discussion: Amputee subjects are able to combine somatotopic ETS and aperture feedbacks for identifying multi-dimensional features in object profiling. The two sensory streams of information can be fused effectively without mutual interference for functional discrimination.

1 Introduction

The fusion of proprioceptive receptors and cutaneous mechanoreceptors contributes to feeling and manipulating the external object (Edin and Abbs, 1991; Edin and Johansson, 1995; Kandel, 1999). Restoring tactile and proprioceptive information resulting from the loss of hand shows its improvement for the daily life of forearm amputees (Marasco et al., 2021; Vargas et al., 2021; Rostamian et al., 2022). Neural tactile afferents from hand digits provide necessary information about spatiotemporal patterns of contact for functional discrimination of physical features of objects, such as compliance and texture. The proprioceptive afferents inform movement and position of hand digits during grasping, which leads to the perception of object size. Their efficient fusion allows humans to interpret complex object shapes (Jones and Lederman, 2006). The absence of these sensory afferents in amputees limits their ability of functional perception using commercial prosthetic hands (Johansson and Flanagan, 2009). Thus, they often rely heavily on visual information for prosthetic control (Biddiss and Chau, 2007; Engdahl et al., 2015). Although various techniques of invasive or non-invasive somatotopic sensory feedback have been developed in the recent decade to convey tactile information for forearm amputees (Bensmaia et al., 2020; Weber et al., 2020; Mendez et al., 2021), the lack of proprioceptive information of prosthetic finger movements makes it difficult for subjects to discriminate multiple features of objects, such as shape.

Various invasive or non-invasive techniques have been widely developed to elicit localized sensations referred to the missing hand (Lawrence et al., 2004; Polasek et al., 2009; Boretius et al., 2010; Sharma et al., 2011; Antfolk et al., 2012; Ortiz-Catalan et al., 2014; Tan et al., 2014; Chai et al., 2015; D'Anna et al., 2017; Shin et al., 2018; Marasco et al., 2021; Zhang et al., 2022b). The restored contact tactile information allows forearm amputees to perceive the intensity of contact force for discrimination of object compliance (Raspopovic et al., 2014; D'Anna et al., 2017; George et al., 2019; Valle et al., 2020; Vargas et al., 2021), while it is also possible to identify simple object shapes or locations via contact timing difference of different parts of hand after training (Raspopovic et al., 2014; D'Anna et al., 2017; Valle et al., 2020). One technique of non-invasive somatotopic tactile sensory feedback developed leverages transcutaneous electrical nerve stimulation (TENS) on the specific projected finger map (PFM) area at the stump (Chai et al., 2015). The induced finger-specific sensation, referred to as evoked tactile sensation (ETS), is conveyed to amputees via a direct afferent pathway from periphery receptors in the stump skin to the primary somatosensory cortex (SI) (Hao et al., 2020). The activation of sensory nerves and mechanoreceptors at the stump PFM elicits rich sensory percepts for robust encoding of multiple modalities of sensory information (Li et al., 2021; Zhang et al., 2022b). This explains that ETS can be interpreted and utilized intuitively by amputees for perceiving finger-specific contact or force (Hao et al., 2020; Zhang et al., 2022a). ETS shows a high spatial resolution for perception of individual fingers, which leads to the feasibility of sensory coding for up to five fingers (Chai et al., 2015; Hao et al., 2020). The information on contact pattern and force intensity via ETS feedback enables forearm amputees to identify object length or compliance (Zhang et al., 2022a). However, it conveys only tactile information that may be limited for complex object profiling.

Stable proprioception remains difficult to elicit by activating directly the sensory afferent pathway. Proprioception is mediated by sensory fibers innervating muscle spindles and tendon organs (Proske and Gandevia, 2012) or receptors around the joint (Edin and Abbs, 1991; Edin and Johansson, 1995; Collins and Prochazka, 1996). The lack of this information in prosthetic hands forces amputees to rely on visual feedback to control prosthetic hand motion (Atkins et al., 1996; Biddiss and Chau, 2007). Proprioceptive and kinesthetic sensations are elicited in a few invasive feedback methods (Dhillon and Horch, 2005; Horch et al., 2011; Tan et al., 2014; George et al., 2019; Segil et al., 2020). However, the limited modulation range for proprioception makes it difficult to encode the spatiotemporal proprioceptive information (Segil et al., 2020). Sensory substitution is extensively utilized to restore proprioception (Schiefer et al., 2018; Chai et al., 2019; D'Anna et al., 2019; Marasco et al., 2021; Zhang et al., 2022a). The combination of somatotopic tactile and substitute proprioceptive feedback is demonstrated to be a feasible way to expand the information bandwidth of a single afferent stream (Schiefer et al., 2018; D'Anna et al., 2019; Marasco et al., 2021; Zhang et al., 2022a). Our preliminary study also shows the possibility of integrating ETS tactile and substitute electrotactile feedback for the identification of single physical property (Zhang et al., 2022a). Combining tactile and proprioceptive information gives the feasibility for multi-dimensional discrimination in object profiling. However, simultaneous surface electrical stimulation to elicit ETS and electrotactile sensation may cause a mutual interference in communicating different sensory information clearly and in discriminating multiple features of objects accurately.

In this study, we hypothesize that dual-modality sensory information from somatotopic tactile feedback with substitute aperture feedback could be fused and interpreted by amputee subjects for multi-dimensional discrimination. We investigated the functional benefits of fusing the two steams of sensory information for perception and discrimination of object length and size. Specifically, the mutual interference effect of ETS and electrotactile stimulation was also examined. The natural, finger-specific ETS stimulation and substitute electrotactile stimulation were integrated to inform amputees of the contact pattern of five fingers and the hand aperture. This two-modality sensory feedback was incorporated into a commercial prosthetic hand to achieve closed-loop control and profile perception. To evaluate the mutual interference of somatotopic ETS and proprioceptive substitution, the task of length and size identification of objects was conducted separately or in combination. The ETS feedback in discriminating object length was robust with or without additional substitute electrotactile stimulation. The interference of ETS stimulation on substitute sensory perception was evaluated with stimulation sites at either the forearm or upper arm. The task of combined identification of length and size was then conducted to demonstrate the efficacy of fusing two afferent information streams. The findings confirmed the hypothesis for combining the non-invasive somatotopic tactile information with substitutive aperture feedback to improve the sensory capabilities of hand prostheses. Preliminary work was presented in a conference paper elsewhere (Li et al., 2021).

2 Materials and methods

2.1 Subjects

Two transradial amputees were recruited to participate in this study. Both subjects (S1, male, 54 years old; S2, male, 66 years old) were with left wrist disarticulation. All the experimental protocols were approved by the Institutional Review Board for Human Research Protections, Shanghai Jiao Tong University. Subjects were informed about all experimental procedures and signed the consent form before joining the study.

2.2 Experimental setup

Five force sensors (FlexiForce A201, Tekscan Inc, the United States) were placed on the fingertips of the prosthetic hand to collect the contact force. Two flexible bend sensors (Spectra Symbol, Salt Lake City, UT) were fixed along the thumb and index finger for measuring hand aperture. The hand aperture distance between the fingertips of thumb and index was calculated by the difference in voltage outputs of the two flexible sensors. A real-time non-invasive multi-channel stimulation pattern generator (Figure 1) was costume designed and constructed for this study. It could sample five channels of contact force signals and two channels of hand aperture signals at the frequency of 100 Hz, and deliver six channels of stimulation current pulses to corresponding stimulation electrodes. Either non-woven fabric circular electrodes (Yancheng Dalun Medical Equipment Co. Ltd, China) with 1 cm diameter were used in this study. Non-woven fabric reference electrode (with 5 cm diameter) of each channel was placed near the olecranon of elbow.

FIGURE 1

Figure 1. Experiment setup and the illustration of an amputee using closed-loop control of a prosthetic hand for grasping and identifying different objects via non-invasive sensory feedback. The subject was blindfolded and wore noise-canceling headphones. The sensory feedback signals were obtained by force and bend sensors in the prosthetic hand. The somatotopic ETS feedback was utilized to provide intuitive contact information with TENS at the PFM of stump skin, while the substitute aperture feedback located on the upper arm or forearm supplied the hand aperture information. (A) In Experiments 1 and 2, the contralateral right hand was used for grasp control (Blue Line). (B) In Experiment 3, the ipsilateral left hand operated grasp control and perceived the sensory information with the anti-stimulus artifact module. A response button was pressed after reporting the identification to record the response time.

Figure 1 illustrates the experimental setup, in which the subject was seated at a table with eyes blinded and ears blacked with noise-canceling headphones. These setups were applied to block visual feedback and other incidental feedback, including auditory clue, transmitted by motor rotations. Two surface electromyographic (sEMG) sensors were placed on the wrist extensor and flexor carpi radialis to capture sEMG signals. In Experiments 1 and 2, the sEMG signals from the contralateral right arm were used to control the opening and closing of the Kesheng prosthetic right hand (Shanghai Kesheng Prostheses Co. Ltd, China) directly (Figure 1A). In Experiment 3, to further improve the integrated level of bidirectional prosthetic hand, the prosthetic hand (Otto Bock HealthCare, Germany) was changed into the ipsilateral control/sensing. The EMG signals of ipsilateral wrist flexor and extensor passed the anti-artifact model to remove the stimulation artifacts in real time (Yu et al., 2022) (Figure 1B).

2.3 Sensory feedback and coding strategy

In the somatotopic ETS feedback, the PFM was identified by pressing surface skin regions of the residual stump (Zhang et al., 2022b). The stimulation sites were fixed as the most sensitive points of five finger-specific PFM for tactile feedback of five fingers. For substitute aperture feedback, the stimulation site was set on the skin of the forearm or upper arm on the ipsilateral side of the amputation, which was both 10 cm away from the elbow and shoulder (Li et al., 2021).

The stimulation pulses of each channel were biphasic, charge-balanced, cathodic-first, rectangular pulses trains. An inter-pulse delay of 10 μs was present between the negative and the positive pulses (Szeto and Saunders, 1982). The pulse amplitude (PA), pulse width (PW) and pulse frequency (PF) of the current can be adjusted through the host computer. Since the buzz sensation was shown to have a wide pulse width modulation range and high sensitivity (Zhang et al., 2022b), it was chosen to encode sensory information.

For each stimulation site, PF was fixed at 50 Hz throughout the experiments. PA values and PW ranges were customized for each subject based on our previous protocols (Yang et al., 2020). PA increased from 1 mA with the step of 0.5 mA with fixed PF and PW (200 μs). The value that the buzz sensation could be perceived was identified as the fixed PA for this stimulation site. Then PW was increased from 10 to 600 μs with a step of 20–60 μs to find each PW_up and PW_low, the maximum and minimum values corresponding to the buzz sensation.

The coding strategy relationships between the sensor readouts and electrical stimulus are shown in Figure 2. The maximum and minimum force readouts of each finger were recorded and used as the upper and lower limits of the contact force, F_up and F_low, respectively. Contact forces were mapped to PWs (Figure 2A) according to Equation (1),

$\begin{array}{ll}PW=\frac{(P{W}_{up}-P{W}_{low})}{F_{up}-F_{low}}\ast \left(F-F_{low}\right)+P{W}_{low}& (1)\end{array}$

where F is the contact force between the prosthetic fingertip and the object. F_up is set as 15 N and F_low is 0 N.

FIGURE 2

Figure 2. The coding strategy of contact force and hand aperture. (A) The linear coding relationship between fingertip force and stimulus PW output, which was shown in Equation (1). PW_up and PW_low are the maximum and minimum pulse width thresholds, respectively. F_up is the maximum contact force during grasping for each finger, while F_low is the minimum force readout before grasping. (B, C) Show the staircase coding in Equation (2) and linear coding in Equation (3) for hand aperture.

The hand aperture (opening distance) was mapped to PW (Figure 2B) by segments according to Equation (2),

$\begin{array}{ll}PW=\{\begin{array}{c}P{W}_{up},When0<A\le A_{small}\\ \frac{(P{W}_{up}+P{W}_{low})}{2},When{A}_{small}<A\le A_{med}\\ P{W}_{low},When{A}_{med}<A\le A_{large}\\ 0,WhenA>A_{large}\end{array}& (2)\end{array}$

where A is the distance between the thumb and index finger, A_small, A_med, and A_large are equal to 4, 6, and 8 cm, respectively.

In particular, the complexity in Experiment 3 was adjusted with a design that was more relevant to activities of daily living, allowing the amputee to identify the opening/closing distance of the object electrical stimulus coding from the segments Equation (2) way to a linear coding Equation (3), by linear coding the amputee to judge the object as “sm” (small) or “lg” (large) (Figure 2C). Since the surface electrical stimulation for the same object was consistent, subjects could utilize same strategy to identify object size in different coding strategy.

$\begin{array}{ll}PW=\frac{(P{W}_{up}-P{W}_{low})}{P_{up}-P_{low}}\ast \left(A_{\lg }-A\right)+P{W}_{low}& (3)\end{array}$

where PW changes linearly with A when A is between 2 and 10 cm (A_lg).

2.4 Experiment design

2.4.1 Experiment 1: evaluation of length identification

In this task, subjects were asked to control the prosthetic hand to grasp the object using power grip. With this grip, all fingers were bent toward the object. When the finger touched the object, its contact force information was registered and conveyed back to the subject via ETS stimulation. If not, he finger continued to bend down until maximal flexion with no contact force. Then, they would identify the length of an object, chosen from four object lengths [6 cm (very short), 8 cm (short), 10 cm (long), and 12 cm (very long)] (Figure 3A) and reported orally. Different object lengths led to different numbers of contact fingers that would convey to subjects (Figure 3B). The ETS feedback was utilized to supply contact information, while the substitute subject on the ipsilateral left forearm (SF-fore) or left upper arm (SF-upper) was supplied to evaluate its effect on the efficacy of ETS feedback. The grasp of the prosthetic hand was controlled by the amputee's contralateral right hand. Each object appeared 15 times with ETS feedback and 10 times with other conditions randomly in 5 sessions.

FIGURE 3

Figure 3. Identification of object length. (A) Four objects with different lengths and same size were used in the length identification task and their labels. (B) The pattern of contact forces of five fingers from both subjects when grasping objects of four different lengths under all conditions. (C, D) The accuracy and response time with respect to three different feedback conditions. “#” identifies the level that is significantly different from the chance level. (E) The performance in the confusion matrix for both subjects identifying each object. *p < 0.05, ***p < 0.001).

2.4.2 Experiment 2: evaluation of hand aperture identification

During the size identification, three blocks of equally spaced sizes (4, 6, and 8 cm for small, medium and large blocks) were used (Figure 4A). Different sizes of objects during grasping were recorded (Figure 4B). With SF-upper/SF-fore, the subject was asked to grasp the object with power grip and report its size with or without the additional ETS feedback (ETSF). The grasp of the prosthetic hand was controlled by the amputee's contralateral right hand. Each object was randomly presented 15 times on each condition in five sessions.

FIGURE 4

Figure 4. Identification of object size. (A) Objects with three sizes at the same length used in size identification experiments. (B) The hand aperture information when S1 and S2 grasped objects of three different sizes under all conditions, i.e., substitute aperture feedback on upper-arm (SF-upper) and forearm (SF-fore), as well as with additional ETS feedback (ETSF) simultaneously. (C, D) The accuracy of size classification under four feedback conditions in the form of confusion matrix and overall performance, respectively. “#” identifies the level that is statistically different from the chance level. (E) The response time for S1 and S2 under four feedback conditions. *p < 0.05, **p < 0.01, and ***p < 0.001.

2.4.3 Experiment 3: evaluation of length and hand aperture identification

In the combined size and length identification task, objects with the same lengths as the length identification task and two sizes [2 cm (small) and 9 cm (large) in width] were used (Figure 5A). Both contact finger and aperture information were conveyed to subjects by ETS and substitute aperture feedback on upper-arm (Figure 5B). To further test subject's ability of ipsilateral closed-loop control for prosthetic hand, a Bebionic hand with the same myoelectric control strategy was utilized in Experiment 3. The change to ipsilateral side may influence the task complexity for amputee subjects to control the bidirectional prosthetic hand. Thus, before Experiment 3, each subject was given a training session of 5–10 min to familiarize the ipsilateral control and sensing. The subject was asked to answer the length and size of the object to be grasped and pressed the response button the simultaneous time under the control of the prosthetic hand of the amputated side. This experiment consisted of five sessions and each object was presented 10 times randomly.

FIGURE 5

Figure 5. Identification of object length and size. (A) Eight wooden blocks with four different lengths and two different sizes in this experiment session. (B) The overall contact force of five fingers and hand aperture information when grasping different objects by all subjects. (C) Performance during the task with ETS feedback and substitute aperture feedback on the upper-arm for S1 and S2 in the form of a confusion matrix (left) and boxplot identifying size/length (right). “#” identifies the level that is statistically different with the chance level. (D) Response time to identify each object in single size identification with ETS feedback (ETSF), single length identification with substitute aperture feedback on upper-arm (SF-upper), and the combined size and length identification task. “***”p < 0.001.

Identification accuracy and response time were recorded to evaluate the subjects' ability to perceive multiple fingers. Response time was defined as the interval between the moment when the contact force rose and when the subject reported the answer in Experiments 1 and 2, or when he pressed the response button in Experiment 3. Confusion matrixes were constructed for block size identification experiments to show the actual and perceived results by the subjects.

2.5 Statistical analysis

All data were analyzed using IBM SPSS Statistics version 26.0 (SPSS Inc., Chicago, IL, USA, 2019). The one-sample Kolmogorov-Smirnov test was used to test if the dataset was normally distributed. Since none dataset passed the test, Kruskal-Wallis test (K-W test) was used to determine whether there was a significant difference in accuracy, response time and aperture. The identification of both subjects was compared against chance level with the Binominal test.

3 Results

3.1 The ETS stimulation for multiple digits feedback

The role of multi-digit ETS feedback in the identification of object dimension was evaluated first with objects of four different lengths (Figure 3A). Without visual and auditory feedbacks, length identification of objects resulted primarily from distinguishing the fingers contacting with the object (Figure 3B). Evaluation of efficacy of ETS feedback was conducted under three conditions: (1) ETS feedback only (ETSF), ETS feedback with substitute stimulation on upper-arm (SF-upper) (2) or forearm (SF-fore) (3). Results showed first that ETS feedback alone allowed both subjects to identify object length with high accuracy (98.3% for S1 and 90.0% for S2) (Figure 3C). This indicated that the ETS feedback provided sufficient information for length classification through the pattern of contacting fingers. Second, both subjects could maintain a high accuracy of length identification with additional substitute stimulation on upper-arm (90.0% for S1 and 87.5% for S2) or forearm (100% for S1 and 87.5% for S2) (Figure 3C). Although some confusion occurred between “long” and “very long” lengths (Figure 3E), the overall performance was all above the chance level (Figure 3C). These results suggested that both subjects acquired the ability of robust perception of the basic pattern of contacting fingers via ETS feedback despite the additional substitute stimulation (Figure 3B). Response time of identification appeared to vary with subjects and testing conditions, but both subjects could in general interpret basic information provided in ETS feedback on the order of 3–5 (s) (Figure 3D). This finding further supported the notion that it is possible to combine ETS feedback with substitute stimulation to inform more sensory information.

3.2 Substitute stimulation for hand aperture (opening) feedback

The role of substitute stimulation on the upper-arm or forearm to inform object size was evaluated with an aim to combine with the ETS feedback. Test results showed significantly different levels of stimulation relating to hand apertures (Figure 4B). Under substitute stimulation only, both subjects could identify object size with high accuracy on either upper-arm stimulation (93.3% for S1 and 93.3% for S2) or forearm stimulation (100% for S1 and 93.3% for S2) (Figures 4C, D). Distal ETS stimulation in both subjects affected size perception with substitute simulation on forearm more notably than that on upper-arm (Figure 4C). The performance under four different feedback conditions was all above the chance level (Figure 4D). Response time appeared to vary with subjects and ETS stimulation (Figure 4E). These results suggested that upper-arm substitute feedback might better isolate perception interference from ETS. Nevertheless, substitute feedback conveyed sufficient information regarding the size dimension of object via hand aperture/opening. Therefore, it might be beneficial to fuse the ETS feedback and substitute feedback at upper-arm stimulation to provide multi-modality sensory information.

3.3 Fusion of ETS and substitute stimulation for multi-modality sensory feedback

A more challenging task of recognizing object size and length simultaneously was conducted to understand the efficacy of perceiving multi-modality sensory information via ETS and aperture feedbacks on upper-arm. The ETS pattern of contact digits and hand aperture information was presented to the subject for identification (Figure 5B). The overall accuracy for length and size identification was 95% for S1 and 58.75% for S2, all of which were above the chance level (Figure 5C). The feature of only size or length was all identified above chance levels (Figure 5C). But S2 had a much higher rate of success in recognizing size than length (Figure 5C). Results indicated that information provided by the two sensory feedbacks could be integrated for size and length perception by the two subjects. In both cases, the response time in the combined size and length identification was longer than that in a single size or length identification (Figure 5D). This showed that task complexity added a greater burden to the cognitive integration and processing of sensory information. This test established that ETS-based multi-digit tactile sensory feedback could be combined with substitute aperture feedback to provide two-dimensional information about objects, and the information could be integrated and processed by both subjects in distinguishing object size and length.

4 Discussion

The sensory information bandwidth of a single sensory feedback approach is limited for object profiling. To provide multiple sensory information, the combination of somatotopic and substitute sensory feedback approaches has been demonstrated to be effective for the identification of various physical properties (Schiefer et al., 2018; D'Anna et al., 2019; Vargas et al., 2021; Zhang et al., 2022a). In our approach with ETS-based somatotopic sensory feedback, previous studies showed the ability to restore the finger-specific tactile sensation via a naturally regenerated afferent pathway (Hao et al., 2020; Zhang et al., 2022a). This study was to explore the neurocognitive ability of amputees to fuse two different streams of somatotopic tactile and substitute aperture information. Results confirmed the robustness of somatotopic ETS and substitute aperture feedbacks for dimensional perception. The study demonstrated the feasibility to combine two different information streams by amputee subjects for effective functional object profiling. Results also suggested the optimal location for substitute aperture stimulation in the upper arm of amputees.

Somatotopic ETS feedback displayed robust communication of tactile information against distortion from distant stimulation on the ipsilateral arm for substitute aperture feedback. The evoked afferents can be perceived by amputees as occurring in the lost digits in a natural, intuitive and direct way (Hao et al., 2020). The finger-specific sensation allowed subjects to identify the object length effectively (Figure 3B). Both subjects achieved high success rates by relying on discriminating the contact pattern of fingers provided by ETS (98.3% for S1 and 90.0% for S2). The performance was comparable to those reported in the literature in similar tasks that leveraged timing differences in contact location information with invasive peripheral nerve stimulation at 97.3% (Raspopovic et al., 2014) or 95% (Segil et al., 2020), and proximal nerve stimulation at 84.3% (D'Anna et al., 2017) (Table 1). Among these techniques, ETS-based feedback is one of the few methods (Segil et al., 2020) capable of both individually and simultaneously conveying tactile information up to five fingers. There was no difference in accuracy with or without substitute feedback on forearm and upper arm (p > 0.05, Figure 3C). This demonstrated that the ability of both amputee subjects to interpret the somatotopic tactile information were preserved even with concurrent substitute electrotactile stimulation (Figure 3C). The impact of additional sensory stimulation on the efficacy of ETS feedback mainly showed in the response time of identification (Figure 3D). Both subjects needed less identification time when the sensory substitution was closer to the PFM area (p < 0.05). It may be related to the cognitive process of perceiving two sources of information separated in different locations in the arm. The confusion of identification for S2 mainly occurred between “long” and “very long” objects (Figure 3E). It may be due to that the pinky finger had relatively low force intensity compared to other fingers, which may be related to the relative position of object and prosthetic hand. This task demonstrates that amputee subjects could focus and interpret the intuitive finger-specific tactile information even with two simultaneous afferent information streams. Since the ETS feedback delivers a natural digit-specific afferent stream to the proprio-somatotopic hand areas in SI (Hao et al., 2020), it does not interfere with the substitutive afferent delivered to the upper arm area in SI. This implies that the somatotopic separated sensory afferents can be more clearly represented in SI and more robustly interpreted in the fusion of sensory information (D'Anna et al., 2019; Valle et al., 2020).

TABLE 1

Table 1. Comparison of literatures in object feature identification via multiple-finger information.

Different from robust ETS-based feedback, the substitute sensory feedback could be disturbed by distal ETS stimulation. For size identification with upper-arm stimulation, our performance (93.3% for S1 and 93.3% for S2) was comparable to that of peripheral nerve stimulation at 96.9% (George et al., 2019), or substitute stimulation at 78% (D'Anna et al., 2019) and 84% (Su et al., 2022). Meanwhile, the average response time of our subjects was 4.01 (s) in S1 and 4.72 (s) in S2, much shorter than 8.94 (s) in the previous study (George et al., 2019) (Table 2). The identification accuracy of S1 was significantly impacted by ETS stimulation when the substitute aperture feedback was placed on the forearm (p < 0.05, Figure 4D). In this feedback scenario, both subjects required more response time with additional ETS stimulation (p < 0.01, Figure 4E), where upper arm stimulation required less time than forearm stimulation (p < 0.01). This space-dependent effect implied that there may be an electric field interference to substitute perception. Considering the robustness of ETS feedback and the minimal crosstalk in both electric field and cortical SI representation, placing the sensory substitution stimulation on the upper arm may be a preferred choice for further integration of multi-modality sensory feedback. It suggests a guideline for choosing the location of sensory substitution in the integration of an electrotactile feedback with the ETS feedback in the ipsilateral arm of amputees.

TABLE 2

Table 2. Comparison of literatures in object size identification.

The ability to interpret two different information streams relating to the ETS in digits and the aperture of hand was further evaluated. In combined identification tasks with length information via ETS and size information via substitute stimulation, both length and size of objects could be identified simultaneously with an accuracy of 95% by S1 and 58.75% by S2, all were above the chance level (p < 0.05, Figure 5C). Our previous study showed that this hybrid multi-modality feedback enabled the forearm subject to identify object compliance with the accuracy of 100 % (Zhang et al., 2022a). These results illustrated that amputee subjects could not only fuse two different afferent sensory information to confirm a single object feature, but also flexibly utilize them for the identification of multiple physical properties. It also indicated the feasibility of hybrid ETS and substitute aperture feedback, which allowed effective identification of object length, size, and compliance. The information dimension conveyed by this non-invasive approach was comparable to that of previous studies (Schiefer et al., 2018; D'Anna et al., 2019; Vargas et al., 2021). Both subjects required more time in the identification of combined features than single size or length (p < 0.001, Figure 5D), which indicated that the cognitive process may be related to the complexity of the task and the channels of information being processed simultaneously (Freides, 1974; Ernst and Bülthoff, 2004; Antfolk et al., 2013), as well as the prior information available for the integration (Klatzky et al., 1985; Ernst and Bülthoff, 2004). The fusion of ETS and substitute aperture feedbacks for object profiling requires a higher level of ability to integrate received information and to make an optimal judgment based on perception (Ernst and Banks, 2002). Since these two sensory afferents interact with each other subconsciously, the reliability and variance of sensory fusion may require learning to obtain optimal discrimination of integrated information (Ernst and Bülthoff, 2004; Dayan and Daw, 2008; Štrbac et al., 2017). However, the effects of learning await further investigation in future studies.

There were two limitations in this study. Although the sensory fusion study was tested only in two amputee subjects, the performance trends of both subjects were consistent, demonstrating that the neurocognitive ability of subjects is capable of processing two modalities of simultaneous sensory information effectively. In future studies, this multi-modality sensory feedback will be evaluated in a larger population of forearm amputee subjects to demonstrate functional improvements in tasks more relevant to daily living and working. Second, two different commercial prosthetic hands were used in the combined identification task and single feature identification using the bidirectional prosthetic hand. This was a necessary step to upgrade from contralateral control/sensing to more integrated ipsilateral control/sensing, and test the effectiveness of ipsilateral bi-directional communication. With some training, both subjects were able to overcome the change brought about by switching from contralateral control/sensing to ipsilateral communication. However, the control mode and experimental paradigm for both prosthetic hands remained the same in all tasks, which ensured consistency in results.

This study investigated and demonstrated the neurocognitive ability of amputees to combine dual-modality of sensory information based on the somatotopic ETS and the substitute stimulation. The ETS feedback showed a robust ability to convey finger-specific information in the presence of additional sensory stimulation. The farther away the aperture stimulation location from the stump site of ETS stimulation, the less interference to aperture perception from ETS stimulation. Thus, the substitute aperture feedback in upper arm stimulation was the preferred way to incorporate with the ETS feedback in amputees. Test results confirmed that the hybrid sensory feedback enabled both subjects to identify the combined size and length of grasped objects effectively. This study reveals the robustness of non-invasive ETS to restore somatotopic sensation and illustrates the cognitive ability of amputees to fuse different sensory information for effective functional discrimination. This study established the feasibility of sensory fusion in amputees to combine tactile and proprioceptive information through non-invasive techniques of sensory feedback for improving prosthesis functionality.

Data availability statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics statement

The studies involving humans were approved by Institutional Review Board for Human Research Protections, Shanghai Jiao Tong University. The studies were conducted in accordance with the local legislation and institutional requirements. The participants provided their written informed consent to participate in this study. Written informed consent was obtained from the individual(s) for the publication of any potentially identifiable images or data included in this article.

Author contributions

JZ: Data curation, Methodology, Validation, Writing—original draft, Writing—review & editing. C-HC: Methodology, Software, Writing—original draft. MH: Methodology, Supervision, Writing—original draft. YL: Writing—original draft. YY: Writing—original draft. NL: Writing—original draft, Writing—review & editing.

Funding

The author(s) declare financial support was received for the research, authorship, and/or publication of this article. This research was supported in part by the National Key R&D Program of China (Nos. 2017YFA0701104 and 2020YFC2007903), the National Natural Science Foundation of China (No. 81630050), Key-Area Research and Development Program of Guangdong Province (2020B0909020004), and the Science and Technology Commission of Shanghai Municipality (No. 20DZ2220400).

Acknowledgments

We are grateful to the participants who voluntarily took part in this study.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

The author(s) declared that they were an editorial board member of Frontiers, at the time of submission. This had no impact on the peer review process and the final decision.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

Antfolk, C., Björkman, A., Frank, S., Sebelius, F., Lundborg, G., Rosen, B., et al. (2012). Sensory feedback from a prosthetic hand based on air-mediated pressure from the hand to the forearm skin. J. Rehabil. Med. 44, 702–727. doi: 10.2340/16501977-1001

PubMed Abstract | CrossRef Full Text | Google Scholar

Antfolk, C., D'Alonzo, M., Rosén, B., Lundborg, G., Sebelius, F., Cipriani, C., et al. (2013). Sensory feedback in upper limb prosthetics. Expert Rev. Med. Devices 10, 45–54. doi: 10.1586/erd.12.68

PubMed Abstract | CrossRef Full Text | Google Scholar

Atkins, D. J., Heard, D. C. Y., and Donovan, W. H. (1996). Epidemiologic overview of individuals with upper-limb loss and their reported research priorities. J. Prosthet. Orthot. 8, 2–11. doi: 10.1097/00008526-199600810-00003

CrossRef Full Text | Google Scholar

Bensmaia, S. J., Tyler, D. J., and Micera, S. (2020). Restoration of sensory information via bionic hands. Nat. Biomed. Eng. 7, 443–455. doi: 10.1038/s41551-020-00630-8

PubMed Abstract | CrossRef Full Text | Google Scholar

Biddiss, E. A., and Chau, T. T. (2007). Upper limb prosthesis use and abandonment: a survey of the last 25 years. Prosthet. Orthot. Int. 31, 236–257. doi: 10.1080/03093640600994581

PubMed Abstract | CrossRef Full Text | Google Scholar

Boretius, T., Badia, J., Pascual-Font, A., Schuettler, M., Navarro, X., Yoshida, K., et al. (2010). A transverse intrafascicular multichannel electrode (TIME) to interface with the peripheral nerve. Biosens. Bioelectron. 26, 62–69. doi: 10.1016/j.bios.2010.05.010

PubMed Abstract | CrossRef Full Text | Google Scholar

Chai, G., Briand, J., Su, S., Sheng, X., and Zhu, X. (2019). “Electrotactile feedback with spatial and mixed coding for object identification and closed-loop control of grasping force in myoelectric prostheses,” in 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (Berlin: EMBC), 1805–1808.

PubMed Abstract | Google Scholar

Chai, G., Sui, X., Li, S., He, L., and Lan, N. (2015). Characterization of evoked tactile sensation in forearm amputees with transcutaneous electrical nerve stimulation. J. Neural Eng. 12, 066002. doi: 10.1088/1741-2560/12/6/066002

PubMed Abstract | CrossRef Full Text | Google Scholar

Collins, D. F., and Prochazka, A. (1996). Movement illusions evoked by ensemble cutaneous input from the dorsum of the human hand. J. Physiol. 496, 857–871. doi: 10.1113/jphysiol.1996.sp021733

PubMed Abstract | CrossRef Full Text | Google Scholar

D'Anna, E., Petrini, F. M., Artoni, F., Popovic, I., Simanić, I., Raspopovic, S., et al. (2017). A somatotopic bidirectional hand prosthesis with transcutaneous electrical nerve stimulation based sensory feedback. Sci Rep. 7. doi: 10.1038/s41598-017-11306-w

PubMed Abstract | CrossRef Full Text | Google Scholar

D'Anna, E., Valle, G., Mazzoni, A., Strauss, I., Iberite, F., Patton, J., et al. (2019). A closed-loop hand prosthesis with simultaneous intraneural tactile and position feedback. Sci Rob. 4, eaau8892. doi: 10.1126/scirobotics.aau8892

PubMed Abstract | CrossRef Full Text | Google Scholar

Dayan, P., and Daw, N. D. (2008). Decision theory, reinforcement learning, and the brain. Cogn. Affect. Behav. Neurosci. 8, 429–453. doi: 10.3758/CABN.8.4.429

PubMed Abstract | CrossRef Full Text | Google Scholar

Dhillon, G. S., and Horch, K. W. (2005). Direct neural sensory feedback and control of a prosthetic arm. IEEE Trans. Neural Syst. Rehabil. Eng. 13, 468–472. doi: 10.1109/TNSRE.2005.856072

PubMed Abstract | CrossRef Full Text | Google Scholar

Edin, B. B., and Abbs, J. H. (1991). Finger movement responses of cutaneous mechanoreceptors in the dorsal skin of the human hand. J. Neurophysiol. 65, 657–670. doi: 10.1152/jn.1991.65.3.657

PubMed Abstract | CrossRef Full Text | Google Scholar

Edin, B. B., and Johansson, N. (1995). Skin strain patterns provide kinaesthetic information to the human central nervous system. J. Physiol. 487, 243–251. doi: 10.1113/jphysiol.1995.sp020875

PubMed Abstract | CrossRef Full Text | Google Scholar

Engdahl, S. M., Christie, B. P., Kelly, B., Davis, A., Chestek, C. A., Gates, D. H., et al. (2015). Surveying the interest of individuals with upper limb loss in novel prosthetic control techniques. J. Neuroeng. Rehabil. 12, 53. doi: 10.1186/s12984-015-0044-2

PubMed Abstract | CrossRef Full Text | Google Scholar

Ernst, M. O., and Banks, M. S. (2002). Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415, 429–433. doi: 10.1038/415429a

PubMed Abstract | CrossRef Full Text | Google Scholar

Ernst, M. O., and Bülthoff, H. H. (2004). Merging the senses into a robust percept. Trends Cognit. Sci. 8, 162–169. doi: 10.1016/j.tics.2004.02.002

PubMed Abstract | CrossRef Full Text | Google Scholar

Freides, D. (1974). Human information processing and sensory modality: cross-modal functions, information complexity, memory, and deficit. Psychol. Bull. 81, 284–310. doi: 10.1037/h0036331

PubMed Abstract | CrossRef Full Text | Google Scholar

George, J. A., Kluger, D. T., Davis, T. S., Wendelken, S. M., Okorokova, E. V., and He, Q. Biomimetic sensory feedback through peripheral nerve stimulation improves dexterous use of a bionic hand. Sci. Robot. (2019) 4, eaax2352. doi: 10.1126/scirobotics.aax2352.

PubMed Abstract | CrossRef Full Text | Google Scholar

Hao, M., Chou, C. H., Zhang, J., Yang, F., Cao, C., Yin, P., et al. (2020). Restoring finger-specific sensory feedback for transradial amputees via non-invasive evoked tactile sensation. IEEE Open J. Eng. Med. Biol. 1, 98–107. doi: 10.1109/OJEMB.2020.2981566

PubMed Abstract | CrossRef Full Text | Google Scholar

Horch, K., Meek, S., Taylor, T. G., and Hutchinson, D. T. (2011). Object discrimination with an artificial hand using electrical stimulation of peripheral tactile and proprioceptive pathways with intrafascicular electrodes. IEEE Trans. Neural Syst. Rehabil. Eng. 19, 483–489. doi: 10.1109/TNSRE.2011.2162635

PubMed Abstract | CrossRef Full Text | Google Scholar

Johansson, R. S., and Flanagan, J. R. (2009). Coding and use of tactile signals from the fingertips in object manipulation tasks. Nat. Rev. Neurosci. 10, 345–359. doi: 10.1038/nrn2621

PubMed Abstract | CrossRef Full Text | Google Scholar

Jones, L., and Lederman, S. (2006). Human hand function. Politics 32. doi: 10.1093/acprof:oso/9780195173154.001.0001

CrossRef Full Text | Google Scholar

Kandel, E. (1999). Principles of Neural Science. New York, NY: McGraw-Hill.

Google Scholar

Klatzky, R. L., Lederman, S. J., and Metzger, V. A. (1985). Identifying objects by touch: an “expert system.” Percept. Psychophys. 37, 299–302. doi: 10.3758/BF03211351

CrossRef Full Text | Google Scholar

Lawrence, S. M., Dhillon, G. S., Jensen, W., Yoshida, K., and Horch, K. W. (2004). Acute peripheral nerve recording characteristics of polymer-based longitudinal intrafascicular electrodes. IEEE Trans. Neural Syst. Rehabil. Eng. 12, 345–348. doi: 10.1109/TNSRE.2004.831491

PubMed Abstract | CrossRef Full Text | Google Scholar

Li, Y., Chou, C. H., Zhang, J., Zhang, Z., Hao, M., and Lan, N. (2021). “A pilot study of multi-site simultaneous stimulation for tactile and opening information feedback in the prosthetic hand,” in 2021 10th International IEEE/EMBS Conference on Neural Engineering (NER) (IEEE), 187–190.

Google Scholar

Marasco, P. D., Hebert, J. S., Sensinger, J. W., Beckler, D. T., Thumser, Z. C., Shehata, A. W., et al. (2021). Neurorobotic fusion of prosthetic touch, kinesthesia, and movement in bionic upper limbs promotes intrinsic brain behaviors. Sci Robot. 6, abf3368. doi: 10.1126/scirobotics.abf3368

PubMed Abstract | CrossRef Full Text | Google Scholar

Mendez, V., Iberite, F., Shokur, S., and Micera, S. (2021). Current solutions and future trends for robotic prosthetic hands. Annu. Rev. Control Robot. Auton. Syst. 4, annurev-control-071020-104336. doi: 10.1146/annurev-control-071020-104336

CrossRef Full Text | Google Scholar

Ortiz-Catalan, M., Håkansson, B., and Brånemark, R. (2014). An osseointegrated human-machine gateway for long-term sensory feedback and motor control of artificial limbs. Sci. Transl. Med. 6, 257re.6. doi: 10.1126/scitranslmed.3008933

PubMed Abstract | CrossRef Full Text | Google Scholar

Polasek, K. H., Hoyen, H. A., Keith, M. W., Kirsch, R. F., and Tyler, D. J. (2009). Stimulation stability and selectivity of chronically implanted multicontact nerve cuff electrodes in the human upper extremity. IEEE Trans. Neural Syst. Rehabil. Eng. 17, 428–437. doi: 10.1109/TNSRE.2009.2032603

PubMed Abstract | CrossRef Full Text | Google Scholar

Proske, U., and Gandevia, S. C. (2012). The proprioceptive senses: their roles in signaling body shape, body position and movement, and muscle force. Physiol. Rev. 92, 1651–1697. doi: 10.1152/physrev.00048.2011

PubMed Abstract | CrossRef Full Text | Google Scholar

Raspopovic, S., Capogrosso, M., Petrini, F. M., Bonizzato, M., Rigosa, J., Pino, D. I. G., et al. (2014). Restoring natural sensory feedback in real-time bidirectional hand prostheses. Sci. Transl. Med. 6, 222ra19. doi: 10.1126/scitranslmed.3006820

PubMed Abstract | CrossRef Full Text | Google Scholar

Rostamian, B., Koolani, M., Abdollahzade, P., Lankarany, M., Falotico, E., Amiri, M., et al. (2022). Texture recognition based on multi-sensory integration of proprioceptive and tactile signals. Sci. Rep. 12, 21690. doi: 10.1038/s41598-022-24640-5

PubMed Abstract | CrossRef Full Text | Google Scholar

Schiefer, M. A., Graczyk, E. L., Sidik, S. M., Tan, D. W., and Tyler, D. J. (2018). Artificial tactile and proprioceptive feedback improves performance and confidence on object identification tasks. PLoS ONE 13, e0207659. doi: 10.1371/journal.pone.0207659

PubMed Abstract | CrossRef Full Text | Google Scholar

Segil, J. L., Cuberovic, I., Graczyk, E. L., Weir, R. F., and Tyler, D. (2020). Combination of simultaneous artificial sensory percepts to identify prosthetic hand postures: a case study. Sci. Rep. 10, 6576. doi: 10.1038/s41598-020-62970-4

PubMed Abstract | CrossRef Full Text | Google Scholar

Sharma, A., Rieth, L., Tathireddy, P., Harrison, R., Oppermann, H., Klein, M., et al. (2011). Long term in vitro functional stability and recording longevity of fully integrated wireless neural interfaces based on the Utah Slant Electrode Array. J. Neural Eng. 8, 045004. doi: 10.1088/1741-2560/8/4/045004

PubMed Abstract | CrossRef Full Text | Google Scholar

Shin, H., Watkins, Z., Huang, H., Zhu, Y., and Hu, X. (2018). Evoked haptic sensations in the hand via non-invasive proximal nerve stimulation. J. Neural Eng. 15, 046005. doi: 10.1088/1741-2552/aabd5d

PubMed Abstract | CrossRef Full Text | Google Scholar

Štrbac, M., Isaković, M., Belić, M., Popović, I., Simanić, I., Farina, D., et al. (2017). Short- and long-term learning of feedforward control of a myoelectric prosthesis with sensory feedback by amputees. IEEE Trans. Neural Syst. Rehabil. Eng. 25, 2133–2145. doi: 10.1109/TNSRE.2017.2712287

PubMed Abstract | CrossRef Full Text | Google Scholar

Su, S., Chai, G., Meng, J., Sheng, X., Mouraux, A., Zhu, X., et al. (2022). Towards optimizing the non-invasive sensory feedback interfaces in a neural prosthetic control. J. Neural Eng. 19, 016028. doi: 10.1088/1741-2552/ac4e1b

PubMed Abstract | CrossRef Full Text | Google Scholar

Szeto, A. Y., and Saunders, F. A. (1982). Electrocutaneous stimulation for sensory communication in rehabilitation engineering. IEEE Trans. Biomed. Eng. 29, 300–308.

PubMed Abstract | Google Scholar

Tan, D. W., Schiefer, M. A., Keith, M. W., Anderson, J. R., Tyler, J., Tyler, D. J. A., et al. (2014). neural interface provides long-term stable natural touch perception. Sci. Transl. Med. 6, 257ra.138. doi: 10.1126/scitranslmed.3008669

PubMed Abstract | CrossRef Full Text | Google Scholar

Valle, G., D'Anna, E., Strauss, I., Clemente, F., Granata, G., Iorio, D. I. R., et al. (2020). Hand control with invasive feedback is not impaired by increased cognitive load. Front. Bioeng. Biotechnol. 8, 287. doi: 10.3389/fbioe.2020.00287

PubMed Abstract | CrossRef Full Text | Google Scholar

Vargas, L., Huang, H. H., Zhu, Y., and Hu, X. (2021). Object recognition via evoked sensory feedback during control of a prosthetic hand. IEEE Robot Autom. Lett. 1, 207–214. doi: 10.1109/LRA.2021.3122897

PubMed Abstract | CrossRef Full Text | Google Scholar

Weber, D. J., Hao, M., Urbin, M. A., Schoenewald, C., and Lan, N. (2020). Sensory information feedback for neural prostheses. Biomed. Inform. Technol. 687–715. doi: 10.1016/B978-0-12-816034-3.00021-3

CrossRef Full Text | Google Scholar

Yang, F., Hao, M. Z., Zhang, J., Chou, C. H., and Lan, N. (2020). An experimental protocol for evaluating pulse width modulation ranges of evoked tactile sensory feedback in amputees. Annu. Int. Conf. IEEE Eng. Med. Biol. Soc. 2020, 3869–3872. doi: 10.1109/EMBC44109.2020.9175756

PubMed Abstract | CrossRef Full Text | Google Scholar

Yu, Y., Chou, C. H., Zhang, J., Hao, M. Z., and Lan, N. (2022). “Development of myoelectric control module for prosthetic hand with artifact removal during sensory electrical stimulation,” in 2022 15th International Joint Conference on Biomedical Engineering Systems and Technologies, 118–125. [Epub ahead of print].

Google Scholar

Zhang, J., Chou, C. H., Wu, X., Pei, W., and Lan, N. (2022b). “Non-invasive stable sensory feedback for closed-loop control of hand prosthesis,” in 2022 44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC). Scottish Event Campus, Glasgow, UK: 2022 44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC) (Glasgow).

PubMed Abstract | Google Scholar

Zhang, J., Hao, M., Yang, F., Liang, W., Sun, A., Chou, C. H., et al. (2022a). Evaluation of multiple perceptual qualities of transcutaneous electrical nerve stimulation for evoked tactile sensation in forearm amputees. J. Neural Eng. 19, 026041. doi: 10.1088/1741-2552/ac6062

PubMed Abstract | CrossRef Full Text | Google Scholar