Object stiffness recognition and vibratory feedback without ad-hoc sensing on the Hannes prosthesis: A machine learning approach

Introduction: In recent years, hand prostheses achieved relevant improvements in term of both motor and functional recovery. However, the rate of devices abandonment, also due to their poor embodiment, is still high. The embodiment defines the integration of an external object – in this case a prosthetic device – into the body scheme of an individual. One of the limiting factors causing lack of embodiment is the absence of a direct interaction between user and environment. Many studies focused on the extraction of tactile information via custom electronic skin technologies coupled with dedicated haptic feedback, though increasing the complexity of the prosthetic system. Contrary wise, this paper stems from the authors' preliminary works on multi-body prosthetic hand modeling and the identification of possible intrinsic information to assess object stiffness during interaction.

Methods: Based on these initial findings, this work presents the design, implementation and clinical validation of a novel real-time stiffness detection strategy, without ad-hoc sensing, based on a Non-linear Logistic Regression (NLR) classifier. This exploits the minimum grasp information available from an under-sensorized and under-actuated myoelectric prosthetic hand, Hannes. The NLR algorithm takes as input motor-side current, encoder position, and reference position of the hand and provides as output a classification of the grasped object (no-object, rigid object, and soft object). This information is then transmitted to the user via vibratory feedback to close the loop between user control and prosthesis interaction. This implementation was validated through a user study conducted both on able bodied subjects and amputees.

Results: The classifier achieved excellent performance in terms of F1Score (94.93%). Further, the able-bodied subjects and amputees were able to successfully detect the objects' stiffness with a F1Score of 94.08% and 86.41%, respectively, by using our proposed feedback strategy. This strategy allowed amputees to quickly recognize the objects' stiffness (response time of 2.82 s), indicating high intuitiveness, and it was overall appreciated as demonstrated by the questionnaire. Furthermore, an embodiment improvement was also obtained as highlighted by the proprioceptive drift toward the prosthesis (0.7 cm).

1. Introduction

Upper limb loss is a serious impairment due to its explicit and direct interaction with the external world. To compensate for this loss, prostheses have been introduced to restore the functionality of human limbs during activities of daily living (ADLs). This necessity led to the development of high-tech devices with multiple degrees of freedom (Medynski and Rattray, 2011; Van Der Niet and Van Der Sluis, 2013), capable of performing a variety of gestures and grasps. However, the embodiment of these devices into the human body scheme and their acceptance are also essential elements for reconnection with the outside world (Cuberovic et al., 2019; Castellini, 2020). The term “Embodiment” means the integration of an external object in the internal corporal scheme as if it was part of the body itself. In this specific context, the external object is, precisely, the prosthesis (Longo et al., 2008). Embodiment comprises three correlated factors, namely, ownership, localization, and agency (Stiegelmar et al., 2020), and it has been suggested to promote intuitive control, learning, and comfort when using new tools, thus providing the opportunity to improve the user interface for devices such as artificial limbs. The introduction of direct feedback modalities can prevent amputees to rely exclusively on sight (Biddiss et al., 2007; Pylatiuk et al., 2007), reducing the mental effort and, therefore, facilitating the communication between user intention and prosthesis action (Markovic et al., 2018; Valle et al., 2018; Clemente et al., 2019). In fact, it has been demonstrated that the introduction of haptic feedback improves the control of the prosthesis (Mayer et al., 2020; Sensinger and Dosen, 2020; Yildiz et al., 2020; Chai et al., 2022) due to its fundamental role during human–objects interactions (Hsiao et al., 2011; Valle et al., 2018; Pena et al., 2019; Di Pino et al., 2020; Shehata et al., 2020; Raspopovic et al., 2021), allowing subjects to embody the device (Antfolk et al., 2013; Svensson et al., 2017; Raspopovic et al., 2021), hence, improving the compliance among the user, the prosthesis, and the grasped objects (Osborn et al., 2016). In the literature, this interaction is mainly assessed by providing grasp force or proprioceptive information (Stephens-Fripp et al., 2018). Contrarily, the aim of this study is to deliver information about the grasped object's stiffness that in normal conditions, occurs due to the combination of visual sensory information, proprioceptive sensations related to shape and size, and tactile sensations related to stiffness (Garland and Miles, 1997). Therefore, the current research activity offers an intuitive, non-invasive, and easy-to-use prosthetic system capable of identifying simple grasped object proprieties when visual sensory information of the user is not available or limited (Sensinger and Dosen, 2020). For instance, when the user is taking an object from a bag without looking at it or when the light in the environment is off. This situation was also treated by the Cybathlon 2020 competition, which introduced the Haptic Box task, considering it as a common ADL (Caserta et al., 2022).

Focusing on tactile sensations, several studies tried to reproduce the properties of human skin endowing the device with tactile sensing technologies that typically requires cumbersome add-on like sensing skin with different kinds of sensors such as piezoresistive (Osborn et al., 2018), capacitive (Cannata et al., 2008), piezoelectric (Yi and Zhang, 2016), and also optical (Zhao et al., 2016). The measurements acquired by these tactile sensors are often given as input to machine learning algorithms, which extract useful information that may be conveyed to the prosthesis users, as described by Jamali and Sammut (2011), Liarokapis et al. (2015), Konstantinova et al. (2017), Devaraja et al. (2020), Huang and Rosendo (2022).

Once the tactile information has been extracted, it is necessary to effectively provide it to the subject. The sensory substitution process can be exploited non-invasively, involving the connection of a certain event with specific feedback that is not the natural one, such as tactile sensory feedback (Clemente et al., 2015; Dosen et al., 2016; Patel et al., 2016; Štrbac et al., 2016; Sensinger and Dosen, 2020). For example, the subject can be taught to associate a certain vibratory stimulus with the contact of the prosthesis with an object (Antfolk et al., 2012b; Clemente et al., 2015, 2019; Dosen et al., 2016; Štrbac et al., 2016; De Nunzio et al., 2017; Nemah et al., 2020; Mamidanna et al., 2021). In contrast, superficial stimulation could target portions of the missing limb's skin that are innervated by afferent neurons after the amputation, the so-called referred touch, to stimulate the phantom limb and improve the embodiment (Antfolk et al., 2013; Masteller et al., 2021), such as kinesthetic sensory feedback.

The most common feedback restoration method is through vibration (Masteller et al., 2021), given its compatibility with electromyography (EMG) control and better acceptance by the subjects with respect to electrostimulation, capable of stimulating phantom limb sensation with electric surface charge (Shannon, 1976; Kaczmarek et al., 1991; Vargas et al., 2019). It is possible to provide different types of information acting on the amplitude and frequency of the vibration, as exploited in the study of Witteveen et al. (2013), in which the magnitude of the grasp force was transmitted using different levels of amplitude. An alternative to this feedback is the mechanotactile, as proposed using tactors by Meek et al. (1989), producing a one-to-one correspondence of touch sensation to user stimulation, or with a cuff, as proposed by Casini et al. (2015).

However, despite the high potentiality offered by these solutions, they are mainly bulky and heavy, and difficult to integrate, along with high-power consumption due to high computational burden. An example is proposed by Antfolk et al. (2012a) who designed a touch sensory feedback via air-mediated pressure from the hand to the forearm skin. This is a no-power solution that has neither impact on power consumption nor on computational burden. However, the final integration within the prosthesis does not guarantee the anthropomorphism of the hand device. It is also important to point out that the quick disconnection between the socket and the hand prosthesis is lost due to the mechanical connections running from the fingers' hand to the on-socket actuators. Standard devices use an electronic slip ring combined with a quick disconnect mechanism integrated into the prono-supinator wrist to guarantee the overall disconnection of the hand prosthesis from the socket in case of emergency. However, in the proposed design, this feature is compromised. Other examples are Oddo et al. (2016) and Shehata et al. (2018) who proposed an artificial fingertip to improve the performance of prosthetic hands by using intraneural stimulation. That solution can be nicely integrated into a fingertip by maintaining the anthropomorphic characteristics. However, the on-board electronics that record, process the tactile information, and encode the stimulation are cumbersome. Moreover, the high-power consumption of the FPGA-based solution does not permit the entire system to last for an entire day and to fit into a standard socket. Similarly, Clemente et al. (2019) developed a solution whose electronic skin offers high sensitivity ranging from light touch to heavy touch. However, a similar integration problem of the dedicated board occurs. In contrast, Vargas et al. (2021) finally proposed force and position sensors on the fingers to provide object stiffness recognition on amputees through vibrotactile feedback. That solution can be easily integrated; however, the performances of such a solution are limited in comparison with our results. Due to these issues, the lack of a suitable feedback restitution method in the prosthetic field is still far from being solved. Two other solutions for object stiffness recognition, without dedicated sensors, were implemented by Balasubramanian et al. (2021) and Wang et al. (2021). Their studies demonstrated the feasibility of these approaches in a robotic scenario using an actuated mechanical gripper.

Considering the advantages of providing feedback to amputees to improve the comfort between the user and the device, in this study, we first investigated the possibility of detecting void grasp and object grasp. Then, we identify the softness and hardness of the objects, therefore, permitting the user to discriminate among “void grasp,” “rigid object,” and “soft object” without visual sensory information. In the first preliminary study (Bruni and Bucchieri, 2021), a virtual multi-body model of Hannes was developed to offline demonstrate, with a virtual simulation, how the motor-side current absorption and the position measurement could be correlated with the hand grasp force and the grasped object's stiffness. Subsequently, in the following study (Bruni et al., 2022), an Ensemble Bagged Trees classifier was implemented and offline tested with simulated data to validate an approach to distinguish two different objects' stiffness.

Consequently, in the present article, we exploited the previously preliminary validated approach to develop an online (real-time) solution to perform object stiffness recognition and sensory feedback. The performance of this solution was assessed on end-users, both able-bodied and amputees. A non-linear logistic regression (NLR) classifier was used to recognize rigid or soft objects and void grasps. We excluded embedded force sensors, whose introduction would require facing many challenges, starting from the choice of the right sensor with basic requirements like high resolution, high sensitivity, and robustness, to the difficulties of managing the wiring (Kappassov et al., 2015). Instead, we proposed a methodology that uses intrinsic sensors (sensors and parameters already available on the prosthesis) for the normal functionality of the prosthesis that does not increase the cost and complexity of the device. In particular, we exploited the following intrinsic sensors: the motor-side current, whose relationship with the contact stiffness has been analytically demonstrated by Deng et al. (2020); the reference position, given as input to control the device closure; and the position effectively measured by the encoder (encoder position). We implemented a closed-loop vibratory feedback, using a single vibromotor embedded in the Hannes system, closely related to the predictions made by the classifier. In detail, we applied the strategy of strong vibration for rigid objects and small vibration for soft objects, which was identified in this study as “Two Feedback (2FB) condition” (Cipriani et al., 2011; Tejeiro et al., 2012). In the first phase, the classifier performance and the 2FB effectiveness were evaluated with 18 able-bodied subjects by measuring the classification accuracy through F1Score. In the second phase, a comparison between our proposed feedback method (2FB) and three other control feedback conditions was carried out on five amputees. This comparison was performed both objectively by measuring F1Score, users' response time, and proprioceptive drift, and subjectively through the questionnaire to investigate the users' appreciation of the feedback strategies and identify the most intuitive and effective one.

2. Materials and methods

2.1. Subjects

A total of 18 able-bodied subjects aged between 24 and 50 years (28.8 ± 6.2) and 5 mono-lateral amputees (right transradial amputees and users of active prostheses) were recruited for this study, with the definition described in Table 1. Written informed consent was obtained from all the subjects. The experimental protocol was approved by the Area Vasta Emilia Centro (AVEC) Ethics Committee (Protocol Code: CP-PPRAS1/1-03) and performed in accordance with the guidelines of the Declaration of Helsinki.

TABLE 1

Table 1. Population of amputees.

2.2. Experimental setup

The experimental setup that used for performing the entire experiment (Figure 1) was composed of (A) the myoelectric prosthesis Hannes, fixed on a rigid cone; (B) a custom master-board to control the hand, decode the stiffness of the grasped object, and communicate with the PC via Bluetooth; (C) two EMG sensors (standard Ottobock, 13E200 = 50 AC) to close or open the hand; (D) an eccentric rotating mass (ERM) vibromotor to convey the feedback; (E) a power supply for the prosthetic system; (F) two wristbands to attach the EMG sensors and the vibromotor to the subject's forearm; (G) three rigid objects and three soft objects with spherical, cubic, and cylindrical shape used during the Cybathlon 2020 edition (Medynski and Rattray, 2011; Caserta et al., 2022); (H) a laptop to choose the feedback condition and to collect the data; (I) a keyboard, placed in front of the subject, to press the left (rigid object) and right (soft object) arrows to indicate the guessed stiffness of the grasped object; and (J) headphones reproducing white noise to prevent the users from hearing the prosthesis motor.

FIGURE 1

Figure 1. Experimental setup used to perform the task on healthy subjects and amputees. It consists of: (A) socket with the Hannes prosthesis; (B) the EMGM to control the Hannes hand, recognize the grasped object, and provide feedback; (C) EMG sensors; (D) vibromotors; (E) power supply; (F) elastic band to attach the EMGs and vibromotors to the forearm; (G) the objects used to perform the task; (H) laptop with the virtual reality; (I) keyboard to choose between rigid, soft, or void; (J) headphones to isolate the participants during the experiment.

The vibromotor was inserted into a custom silicone holder to localize and absorb the radiating stimulation and to avoid the possible heating of the skin due to prolonged vibration. The vibromotor was placed vertically with respect to the skin to produce a stronger and more focused sensation. The vibration frequency was set to 200 Hz, using a supply voltage of 2.5 V (Vybronics, 2021)¹, and the amplitude was varied through pulse width modulation (PWM).

2.2.1. The Hannes hand

Hannes is an under-actuated poly-articulated prosthetic hand characterized by a leader-follower wire configuration used to control the movements of fingers (Laffranchi et al., 2020). The hand powertrain consists of a single DC motor coupled with a custom planetary gearhead, which drives the grasping movement (refer to Supplementary material). The actuation system is controlled by a position reference (ϑ_ref) synthesized from the user's EMG signals. A magnetic encoder measures the slow shaft position (ϑ_out) of the hand drive train, therefore, controlling the desired grasp configuration. The low-level control system is based on a series of proportional–integrative–derivative (PID) controllers. The outer loop is position based (where only proportional and derivative (PD) terms are deployed), while the inner loop is current based and concerns proportional and integrative (PI) terms only. In particular, the error (ε_pos) between ϑ_ref (hand control command) and ϑ_out (outer feedback) is fed to the outer PD loop. The related output is then multiplied by a proportional gain, resulting in a current reference (i_ref) which is subtracted with the measured one (i_out, inner secondary feedback which is the current absorbed by the DC motor during hand movement and grasp). As consequence, the related error (ε_i) is then fed to the inner PI controller, hence, generating the control command (V) to be delivered to the motor driver. As with many under-actuated prostheses, Hannes is under-sensorized. Indeed, the only available measurements are motor-side current and position.

2.2.2. Feedback conditions

The following four different feedback conditions were assessed in this study: (i) no FB condition (NoFB); (ii) audio FB condition (AFB); (iii) one FB condition (1FB); and (iv) two FB conditions (2FB). The NoFB condition was characterized by the absence of any possible feedback. The subjects were visibly (with closed eyes) and auditorily (headphones with white noise) blind and without any vibratory feedback. In the AFB condition, no vibratory feedback was supplied to the user, but the absence of the headphones permitted accidental auditory feedback of the moving prosthesis. In the 1FB condition, the vibratory feedback was provided, but the same vibration intensity (30% of PWM) was associated with both rigid and soft objects, while no vibration was given during void closures (refer to the table in Figure 2). The 2FB condition provided a strong vibration for rigid objects (100% of PWM) and a light vibration for soft objects (30% of PWM, a value found during some previous pilot tests to be perceived sufficiently different from the 100% used for rigid objects; refer to the table in Figure 2). As in the 1FB condition, void closures did not provide any kind of vibration. The no FB condition was implemented as a baseline for validation and comparison of subjects' performance. In fact, in the total absence of feedback, subjects' performance should be close to a random guess. The audio FB condition was introduced, since it represents a reasonable scenario of the use of the prosthetic hand by amputees, namely with no direct vision of the prosthesis but accidental auditory information from the prosthesis motion. Therefore, this second condition works as a real-case scenario ground truth for the user. The other two conditions, i.e., 1FB and 2FB, were implemented to observe, respectively, if additional vibratory feedback could improve the stiffness estimation performance, and if a different degree of vibration could further help amputees in discerning between harder and softer objects.

FIGURE 2

Figure 2. Scheme representing the object stiffness classification process. The motor-side current, the reference, and the encoder positions are acquired from Hannes and sent as input to the classifier, which gives as output the resulting stiffness.

2.3. Non-linear logistic regression

2.3.1. Algorithm model

The algorithm chosen for the object stiffness discrimination task is the NLR classifier. This machine learning algorithm was selected given the good performance shown for multiclass classification problems, and for simplicity reasons, since the NLR is already employed for the Hannes pattern recognition control strategy (Marinelli et al., 2020; Di Domenico et al., 2021). It is based on the calculation of the class membership probability through the following formulation:

$\begin{array}{l}P(1|x,\vartheta )=\{\begin{array}{c}g({\vartheta }^T·x)=\frac{1}{1+e^{-({\vartheta }^T·x+{\vartheta }_0)}}\\ 1-P(y=0|x,\vartheta )\\ \end{array}\end{array}$

Where ϑ and ϑ₀ are the internal parameters vector of the classifier and the bias term, respectively; x is the input feature vector, while g(·) is the sigmoid logistic function. The class prediction is obtained from the comparison between the distribution P(y|x) with a decision threshold (TH) as:

$\begin{array}{l}{h}_{\vartheta }(x)=\{\begin{array}{c}P(1|x,\vartheta )\ge TH\to 1\\ P(1|x,\vartheta )\le TH\to 0\\ \end{array}\end{array}$

The TH value was obtained after an optimization phase on the validation set. Since the NLR is a binary classifier, a One-vs-All approach was implemented to address the multiclass classification problem for the discrimination between rigid, soft, and void closures. This involves the use of as many binary classifiers as the classes for prediction, and each of them is trained to recognize the specific class. The model parameters (ϑ) are the result of an optimization process that involves the minimization of a cost function called cross-entropy error J:

$\begin{array}{l}J(\vartheta ,{\vartheta }_0)=-\frac{1}{m}·\left[{\displaystyle \sum _{i=1}^m}y(i)·ln(g({\vartheta }^T·x+{\vartheta }_0))\right]\\ -\frac{1}{m}·\left[{\displaystyle \sum _{i=1}^m}(1-y^{(i)})·ln(1-g({\vartheta }^T·x+{\vartheta }_0))\right]\end{array}$

Where m is the number of samples used to train the algorithm and y(i) is the known class membership of the i_th sample (Dellacasa Bellingegni et al., 2017; Marinelli et al., 2020).

2.3.2. Algorithm training

To adapt the model to distinguish multiple rigidities, the classifier required a training phase involving the repetitive closure of the prosthesis on objects of different stiffness. To simplify this work and to create a reproducible acquisition setup, a custom-made object was 3D printed. This device, as shown in Figure 3, was designed to reproduce the same shape and dimension of the Go Direct^® Hand Dynamometer (Vernier, 2021), used in the previous study (Bruni and Bucchieri, 2021), which offers the possibility to insert springs of different stiffness, simulating the grasping of soft and rigid objects, as shown in the table of Figure 3.

FIGURE 3

Figure 3. Custom-made hand dynamometer mounted on test bench used for the classifier training. The device is composed of a base where at maximum four strings can be inserted, and a cover which can slide along the base when compressed by the prosthesis fingers. The test bench consists of a custom-made system composed by movable parallel arms and a two-cuff system acting as holder. The table represents the different spring force and levels to characterize the stiffness of the dynamometer.

The device was mounted in an ad-hoc designed test bench. It was composed of movable parallel arms and a two-cuff system acting as a holder. The prosthesis was fixed at the base of this test bench, as shown in Figure 3, in such a way that only the distal phalanges of the four fingers had an impact on the upper plate of the device when performing a closure.

Hannes was controlled through a USB GUI, which allowed the data acquisition (motor-side current and encoder position) as well.

The NLR model generation was performed offline through MATLAB and it required training and test datasets, both characterized by the following four-column structure: (i) the motor-side current, (ii) the reference position sent as input, (iii) the encoder position measured, and (iv) the labels of the objects (rigid, soft, and void), as it is a supervised learning algorithm. All these measurements are fed to the classifier as analog signals; thus, they are directly used as the input dataset. Moreover, the label zero was associated with void closures (for motor-side currents, below 300 mA), one to the rigid objects and two to the soft objects. The dataset was created using the test bench described in Figure 3.

The choice of relying on only the motor current and the reference and measured motor encoder position was based on the immediate and relevant available sensor information on the prosthesis. Specifically, the motor current is proportional to the motor torque and, thus, to the grasp force, while the encoder position is related to the grasping motion of the fingers. In addition, the difference between reference and measured encoder position provides good information regarding the distinction between a void closure and the actual grasping of an object (this is due to the variation between the reference encoder positions that continues to grow due to EMG residuals, while the actual measured encoder position stops when encountering an object during grasp). These three quantities (current, reference, and measured positions) represent, according to the authors, the minimum set of variables to properly classify the different types of grasping (refer to the “Results” section for details on the performances). Nonetheless, it is worth mentioning that additional sensors or derived quantities could be beneficial for a more complex classifier structure. For example, motor speed, if not particularly noisy or delayed, could help in more advanced classification algorithms.

To generate the variability of the data, multiple grasps with various stiffness were performed by the prosthesis, which was controlled by both EMG and sinusoidal references. The hand dynamometer was used to simulate rigid objects, while four types of springs with distinct stiffness were used to reproduce a range of softness/soft objects, as shown in the table of Figure 3. The springs are placed under a bar to distribute the stiffness of their combination to the entire grasp. The chosen combination of springs is different for each case because the total stiffness of parallel springs varies according to their sum, thus affecting the total grasp behavior. In particular, several closures were performed for each case, as described in Table 2, to collect data for the training and validation of the NLR model. The training dataset was split into a training set (80%), used for the model generation (selection of the best model parameters (ϑ) by minimizing the cost function J), and a validation set (20%) to find the best threshold (TH). Lastly, the classifier was evaluated on the test dataset.

TABLE 2

Table 2. Dataset realization for training the NLR for object stiffness recognition algorithm.

2.4. Experimental protocol

The subjects were seated comfortably in front of a table (refer to Figure 4) with EMG sensors positioned on the forearm or stump using an elastic band. The electrodes measured the activity of the forearm muscles involved in the opening and closing of the hand (Flexor Carpi Ulnaris and Extensor Carpi Ulnaris, respectively), which were selected by manual inspection. The Hannes system was detached from the users' bodies (except for the two EMGs) and fixed on the table, lying between the subjects' arms with the palm up, to allow the experimenter place the objects to be grasped within the prosthetic hand. Hence, subjects were only asked to close and open their hand, not to approach or grasp the objects. The prosthesis was commanded in proportional-speed-control mode through the EMG signals. To convey the vibratory feedback, the vibromotor was positioned on the pisiform bone for able-bodied subjects and on the lateral epicondyle for the amputees by means of a second elastic band.

FIGURE 4

Figure 4. Example of trial involving an amputee. Two EMGs are attached to the stump for the dual-side control of the prosthesis with an elastic band. The vibromotor is fixed to the upper side of the stump with a second elastic band for the feedback restitution. The objects are placed within Hannes hand by the experimenter while the participant has closed eyes. The keyboard, placed in front of the participant, is used to indicate the grasped object stiffness by the user using the left hand.

First, the minimum and maximum amplitude for the vibromotor was determined using the method of limits (Prins, 2016), to find the minimum level of perception and avoid discomfort. To this aim, the vibration intensity was increased in small steps (4–5% in the normalized scale of PWM). When the subject warned, as soon as it was perceptible, the sensing of a small and then of a strong sensation, the respective PWM was saved. Subsequently, 30% of the PWM range was adopted for soft objects and 100% was adopted for rigid objects. The vibration intensity was then modulated between these two values to generate clearly perceivable and localized vibrations that were not intrusive to the subject but intuitive for the encoding of the object stiffness.

Six objects (Figure 1) were randomly presented three times to the user by the experimenter and three void closures were also inserted along the test, to have a total number of 21 trials. Before the test phase, a training phase was performed to let the user become familiar with the feedback. A total of six closures were performed, alternating between rigid and soft objects without headphones and with open eyes, so the user could learn to associate the proper feedback with the right stiffness. Furthermore, the involved upper limb side was covered with a black blanket to strengthen a possible embodiment effect.

In the first phase to evaluate the classifier performance and the feedback effectiveness, the able-bodied subjects underwent a single test with a single condition. They performed the test with the 2FB condition. The participant was asked to wear headphones with white noise and to close the eyes (avoiding the sight of the prosthesis and the grasped object). The subject was not required to reach out to the object. Instead, the experimenter proceeded to insert it directly into the prosthesis, asking the subject to perform a full closure, and then to identify the stiffness of the squeezed object. The answer was provided by the subject's left hand pressing the keyboard arrows, left for rigid objects and right for soft objects. No button needed to be pressed when the prosthesis performed a void closure. Finally, the subject could reopen the eyes to check if the answer was correct.

In the second phase, a comparison between the four different feedback conditions, discussed in the “Feedback conditions” section, was carried out by five transradial amputees. The order of these four sessions was randomly presented to the amputees. Each condition had the same test protocol already described in the first phase with able-bodied subjects, in which the experimenter places the object inside the prosthetic hand and the amputee performs a grasp with closed eyes and gives the answer using the keyboard. At the end of each session, the proprioceptive drift was detected with respect to the initial arm position (refer to the “Amputees” section) and an ad-hoc questionnaire was administered (refer to Supplementary material).

2.5. Data analysis

All the outcomes and the evaluation methods used in this study were tested for normality using the Shapiro–Wilk test. A repeated measure one-way ANOVA or Friedman test was conducted depending on the outcome of the normality test (for the analysis of dataset with missing data, Skilling's Mack was applied in substitution of the Friedman test), while the multiple comparison test with Bonferroni correction was used for a post-hoc analysis. Mathworks MATLAB 2020b was used for the statistical analysis. The average of the measures used (error and efficiency) was computed for each subject and condition and compared across conditions. The threshold for statistical significance was set at p < 0.05. The results in the text are reported as mean and standard deviation.

2.5.1. Able-bodied subjects

The primary outcome measure was the F1Score of the classifier on detecting the grasped object's stiffness expressed as a percentage, which takes into account the rate of false and true positives and false negatives (Powers, 2020). This result demonstrates that our approach to intrinsic sensor stiffness detection works properly. In addition, the F1Score was calculated on users' performance in recognizing objects' stiffness using the 2FB approach described in the “Feedback conditions” section. This latter was used to verify the usability and clarity of our feedback method.

2.5.2. Amputees

The second phase involving five amputees was carried out to compare the four feedback conditions. To validate and demonstrate that the 2FB condition was effective and the best feedback restoration for the recognition of objects' stiffness, our hypothesis involving the following four evaluation methods was used: (i) F1Score of performance; (ii) reaction time to recognize the stiffness of the objects; (iii) proprioceptive drift; and (iv) ad-hoc questionnaire.

The F1Score of amputees' performance was calculated in all feedback conditions. Furthermore, the response time of each trial was also recorded for the four conditions. Low response times were considered positive results. For each amputee, the mean response time of each feedback condition was calculated to allow comparison.

As a quantitative measure of the embodiment, the proprioceptive drift toward the artificial limb was detected (Tsakiris and Haggard, 2005). Before covering the involved upper limb side with a black blanket, the initial position of the hand was marked with white tape. Immediately, after the experiment, the blanket was removed and the amputees were asked to close their eyes, raise their stump, and replace it in the perceived initial position. The lateral distance between the initial position and the one estimated after the trials was measured by the experimenter with a ruler in centimeters, together with the direction of the deviation (Barresi et al., 2021). Deviations toward the prosthesis were considered an effect of the embodiment process.

At the end of each session, amputees also had to complete a Likert-type 5-point questionnaire, providing a subjective evaluation. The questionnaire (refer to Supplementary material) aimed to assess subjectively the intuitiveness and comfortability of the feedback (seven questions), its utility for ADLs (three questions), and the embodiment (four questions). The possible answers ranged between 1 (strongly disagree) and 5 (strongly agree). Since all amputees performed the test in all conditions, the experimental design is within-subject.

3. Results

3.1. Able-bodied subjects

The classifier's average accuracy in identifying the object stiffness was tested on a total of 378 grasps (21 grasps × 18 subjects). Its average F1Score resulted to be 94.93% ± 3.94. The able-bodied subjects instead, due to the 2FB condition, reached an average F1Score of 94.08% ± 4.0 for the object's stiffness discrimination task.

Figure 5A shows the F1Score obtained by able-bodied subjects during the 2FB condition compared to the F1Score of the classifier performance. Since these data did not present a normal distribution, the Friedman test was applied to demonstrate that no statistical difference was detected between the two populations (p = 0.1).

FIGURE 5

Figure 5. (A) F1Scores results of classifier with respect to able-bodied subjects. First, the F1Score of the classifier is calculated (blue box). Then, the F1Score of 18 able-bodied subjects is evaluated, based on the answer of classifier, in recognizing the objects' stiffness while receiving the 2FB condition (red box). A comparison between classifier's and subjects' F1Score was assessed and no significant difference was found between the two populations. (B) F1Scores results for amputees. F1Score obtained by amputees for each feedback condition. The Box chart shows the comparison between the distributions of the F1Scores obtained in each condition. The statistically significant difference is indicated by “^*” (p < 0.05). (C) Response time results for amputees. Response time obtained by amputees for each experimental condition. The Box chart shows the comparison between the distributions of the response time obtained in each condition. The statistically significant difference is indicated by “^*” (p < 0.05). (D) Proprioceptive drift of amputees for the different feedback conditions as a quantitative embodiment measure. The barplot shows the mean and standard deviation of the drift over amputees for each condition, the left direction is toward Hannes hand, and the right direction indicates a movement on the opposite side of Hannes hand. Deviations toward the prosthesis were considered an effect of the embodiment process.

3.2. Amputees

Figure 5B shows the boxplot of F1Score obtained by amputees for each of the four feedback conditions. It is possible to observe an ascending trend in the scores from the NoFB condition to the 2FB condition. In the NoFB condition, Amputee A3 data are missing because he found it impossible to accomplish the task without any feedback, stating that it was not possible to understand if the prosthesis was opened or closed. In the 1FB condition, Amputee A5 data are missing due to a recording problem.

For the NoFB condition, the F1Score among amputees is 31.41% ± 8.57, as indicated in Figure 5B with points, which is below the random chance probability of 33%. The statistically significant difference is indicated by “^*” (p < 0.05). Only Amputee A1 achieved a higher F1Score with respect to random chance (F1Score = 44.03%). The distributions resulted to be normal, so the statistical analysis applied was the ANOVA. As shown in Figure 5B, the 2FB condition presents a statistically significant difference with respect to the NoFB (p < 0.001) and AFB (p < 0.001) conditions. Furthermore, the 1FB condition is statistically different from the NoFB condition (p = 0.0031). The average F1Score calculated from the five amputees' responses during the 2FB experimental session is 86.41% ± 11.6.

Figure 5C shows the average response time for amputees in each feedback condition, in which the statistically significant difference is indicated by “^*” (p < 0.05). All amputees, except A1, achieved the lowest response time during the 2FB condition (2.82 s ± 1.2), which also produced the best results in terms of F1Scores. A statistical analysis was performed between the different conditions. The distribution resulted to be not normal and, given the presence of some missing data, the Skillings–Mack test was applied. As shown in Figure 5C, there is a statistically significant difference between AFB and 1FB conditions (p = 0.02) and between 1FB and 2FB conditions (p = 0.04).

The mean proprioceptive drift for each feedback condition was calculated, and it is reported in Figure 5D as bar plots with standard deviations. On average, the five amputees estimated the position of their right arm after the experiment as 1.8 cm ± 1.17 right (in the opposite side of the prosthesis) during the NoFB condition while 0.4 cm ± 0.58, 0.3 cm ± 0.4, and 0.7 cm ± 1.17 toward left and hence Hannes during AFB, 1FB, and 2FB conditions, respectively. The only significant difference was found between NoFB and 2FB conditions (p = 0.017) with the Nemenyi test for a post-hoc comparison.

According to comparisons performed through the Friedman test (because the scales are discrete and the actual data do not match the assumptions for other inferential techniques), three scales of the questionnaire showed significant effects of the feedback conditions. The subjective evaluations collected about the sessions show that in the 2FB condition:

- Scale 1 made it significantly easier to perceive the difference between soft and hard objects (p = 0.027);

- Scale 2 was significantly more intuitive for soft objects (p = 0.015);

- Scale 3 was significantly intuitive for rigid objects (p = 0.005).

According to Scale 3 scores, the post-hoc comparisons performed through the Nemenyi test show how the proposed feedback condition was significantly more intuitive for rigid objects than the NoFB condition (p = 0.025) and 1FB condition (p = 0.017). Furthermore, the Friedman test showed a significant effect of the 2FB condition (p = 0.019), especially considering the results of the Nemenyi test for the post-hoc comparison between 2FB and NoFB conditions (p = 0.017).

4. Discussion

The present study explored the possibility to recognize objects' stiffness with an under-sensorized prosthesis. The reference position, the encoder position, and the motor-side current available on Hannes were used to feed a pattern recognition algorithm, capable of generating different vibratory feedback to allow the subject to decode the relative objects' stiffness.

The F1Score of the classifier during the 2FB condition tested with able-bodied subjects (Figure 5A) was very high (94.93% ± 3.94), demonstrating that the only sensors available on Hannes (motor-side current and encoder position) provide sufficient information for an object stiffness discrimination task using an NLR algorithm. However, it is necessary to consider that the classifier, during this experiment, was only tested on six objects of different shapes but with almost the same dimensions (chosen to replicate the objects used during the Cybathlon race, Caserta et al., 2022). Furthermore, the F1Score obtained by able-bodied subjects (Figure 5A) on discriminating the object stiffness was also very good, proving the usability and efficacy of this feedback approach on a user case.

The positive results of the first phase allowed us to evaluate the object stiffness recognition approach on five transradial amputees. In this second phase, we tested four approaches of the feedback scheme (Figure 5B). In the NoFB condition, we expected a correct identification of the right stiffness around the random chance probability (33%). Actually, the F1Score for the NoFB condition was even lower than this percentage (F1Score = 31.41% ± 8.57), as amputees stated they were forced to guess since being deprived of any possible clue. The AFB condition presents a higher average F1Score (48.62% ± 12.56) with respect to the NoFB one, indicating that the motor noise provides less help in this kind of task. This is true for expert users like Amputee A1, who reached the highest score (62.78%), while it is less evident from others like Amputees A3 (34.85%) and A5 (36.11%), who scored almost as random chance. Differently, in the 1FB condition, almost everyone improved their performance (F1Score = 65.67% ± 10.34) with respect to NoFB and AFB conditions. In this condition, the users were clearly helped in recognizing the void closures, since those were the only ones without vibratory feedback. Moreover, most of the amputees declared that even if the intensity of the vibration was the same for rigid and soft objects, they were able to perceive a difference based on the vibration onset. Since soft objects are more compliant, the motor-side current takes more time to rise with respect to a rigid object. Hence, the vibration is slightly late. For this reason, the 1FB condition resulted to be statistically better than the NoFB one, unlike the AFB condition which has no significant difference with respect to the NoFB condition. Overall, the 2FB condition provided the best results (F1Score 86.41% ± 11.6), demonstrating to be significantly more helpful with respect to the other conditions and indicating that the difference in vibration, correspondent to the rigid and soft objects, was sufficiently distinguishable by the users, as we expected. This proves the advantages that this type of feedback can provide to prosthesis users as additional information to the incidental feedback (i.e., auditory feedback).

The reduction in the response time (Figure 5C) in the 2FB condition (2.82 s ± 1.2) is another proof of the efficiency of the implemented distinct vibratory feedback, meaning the amputees needed a short time to understand object's stiffness and enhancing the intuitiveness of the method. This parameter is significantly lower in 2FB (2.82 s ± 1.2) condition with respect to the NoFB (3.52 s ± 0.8), AFB (3.7 s ± 0.83), and 1FB (4.35 s ± 1.28) ones, suggesting that in these latter, the amputees needed to put quite an effort in discriminating between the objects instead.

The proprioceptive drift (Figure 5D) shows an effect of the feedback on the embodiment, especially according to the comparison between 2FB (0.7 cm ± 1.17 toward Hannes hand) and NoFB (1.8 cm ± 1.17 opposite to Hannes hand) conditions. Interestingly, the results could indicate that the presence of a source of feedback is important for summoning the embodiment process. Precisely, the highest impact on the proprioceptive drift was found with the 2FB condition, suggesting that this specific vibratory feedback was the most effective one during the embodiment process. However, a larger sample size is necessary to check potentially higher effects caused by the 2FB condition.

Three scales in the subjective questionnaire significantly highlight the benefits offered by the stimulations provided in the 2FB condition as intuitive feedback, especially for rigid objects. This indicates a possible effect of the feedback on the embodiment (refer to Figure 5D). However, a larger sample is necessary to deepen our understanding of the potential effects of the 2FB condition on embodiment measures in dedicated experiments. Overall, and regardless of the statistical significance, the results seem to point out the superiority of the 2FB condition over all aspects of user experience considered in this study. The qualitative observations provided by the amputees need a larger sample to extract potential user requirements.

5. Conclusion

This study presents the implementation of an online, i.e., real-time, dedicated stiffness detection strategy to provide grasp-oriented vibratory feedback using the Hannes prosthetic hand in a closed-loop scenario. As a further progression of our previous studies, in which we exploited a virtual simulation to find the intrinsic variables correlated to the grasped object's stiffness, this study builds upon those preliminary findings and presents a refined and improved methodology, its implementation, and its clinical validation. The main aim was to implement an online strategy exploiting such measurements (motor-side current, encoder position, and reference position) to detect the stiffness of real objects (without increasing the system complexity with ad-hoc force sensing) and to validate such strategy with a first preliminary study with end-users.

The classifier was tested by 18 able-bodied subjects on six objects and resulted to be sufficiently accurate in discriminating between void, soft, and rigid grasps. The stiffness information was conveyed to the users through a single vibromotor, whose intensity changed based on the grasp type, i.e., high intensity for rigid objects and low intensity for soft objects in our proposed feedback condition (2FB condition). This feedback modality was compared to three other control conditions (NoFB, AFB, and 1FB) in a user study involving five mono-lateral amputees. Results showed a statistically significant improvement in users' performances both in terms of F1Score and response time for the 2FB condition. Moreover, this condition was appreciated by the users, as demonstrated by the subjective questionnaires, which highlighted its intuitiveness, comfortability, and usefulness. This result was also confirmed by the analysis of the proprioceptive drift, which showed an improvement in the prosthesis embodiment. Hence, we can state that our proposed feedback modality was the best among those tested.

In the future, the classifier should be tested on a higher variety of objects with different dimensions and stiffness, especially to investigate the influence of the dimension on the algorithm's performance. Reach and grasp tasks, with active usage of prosthesis, will be implemented to provide a more realistic validation of the usability and effectiveness of our solution within ADL and real scenarios. A higher number of prosthesis users will be involved to better assess the effect of the feedback on the embodiment and its appreciation.

The present study can have a relevant impact on the application of intrinsic sensor detection of object stiffness, as it points out that this object recognition strategy and vibrotactile feedback restitution on upper limb prosthesis could be effectively used as an intuitive and effective closed-loop daily living solution. Such a solution could facilitate the identification of a precise and delicate grasp rather than a strong and powerful one during different object manipulations.

Data availability statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics statement

The studies involving human participants were reviewed and approved by AVEC Area Vasta Emilia Centro Protocol Code: CP-PPRAS1/1-03. The patients/participants provided their written informed consent to participate in this study.

Author contributions

GiuB, AM, NB, GiaB, and GC conceived the study. GiuB and GC administered the experiments. GiuB, AM, NB, and GC designed the figures. AM prepared the figures. All the authors contributed to the writing and reading and approved the final version of the manuscript.

Funding

This study was partially supported by the INAIL-IIT project PR19-PAS-P1-iHannes.

Acknowledgments

The authors thank Paolo Rossi for his help in assembling the prosthetic devices. The Open University Affiliated Research Centre at Istituto Italiano di Tecnologia (ARC@IIT) is part of the Open University, Milton Keynes MK7 6AA, United Kingdom.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fnins.2023.1078846/full#supplementary-material

Footnotes

1. ^Vybronics is a datasheet of a product available in the following link: https://www.vybronics.com/wp-content/uploads/datasheet-files/Vybronics-VC0625B001L-Jinlong-C0625B001L-datasheet.pdf.

References

Antfolk, C., Björkman, A., Frank, S.-O., Sebelius, F., Lundborg, G., and Rosen, B. (2012a). Sensory feedback from a prosthetic hand based on air-mediated pressure from the hand to the forearm skin. J. Rehabil. Med. 44, 702–707. doi: 10.2340/16501977-1001

PubMed Abstract | CrossRef Full Text | Google Scholar

Antfolk, C., D'alonzo, M., Controzzi, M., Lundborg, G., Rosen, B., Sebelius, F., et al. (2012b). Artificial redirection of sensation from prosthetic fingers to the phantom hand map on transradial amputees: vibrotactile versus mechanotactile sensory feedback. IEEE Trans. Neural Syst. Rehabil. Eng. 21, 112–120. doi: 10.1109/TNSRE.2012.2217989

PubMed Abstract | CrossRef Full Text | Google Scholar

Antfolk, C., D'alonzo, M., Rosén, B., Lundborg, G., Sebelius, F., and Cipriani, C. (2013). Sensory feedback in upper limb prosthetics. Expert Rev. Med. Dev. 10, 45–54. doi: 10.1586/erd.12.68

PubMed Abstract | CrossRef Full Text | Google Scholar

Balasubramanian, A. B., Magee, D. P., and Taylor, D. G. (2021). “Stiffness estimation in single degree of freedom mechanisms using regression,” in IECON 2021-47th Annual Conference of the IEEE Industrial Electronics Society (IEEE), 1–6.

Google Scholar

Barresi, G., Marinelli, A., Caserta, G., De Zambotti, M., Tessadori, J., Angioletti, L., et al. (2021). Exploring the embodiment of a virtual hand in a spatially augmented respiratory biofeedback setting. Front. Neurorobot. 15, 683653. doi: 10.3389/fnbot.2021.683653

PubMed Abstract | CrossRef Full Text | Google Scholar

Biddiss, E., Beaton, D., and Chau, T. (2007). Consumer design priorities for upper limb prosthetics. Disab. Rehabil. Assistive Technol. 2, 346–357. doi: 10.1080/17483100701714733

PubMed Abstract | CrossRef Full Text | Google Scholar

Bruni, G., and Bucchieri, A. (2021). Analysis and Optimization of Hannes Prosthetic Hand: A Multi-Body Approach. Available online at: politesi.polimi.it

Google Scholar

Bruni, G., Bucchieri, A., Tessari, F., Boccardo, N., Marinelli, A., De Momi, E., et al. (2022). “A multi-body model of an upper-limb prosthesis for grip force estimation and related object interaction application,” in 2022 9th IEEE RAS/EMBS International Conference for Biomedical Robotics and Biomechatronics (IEEE), 1–7. doi: 10.1109/BioRob52689.2022.9925478

PubMed Abstract | CrossRef Full Text | Google Scholar

Cannata, G., Maggiali, M., Metta, G., and Sandini, G. (2008). “An embedded artificial skin for humanoid robots,” in 2008 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems (IEEE), 434–438. doi: 10.1109/MFI.2008.4648033

CrossRef Full Text | Google Scholar

Caserta, G., Boccardo, N., Freddolini, M., Barresi, G., Marinelli, A., Canepa, M., et al. (2022). Benefits of the Cybathlon 2020 experience for a prosthetic hand user: a case study on the Hannes system. J. Neuroeng. Rehabil. 19, 1–15. doi: 10.1186/s12984-022-01046-y

PubMed Abstract | CrossRef Full Text | Google Scholar

Casini, S., Morvidoni, M., Bianchi, M., Catalano, M., Grioli, G., and Bicchi, A. (2015). “Design and realization of the cuff-clenching upper-limb force feedback wearable device for distributed mechano-tactile stimulation of normal and tangential skin forces,” in 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IEEE), 1186–1193. doi: 10.1109/IROS.2015.7353520

CrossRef Full Text | Google Scholar

Castellini, C. (2020). Upper limb active prosthetic systems—overview. Wear. Robot. 365–376. Available online at: https://www.sciencedirect.com/science/article/pii/B9780128146590000199. doi: 10.1016/B978-0-12-814659-0.00019-9

CrossRef Full Text | Google Scholar

Chai, G., Wang, H., Li, G., Sheng, X., and Zhu, X. (2022). Electrotactile feedback improves grip force control and enables object stiffness recognition while using a myoelectric hand. IEEE Trans. Neural Syst. Rehabil. Eng. 30, 1310–1320. doi: 10.1109/TNSRE.2022.3173329

PubMed Abstract | CrossRef Full Text | Google Scholar

Cipriani, C., D'alonzo, M., and Carrozza, M. C. (2011). A miniature vibrotactile sensory substitution device for multifingered hand prosthetics. IEEE Trans. Biomed. Eng. 59, 400–408. doi: 10.1109/TBME.2011.2173342

PubMed Abstract | CrossRef Full Text | Google Scholar

Clemente, F., D'alonzo, M., Controzzi, M., Edin, B. B., and Cipriani, C. (2015). Non-invasive, temporally discrete feedback of object contact and release improves grasp control of closed-loop myoelectric transradial prostheses. IEEE Trans. Neural Syst. Rehabil. Eng. 24, 1314–1322. doi: 10.1109/TNSRE.2015.2500586

PubMed Abstract | CrossRef Full Text | Google Scholar

Clemente, F., Valle, G., Controzzi, M., Strauss, I., IBERITE, F., Stieglitz, T., et al. (2019). Intraneural sensory feedback restores grip force control and motor coordination while using a prosthetic hand. J. Neural Eng. 16, 026034. doi: 10.1088/1741-2552/ab059b

PubMed Abstract | CrossRef Full Text | Google Scholar

Cuberovic, I., Gill, A., Resnik, L. J., Tyler, D. J., and Graczyk, E. L. (2019). Learning of artificial sensation through long-term home use of a sensory-enabled prosthesis. Front. Neurosci. 13, 853. doi: 10.3389/fnins.2019.00853

PubMed Abstract | CrossRef Full Text | Google Scholar

De Nunzio, A. M., Dosen, S., Lemling, S., Markovic, M., Schweisfurth, M. A., GE, N., et al. (2017). Tactile feedback is an effective instrument for the training of grasping with a prosthesis at low-and medium-force levels. Exp. Brain Res. 235, 2547–2559. doi: 10.1007/s00221-017-4991-7

PubMed Abstract | CrossRef Full Text | Google Scholar

Dellacasa Bellingegni, A., Gruppioni, E., Colazzo, G., Davalli, A., Sacchetti, R., Guglielmelli, E., et al. (2017). NLR, MLP, SVM, and LDA: a comparative analysis on EMG data from people with trans-radial amputation. J. Neuroeng. Rehabil. 14, 1–16. doi: 10.1186/s12984-017-0290-6

PubMed Abstract | CrossRef Full Text | Google Scholar

Deng, H., Xu, X., Zhuo, W., and Zhang, Y. (2020). Current-sensor-based contact stiffness detection for prosthetic hands. IEEE Access 8, 29456–29466. doi: 10.1109/ACCESS.2020.2972588

CrossRef Full Text | Google Scholar

Devaraja, R. R., Maskeliunas, R., and Damaševičius, R. (2020). Design and evaluation of anthropomorphic robotic hand for object grasping and shape recognition. Computers 10, 1. doi: 10.3390/computers10010001

CrossRef Full Text | Google Scholar

Di Domenico, D., Marinelli, A., Boccardo, N., Semprini, M., Lombardi, L., Canepa, M., et al. (2021). “Hannes prosthesis control based on regression machine learning algorithms,” in 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2021). doi: 10.1109/IROS51168.2021.9636391

PubMed Abstract | CrossRef Full Text | Google Scholar

Di Pino, G., Romano, D., Spaccasassi, C., Mioli, A., D'alonzo, M., Sacchetti, R., et al. (2020). Sensory-and action-oriented embodiment of neurally-interfaced robotic hand prostheses. Front. Neurosci. 14, 389. doi: 10.3389/fnins.2020.00389

PubMed Abstract | CrossRef Full Text | Google Scholar

Dosen, S., Markovic, M., Strbac, M., Belić, M., Kojić, V., Bijelić, G., et al. (2016). Multichannel electrotactile feedback with spatial and mixed coding for closed-loop control of grasping force in hand prostheses. IEEE Trans. Neural Syst. Rehabil. Eng. 25, 183–195. doi: 10.1109/TNSRE.2016.2550864

PubMed Abstract | CrossRef Full Text | Google Scholar

Garland, S. J., and Miles, T. S. (1997). Control of motor units in human flexor digitorum profundus under different proprioceptive conditions. J. Physiol. 502, 693–701. doi: 10.1111/j.1469-7793.1997.693bj.x

PubMed Abstract | CrossRef Full Text | Google Scholar

Hsiao, S. S., Fettiplace, M., and Darbandi, B. (2011). Sensory feedback for upper limb prostheses. Prog. Brain Res. 192, 69–81. doi: 10.1016/B978-0-444-53355-5.00005-1

PubMed Abstract | CrossRef Full Text | Google Scholar

Huang, J., and Rosendo, A. (2022). Variable stiffness object recognition with a CNN-bayes classifier on a soft gripper. Soft Robot. 9, 1220–1231. doi: 10.1089/soro.2021.0105

PubMed Abstract | CrossRef Full Text | Google Scholar

Jamali, N., and Sammut, C. (2011). Majority voting: material classification by tactile sensing using surface texture. IEEE Trans. Robot. 27, 508–521. doi: 10.1109/TRO.2011.2127110

CrossRef Full Text | Google Scholar

Kaczmarek, K. A., Webster, J. G., Bach-Y-Rita, P., and Tompkins, W. J. (1991). Electrotactile and vibrotactile displays for sensory substitution systems. IEEE Trans. Biomed. Eng. 38, 1–16. doi: 10.1109/10.68204

PubMed Abstract | CrossRef Full Text | Google Scholar

Kappassov, Z., Corrales, J.-A., and Perdereau, V. (2015). Tactile sensing in dexterous robot hands. Rob. Auton. Syst. 74, 195–220. doi: 10.1016/j.robot.2015.07.015

CrossRef Full Text | Google Scholar

Konstantinova, J., Cotugno, G., Stilli, A., Noh, Y., and Althoefer, K. (2017). “Object classification using hybrid fiber optical force/proximity sensor,” in 2017 IEEE SENSORS (IEEE), 1–3. doi: 10.1109/ICSENS.2017.8234057

CrossRef Full Text | Google Scholar

Laffranchi, M., Boccardo, N., Traverso, S., Lombardi, L., Canepa, M., Lince, A., et al. (2020). The Hannes hand prosthesis replicates the key biological properties of the human hand. Sci. Robot. 5, eabb0467. doi: 10.1126/scirobotics.abb0467

PubMed Abstract | CrossRef Full Text | Google Scholar

Liarokapis, M. V., Calli, B., Spiers, A. J., and Dollar, A. M. (2015). “Unplanned, model-free, single grasp object classification with underactuated hands and force sensors,” in 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IEEE), 5073–5080. doi: 10.1109/IROS.2015.7354091

CrossRef Full Text | Google Scholar

Longo, M. R., Schüür, F., Kammers, M. P., Tsakiris, M., and Haggard, P. (2008). What is embodiment? A psychometric approach. Cognition. 107, 978–998.

Google Scholar

Mamidanna, P., Dideriksen, J. L., and Dosen, S. (2021). The impact of objective functions on control policies in closed-loop control of grasping force with a myoelectric prosthesis. J. Neural Eng. 18, 056036. doi: 10.1088/1741-2552/ac23c1

PubMed Abstract | CrossRef Full Text | Google Scholar

Marinelli, A., Semprini, M., Canepa, M., Lombardi, L., Stedman, S., Bellingegni, A. D., et al. (2020). “Performance evaluation of pattern recognition algorithms for upper limb prosthetic applications,” in 2020 8th IEEE RAS/EMBS International Conference for Biomedical Robotics and Biomechatronics (IEEE), 471–476. doi: 10.1109/BioRob49111.2020.9224466

PubMed Abstract | CrossRef Full Text | Google Scholar

Markovic, M., Schweisfurth, M. A., Engels, L. F., Bentz, T., Wüstefeld, D., Farina, D., et al. (2018). The clinical relevance of advanced artificial feedback in the control of a multi-functional myoelectric prosthesis. J. Neuroeng. Rehabil. 15, 1–15. doi: 10.1186/s12984-018-0371-1

PubMed Abstract | CrossRef Full Text | Google Scholar

Masteller, A., Sankar, S., Kim, H. B., Ding, K., liu, X., and All, A. H. (2021). Recent developments in prosthesis sensors, texture recognition, and sensory stimulation for upper limb prostheses. Ann. Biomed. Eng. 49, 57–74. doi: 10.1007/s10439-020-02678-8

PubMed Abstract | CrossRef Full Text | Google Scholar

Mayer, R. M., Garcia-Rosas, R., Mohammadi, A., Tan, Y., Alici, G., Choong, P., et al. (2020). Tactile feedback in closed-loop control of myoelectric hand grasping: conveying information of multiple sensors simultaneously via a single feedback channel. Front. Neurosci. 14, 348. doi: 10.3389/fnins.2020.00348

PubMed Abstract | CrossRef Full Text | Google Scholar

Medynski, C., and Rattray, B. (2011). Bebionic Prosthetic Design. Myoelectric Symposium. Available online at: dukespace.lib.duke.edu

Google Scholar

Meek, S. G., Jacobsen, S. C., and Goulding, P. P. (1989). Extended physiologic taction: design and evaluation of a proportional force feedback system. J. Rehabil. Res. Dev. 26, 53–62.

PubMed Abstract | Google Scholar

Nemah, M. N., Aldulaymi, O. H., Low, C. Y., Zakaria, N. A. C., and Mohamaddan, S. (2020). “A hybrid haptic feedback stimulation device to recover the missing sensation of the upper limb amputees,” in IOP Conference Series: Materials Science and Engineering, Vol. 834 (IOP Publishing), 012013. doi: 10.1088/1757-899X/834/1/012013

CrossRef Full Text | Google Scholar

Oddo, C. M., Raspopovic, S., Artoni, F., Mazzoni, A., Spigler, G., Petrini, F., et al. (2016). Intraneural stimulation elicits discrimination of textural features by artificial fingertip in intact and amputee humans. Elife 5, e09148. doi: 10.7554/eLife.09148

PubMed Abstract | CrossRef Full Text | Google Scholar

Osborn, L., Kaliki, R. R., Soares, A. B., and Thakor, N. V. (2016). Neuromimetic event-based detection for closed-loop tactile feedback control of upper limb prostheses. IEEE Trans. Haptics 9, 196–206. doi: 10.1109/TOH.2016.2564965

PubMed Abstract | CrossRef Full Text | Google Scholar

Osborn, L. E., Dragomir, A., Betthauser, J. L., Hunt, C. L., Nguyen, H. H., Kaliki, R. R., et al. (2018). Prosthesis with neuromorphic multilayered e-dermis perceives touch and pain. Sci. Robot. 3, eaat3818. doi: 10.1126/scirobotics.aat3818

PubMed Abstract | CrossRef Full Text | Google Scholar

Patel, G. K., Dosen, S., Castellini, C., and Farina, D. (2016). Multichannel electrotactile feedback for simultaneous and proportional myoelectric control. J. Neural Eng. 13, 056015. doi: 10.1088/1741-2560/13/5/056015

PubMed Abstract | CrossRef Full Text | Google Scholar

Pena, A. E., Rincon-Gonzalez, L., Abbas, J. J., and Jung, R. (2019). Effects of vibrotactile feedback and grasp interface compliance on perception and control of a sensorized myoelectric hand. PLoS ONE 14, e0210956. doi: 10.1371/journal.pone.0210956

PubMed Abstract | CrossRef Full Text | Google Scholar

Powers, D. M. (2020). Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation. arXiv [Preprint]. arXiv: 2010.16061.

Google Scholar

Prins, N. (2016). Psychophysics: A Practical Introduction. Academic Press. Available online at: https://books.google.it/books?hl=it&lr=&id=3sHQBAAAQBAJ&oi=fnd&pg=PP1&dq=Psychophysics:+A+Practical+Introduction.+Academic+Press&ots=-M1GjWipBb&sig=KX6c5IzJkTnrBQYu9GdsA6KEDe0#v=onepage&q=Psychophysics%3A%20A%20Practical%20Introduction.%20Academic%20Press&f=false

Google Scholar

Pylatiuk, C., Schulz, S., and Döderlein, L. (2007). Results of an Internet survey of myoelectric prosthetic hand users. Prosthet. Orthot. Int. 31, 362–370. doi: 10.1080/03093640601061265

PubMed Abstract | CrossRef Full Text | Google Scholar

Raspopovic, S., Valle, G., and Petrini, F. M. (2021). Sensory feedback for limb prostheses in amputees. Nat. Mater. 20, 925–939. doi: 10.1038/s41563-021-00966-9

PubMed Abstract | CrossRef Full Text | Google Scholar

Sensinger, J. W., and Dosen, S. (2020). A review of sensory feedback in upper-limb prostheses from the perspective of human motor control. Front. Neurosci. 14, 345. doi: 10.3389/fnins.2020.00345

PubMed Abstract | CrossRef Full Text | Google Scholar

Shannon, G. (1976). A comparison of alternative means of providing sensory feedback on upper limb prostheses. Med. Biol. Eng. 14, 289–294. doi: 10.1007/BF02478123

PubMed Abstract | CrossRef Full Text | Google Scholar

Shehata, A. W., Engels, L. F., Controzzi, M., Cipriani, C., Scheme, E. J., and Sensinger, J. W. (2018). Improving internal model strength and performance of prosthetic hands using augmented feedback. J. Neuroeng. Rehabil. 15, 1–12. doi: 10.1186/s12984-018-0417-4

PubMed Abstract | CrossRef Full Text | Google Scholar

Shehata, A. W., Rehani, M., Jassat, Z. E., and Hebert, J. S. (2020). Mechanotactile sensory feedback improves embodiment of a prosthetic hand during active use. Front. Neurosci. 14, 263. doi: 10.3389/fnins.2020.00263

PubMed Abstract | CrossRef Full Text | Google Scholar

Stephens-Fripp, B., and alici, G., and, M. U. T. L.U., R. (2018). A review of non-invasive sensory feedback methods for transradial prosthetic hands. IEEE Access 6, 6878–6899. doi: 10.1109/ACCESS.2018.2791583

CrossRef Full Text | Google Scholar

Stiegelmar, C., Blustein, D., Sensinger, J., Hebert, J., and Shehata, A. (2020). “Towards quantifying the sense of agency and its contribution to embodiment of myoelectric prostheses,” in MEC20 Symposium.

Google Scholar

Štrbac, M., Belić, M., Isaković, M., Kojić, V., Bijelić, G., Popović, I., et al. (2016). Integrated and flexible multichannel interface for electrotactile stimulation. J. Neural Eng. 13, 046014. doi: 10.1088/1741-2560/13/4/046014

PubMed Abstract | CrossRef Full Text | Google Scholar

Svensson, P., Wijk, U., Björkman, A., and Antfolk, C. (2017). A review of invasive and non-invasive sensory feedback in upper limb prostheses. Expert Rev. Med. Dev. 14, 439–447. doi: 10.1080/17434440.2017.1332989

PubMed Abstract | CrossRef Full Text | Google Scholar

Tejeiro, C., Stepp, C. E., Malhotra, M., Rombokas, E., and Matsuoka, Y. (2012). “Comparison of remote pressure and vibrotactile feedback for prosthetic hand control,” in 2012 4th IEEE RAS & EMBS International Conference on Biomedical Robotics and Biomechatronics (IEEE), 521–525. doi: 10.1109/BioRob.2012.6290268

CrossRef Full Text | Google Scholar

Tsakiris, M., and Haggard, P. (2005). The rubber hand illusion revisited: visuotactile integration and self-attribution. J. Exp. Psychol. Hum. Percept. Perform. 31, 80. doi: 10.1037/0096-1523.31.1.80

PubMed Abstract | CrossRef Full Text | Google Scholar

Valle, G., Mazzoni, A., Iberite, F., D'anna, E., Strauss, I., Granata, G., et al. (2018). Biomimetic intraneural sensory feedback enhances sensation naturalness, tactile sensitivity, and manual dexterity in a bidirectional prosthesis. Neuron 100, 37–45. e7. doi: 10.1016/j.neuron.2018.08.033

PubMed Abstract | CrossRef Full Text | Google Scholar

Van Der Niet, O., and Van Der Sluis, C. K. (2013). Functionality of i-LIMB and i-LIMB Pulse hands: case report. J. Rehabil. Res. Dev. 50, 1123. doi: 10.1682/JRRD.2012.08.0140

PubMed Abstract | CrossRef Full Text | Google Scholar

Vargas, L., Huang, H., Zhu, Y., and Hu, X. (2021). object recognition via evoked sensory feedback during control of a prosthetic hand. IEEE Robot. Autom. Lett. 7, 207–214. doi: 10.1109/LRA.2021.3122897

PubMed Abstract | CrossRef Full Text | Google Scholar

Vargas, L., Shin, H., Huang, H. H., Zhu, Y., and Hu, X. (2019). Object stiffness recognition using haptic feedback delivered through transcutaneous proximal nerve stimulation. J. Neural Eng. 17, 016002. doi: 10.1088/1741-2552/ab4d99

PubMed Abstract | CrossRef Full Text | Google Scholar

Vernier (2021). Go Direct^® Hand Dynamometer. Available online at: https://www.vernier.com/product/go-direct-hand-dynamometer/

Wang, L., Li, Q., Lam, J., and Wang, Z. (2021). Tactual recognition of soft objects from deformation cues. IEEE Robot. Automat. Lett. 7, 96–103.

Google Scholar

Witteveen, H. J., Luft, F., Rietman, J. S., and Veltink, P. H. (2013). Stiffness feedback for myoelectric forearm prostheses using vibrotactile stimulation. IEEE Trans. Neural Syst. Rehabil. Eng. 22, 53–61. doi: 10.1109/TNSRE.2013.2267394

PubMed Abstract | CrossRef Full Text | Google Scholar

Yi, Z., and Zhang, Y. (2016). Bio-inspired tactile FA-I spiking generation under sinusoidal stimuli. J. Bionic Eng. 13, 612–621. doi: 10.1016/S1672-6529(16)60332-3

CrossRef Full Text | Google Scholar

Yildiz, K. A., Shin, A. Y., and Kaufman, K. R. (2020). Interfaces with the peripheral nervous system for the control of a neuroprosthetic limb: a review. J. Neuroeng. Rehabil. 17, 1–19. doi: 10.1186/s12984-020-00667-5

PubMed Abstract | CrossRef Full Text | Google Scholar

Zhao, H., O'brien, K., Li, S., and Shepherd, R. F. (2016). Optoelectronically innervated soft prosthetic hand via stretchable optical waveguides. Sci. Robot. 1, eaai7529. doi: 10.1126/scirobotics.aai7529

PubMed Abstract | CrossRef Full Text | Google Scholar