Continuous joint velocity estimation using CNN-based deep learning for multi-DoF prosthetic wrist for activities of daily living

Introduction: Myoelectric control of prostheses is a long-established technique, using surface electromyography (sEMG) to detect user intention and perform subsequent mechanical actions. Most machine learning models utilized in control systems are trained using isolated movements that do not reflect the natural movements occurring during daily activities. Moreover, movements are often affected by arm postures, the duration of activities, and personal habits. It is crucial to have a control system for multi-degree-of-freedom (DoF) prosthetic arms that is trained using sEMG data collected from activities of daily living (ADL) tasks.

Method: This work focuses on two major functional wrist movements: pronation-supination and dart-throwing movement (DTM), and introduces a new wrist control system that directly maps sEMG signals to the joint velocities of the multi-DoF wrist. Additionally, a specific training strategy (Quick training) is proposed that enables the controller to be applied to new subjects and handle situations where sensors may displace during daily living, muscles can become fatigued, or sensors can become contaminated (e.g., due to sweat). The prosthetic wrist controller is designed based on data from 24 participants and its performance is evaluated using the Root Mean Square Error (RMSE) and Pearson Correlation.

Result: The results are found to depend on the characteristics of the tasks. For example, tasks with dart-throwing motion show smaller RSME values (Hammer: 6.68 deg/s and Cup: 7.92 deg/s) compared to tasks with pronation-supination (Bulb: 43.98 deg/s and Screw: 53.64 deg/s). The proposed control technique utilizing Quick training demonstrates a decrease in the average root mean square error (RMSE) value by 35% and an increase in the average Pearson correlation value by 40% across all four ADL tasks.

1. Introduction

The human upper limb function is crucial to perform daily living activities. The loss of one or both arms causes severe disability that greatly affects a person's ability to perform essential daily activities (Kuiken et al., 2009). To date, there are nearly two million people living with limb loss in the United States, with ~41,000 individuals suffering from major upper limb amputations (Atzori and Müller, 2015). The number of individuals with amputation is increasing, resulting in a significant rise in health care costs. In 2009, hospital costs associated with amputation totaled more than 8.3 billion dollars (Semasinghe et al., 2019). As a result, the development of upper-limb prosthetic devices is considered to be crucial in helping amputees adapt to daily activities and reintegrate into society.

In order to restore the upper limb function of amputees, the development of myoelectric prosthesis started in the early 1940s (Kobrinskiy, 1960; Popov, 1965). A myoelectric prosthesis is electrically-powered, utilizing the electrical signals generated from some flexor and extensor muscles of the residual limb, which are surface electromyography (sEMG) signals that reflect the user's intention. To date, almost all commercial electric prostheses use a “direct myoelectric control” approach, where each direction of a motor in a prosthetic joint or the opening/closure of a specific grasp type in a robotic hand is controlled by a specific muscle. The myoelectric controller often uses the on-off method using a pre-defined threshold, but all commercial manufacturers also provide proportional control that can provide essentially continuous output to the active DoF of the prosthetic system (Fougner et al., 2012). To actuate multiple active degrees of freedom prosthetic devices, state machine technique has been suggested, which employs two sEMG signals to operate a single joint but also permits switching between other joints by co-activation of both muscles (Vujaklija et al., 2016). For example, SSSA-MyHAND (Controzzi et al., 2017) used state-machine, which switched to various grasps such as lateral, bi-directional, power, hook, pointing up and down by co-activation of both muscles. The state-machine complexity increases significantly when the number of prosthetic joints increases (Resnik et al., 2018) and it lacks the capability of simultaneous control of multiple DoFs which hinders the dexterity of the hand movement during daily living tasks.

Pattern recognition has been suggested and widely explored for the past few decades (Hargrove et al., 2007). Based on sEMG activation patterns, the amplitude of sEMG was used to decode the information and transfer the instructions to the motor, that could identify the user's intended hand and wrist motions (Scheme and Englehart, 2011; Parajuli et al., 2019). Statistical methods such as LDA (Linear Discriminant Analysis) and SVM (Support vector machine) were used to classify user intention with feature extraction, which were clinically tested on several amputee trials (Al-Timemy et al., 2013; Stango et al., 2014). For neural-based models, ANN (Artificial neural network) and MLP (Multilayer perceptron) were one of the initial deep learning algorithms researchers explored (Kawasaki et al., 2014). In comparison to traditional methods, these models were easily trainable and have the capability of modeling with non-linear data (Ahmad et al., 2011). Recently, Tam et al. (2021) designed a gesture recognition system using a CNN for myoelectric hand prosthesis control, in which the user could be able to monitor the gesture recognition output in real time. This pattern recognition-based classification method could only support discrete movement classification, which was rather non-intuitive compared to the natural way of controlling hands' pose (Yang et al., 2022).

To overcome the limits of classification approaches, several researchers have used deep learning techniques to control hand movements with regressions. Bao et al. (2021b) proposed the regression supervised domain adaptation (SDA) for estimating wrist angles using sEMG data. This study investigated the domain-shifting problem of the model when handling new subjects by categorizing the dataset of each subject as either the source or target domain and generating pairwise samples instead of single ones. A specific loss function, discrepancy loss, was also introduced for better description of the data. Stival et al. (2018) combined and IMU (Inertial Measurement Unit) features for the control of prosthetic devices. However, the study by Bao et al. was limited to simple wrist flexion/extension movements, while Stival et al.'s study was based on an online database and only presented two movements (flexion of three fingers or flexion of the wrist), which had the best performance.

In this study, to overcome the limitations of existing methods, a CNN-based wrist controller using a regression model is proposed and evaluated based on real-life ADL data. The proposed controller continuously estimates the wrist angle velocity from sEMG sensors placed on the participant's forearm, enabling continuous control of a multi-DoF prosthetic wrist in a more natural way. The model was trained using data collected while participants performing ADL tasks that focused on pronation-supination and dart-throwing-motion of the wrist. To increase the robustness of the model, ADL tasks were conducted to collect movement data at different heights. To use this model by a new participant within a short time, a method utilizing Pre-training and Quick training data is also suggested. This method can be used by participants within the existing data set to reduce the retraining time, as fast training is frequently required for amputee participants due to donning-doffing, muscle fatigue, or contamination (e.g., sweat; Ameri et al., 2020). An overview of the proposed method is shown in Figure 1. The results varied depending on the characteristics of the tasks. For example, tasks with dart-throwing motion showed smaller RSME values (Hammer: 6.68 deg/s and Cup: 7.92 deg/s) compared to tasks with pronation-supination (Bulb: 43.98 deg/s and Screw: 53.64 deg/s). The proposed control technique utilizing Quick training demonstrated a decrease in the average root mean square error (RMSE) value by 35% and an increase in the average Pearson correlation value by 40% across all four ADL tasks.

FIGURE 1

Figure 1. Overview of the proposed method. This multi-DoF controller will estimate the angular speed for pronation/supination (PS) and dart-throwing movements (DTM) with a training strategy.

2. Data collection

The study was approved by the Institutional Review Board of University at Buffalo. Participants provided written consent prior to the experiment. Only individuals with fully functioning biological arms and unrestricted arm movement were included in the study. And, for the current feasibility test, we recruited only right-handed participants to ensure homogeneous data. Participants included 24 healthy individuals. Their average age, height, and weight were 25.38 ± 3.00 years, 171.74 ± 8.40 cm, and 69.90 ± 14.67 kg, respectively. All participates were right-handed.

2.1. Sensor system

The Trigno^Ⓡ Wireless Biofeedback System (Delsys, MA) is a device designed to make and biofeedback signal detection reliable and easy. The system transmits signals from Trigno Avanti^TM sensors to a receiving base station using a time-synchronized wireless protocol that minimizes latency in data transmission across sensors. In this study, eight sensors were placed around the forearm near the elbow to capture muscle signals during experiments, as depicted in Figure 2. The sEMG sensor data was sampled at 2,000 Hz. Ten Vero motion capture cameras (Vicon, UK) were used to capture the movements of the participants. A total of nine markers were placed on the upper body and were divided into four different body segments (Fazil et al., 2022).

FIGURE 2

Figure 2. sEMG sensors were placed beneath the elbow, uniformly spaced from each other.

2.2. Experimental task

Four representative activities of daily life were specifically chosen for the experiment focusing on pronation-supination (PS) movement or dart-throwing movement (DTM). Specifically, PS and DTM were chosen for our prosthetic emulator in Poddar et al. (2021) and Poddar and Kang (2022). The Bulb twisting task and the Screwdriver task were designed for PS movements, and the Hammering task and the Cup drinking task were designed for DTM, as depicted in Figure 3. In each experiment, the participant started the tasks once all the sensors and markers had been placed. For each experiment, the participant was provided with different tools set up on a table in front of them. For the Bulb Twisting task, a custom-made board with a bulb socket fitted in parallel to the participant was placed at the edge of the table, and a bulb was placed within reach to its right. For the Screwdriver task/Hammering task, a steel panel with a nail in the center was fixed by a clamp and placed at the edge of the table, while the screwdriver/hammer was placed within reach to its right. The nail was placed ~2 cm above the table. For the Cup drinking task, a paper cup was placed in front of the center of the participant's body on the table within reach.

FIGURE 3

Figure 3. The activities of daily living (ADL) tasks are trained/tested through the Screw rotation, Bulb twisting, Hammering, and Cup drinking (from left to right).

For each trial, the procedure was as follows: First, the participant started from the T-pose position which stretches the arm shoulder height with palms facing down and feet on designated marks on the floor. The participant's toes were ~40 cm away from the edge of the table, with the distance adjusted based on the reach range of each individual. Recording began after a voice cue. After 2 s of recording, the participant was visually/orally prompted to begin. In the Screwdriver/Hammering/Cup Drinking task, the participant reached forward to pick up the screwdriver/hammer/cup and performed the screwing/hammering/drinking action 10 times. The procedure for the Bulb twisting task was slightly different. The twisting was performed 10 times in a clockwise direction and 10 times in a counterclockwise direction. After the participant completed the final movement, the tools were returned to the initial position on the table.

For each activity of daily living task, the trial was repeated three times by incrementally increasing the height of the table. The height of the table for the first trial started at 78.5 cm and increased by 5 cm each time, ending at 88.5 cm. A verbal cue was given before each trial to start. The participant was instructed to perform the movements at a consistent speed to maintain uniformity and integrity of data. A practice trial was conducted prior to the recording sessions to familiarize the participant with the steps involved in each trial. Participants performed four tasks sequentially in random order.

2.3. Data set generation

The data collection system consisted of a motion capture system, eight Delsys wearable sensors, a height-adjustable table, and four sets of tools for conducting experiments. In the experiment of this study, upper limb motion is measured using ten motion capture cameras and sEMG data were collected from eight wireless Trigno sensors. In the present ADL tasks, two angles were calculated: the pronation-supination (PS) angle and the dart-throwing motion (DTM) angle. These angles were calculated by constructing pairs of vectors within the markers in 3D space and computing the angle between them as in Fazil et al. (2022). As shown in Figure 4, the sEMG data were first filtered using a low-pass Butterworth first-order filter at 1 Hz. To generate feature data, the filtered data from eight sensors were cut into segments using a sliding window. The length of the window was set to 250 frames, which corresponds to 125 ms, with an overlap of 240 frames. The resulting feature data had a shape of (250, 8).

FIGURE 4

Figure 4. Raw data processing pipeline for features and labels.

3. Deep learning wrist controller

3.1. Inception-time model

Hierarchical Vote Collective of Transformation-based Ensembles (HIVE-COTE; Lines et al., 2016) recently emerged as one of the most popular methods for Time Series Classification tasks; Such method is a meta-ensemble built on several classifiers, including Time Series Forest, Shapelet Transform Classifier, and KNN-based classifiers. Although this algorithm has achieved outstanding performance on the benchmark datasets, it suffers from O(n²·T⁴) time complexity. Recently, Ismail Fawaz et al. (2020) introduced a deep Convolutional Neural Network (CNN), called Inception-Time, which not only outperforms the accuracy of HIVE-COTE but is also substantially faster while the complexity of Inception-Time increases almost linearly with an increase in the time series' length. The high accuracy and scalability of Inception-Time make it an ideal candidate for system development. In this study, we adapted the Inception-Time model to handle regression tasks.

• The fully-connected layer at the end of the network is substituted by a fully connected dense layer.

• The loss function is changed to a mean-square-error function.

• In each Inception module, kernel sizes and the numbers of filters are selected to fit the study.

3.2. Quick training strategy

As depicted in Figure 5, a unique training strategy is proposed. In this study, 24 participants performed three trials. The data was divided into four parts: pre-training group, model selection group, “Quick training” group, and test group. The pre-training group consisted of all trials of the first 15 individuals and the first trial of the 16th participant's three trials. The data in this group was used to initially train the modified Inception-Time model. The remaining two trials of the 16th participant were used as the validation set, and the model with the best performance, as measured by Pearson Correlation, was selected. The remaining data from the eight participants were considered new subjects, as they were unseen by the selected model. For each participant, the first trial was used for “Quick training,” and the model was evaluated on the rest two trials.

FIGURE 5

Figure 5. Proposed training strategy including Pre-training process and Quick training process.

For the implementation of the Inception-Time model and Quick training, Python 3.0 was used to design the wrist controller. The NumPy Python library is frequently used for scientific computing operations. The model was built on TensorFlow 2.5.0. Tools which was used for generating labels, normalization, and performance evaluations in Python. Most parts of our programs were computed on an NVIDIA GeForce RTX 3080 10G GPU.

In the present study, four different models were created for each task. The tasks could be divided into pronation-supination based Bulb and Screw tasks and dart-throwing-motion based Hammer and Cup tasks. The input of models was set in the form of (250, 8), which means the length of the sliding window is 250 frames (125 ms), and eight-channel signals were collected from eight sEMG sensors. Besides the Butterworth filter mentioned before, a scaler was used to normalize the data when generating features from the data. Same scaler was also applied to the data of the validation group, “Quick training” group, and test group.

3.3. Performance metrics

In this study, two common measures are used for numerical evaluations: Root Mean Square Error (RMSE) and Pearson correlation (PC), with following formulas. θ_i represents the true angle (PS angel or DTM angle) at time frame i, while $\stackrel{•}{{\theta }_i}$ represents the true joint velocity at time frame i. $\widehat{\stackrel{•}{\theta }}$ stands for the predicted joint velocity, and $\overline{(·)}$ as the mean value of (·). The number of total time frames is denoted as n.

$\text{RMSE}=\sqrt{\frac{{\displaystyle {\sum }_{i=1}^n\Vert \widehat{\stackrel{.}{{\theta }_i}}-\stackrel{.}{{\theta }_i}{\Vert }^2}}{n}}$

$\text{Pearson Correlation}=\frac{{\displaystyle {\sum }_{i=1}^n(\widehat{\stackrel{.}{{\theta }_i}}-\overline{\widehat{\dot{\theta }}})(\stackrel{.}{{\theta }_i}-\overline{\dot{\theta }})}}{\%\sqrt{{\displaystyle {\sum }_{i=1}^n{(\widehat{\stackrel{.}{{\theta }_i}}-\overline{\widehat{\dot{\theta }}})}^2}}\sqrt{{\displaystyle {\sum }_{i=1}^n{(\stackrel{.}{{\theta }_i}-\overline{\dot{\theta }})}^2}}}$

where Pearson Correlation is a measure of linear correlation between two sets of data. It is essentially a normalized measurement of the covariance, such that the result always has a value between −1 and 1.

4. Result

The comparisons between the measured and predicted data with Quick training of four different tasks are depicted in Figure 6. The data shows the data fit better for the positive values compared to the negative angular speed in general. The Screw and Bulb task follows the true value better. The Cup and Hammer task has smaller range of angular speed compared to Screw and Bulb tasks.

FIGURE 6

Figure 6. Comparison between measured (true) and predicted angular velocity of four different tasks with Quick training. (A) Screw, (B) cup, (C) bulb, (D) hammer.

The Bulb task used a model with a depth of 5, which means five Inception blocks are used. In each block, there are three convolutional layers with kernel sizes of 64, 16, and 4, respectively. The number of filters is 128. The numbers of epochs for the pre-training and “Quick training” part are both set as 30. When the model is pre-training, optimizer Adam (adaptive moment estimation) is used with a learning rate starting as 1e-3 and other parameters as default. The learning rate is decayed to half of its original value every 10 epochs. On data from the pre-training group, the selected model has RMSE of 19.723 deg/s, and Pearson Correlation of 0.669. On data from the validation group, the selected model has RMSE of 21.123 deg/s, and Pearson Correlation of 0.628. For the Screw task, an eight-depth model is utilized, which employs eight Inception blocks. Each block is composed of three convolutional layers with kernel sizes of 64, 16, and 4, respectively. The number of filters used is 128. Pre-training is done for 40 epochs, while “Quick training” is done for 30 epochs, using the Adam optimizer as before. The pre-training group achieved RMSE of 9.467 deg/s and Pearson Correlation of 0.849. On the validation group data, the selected model achieved RMSE of 25.265 deg/s and Pearson Correlation of 0.727.

For the Hammer task, a model with a depth of 4 is employed, utilizing four Inception blocks. Each block contains three convolutional layers with kernel sizes of 64, 16, and 4, respectively. The number of epochs for pre-training and “Quick training” is set to 30, and the Adam optimizer is used as before. The selected model achieved RMSE of 5.679 deg/s and Pearson Correlation of 0.817 on the pre-training group data. However, on the validation group data, the selected model achieved RMSE of 5.385 deg/s and Pearson Correlation of 0.166. As for the Cup task, a model with a depth of 3 is used, employing three Inception blocks. Each block consists of three convolutional layers with kernel sizes of 128, 32, and 8, respectively. The number of epochs for pre-training and “Quick training” is set to 30, and the Adam optimizer is used as before. Unfortunately, during the second trial of Subject 2, the sEMG sensors disconnected from the software, so the entire set of Subject 2 had to be dropped. The selected model achieved RMSE of 2.298 deg/s and Pearson Correlation of 0.961 on the pre-training group data. However, on the validation group data, the selected model achieved RMSE of 4.701 deg/s.

In Table 1, RMSE and Pearson Correlation for all new individuals are presented for each task. When tested on eight new participants, the average RMSE increased and the Pearson Correlation decreased, which means the performance drop of the model by unseen data. However, if the “Quick training” process was applied with a small amount of data, the results improved to similar level as those of the training group. For example, in the Bulb task, the selected model had RMSE of 19.723 deg/s, and a Pearson Correlation of 0.669 on the training group. If new participants were applied to the model, RMSE increased to 43.977 deg/s, and Pearson Correlation dropped to 0.526. After the “Quick training” process was utilized, the average RMSE decreased to 25.813 deg/s and the average Pearson Correlation rose to 0.702. Similar trends were also observed through other tasks. In general, the performance improved after the “Quick training” process, however, there were some exceptional cases, especially on the Cup task for participants 1 and 6. This discrepancy of performance between the participants will be further discussed in the following section.

TABLE 1

Table 1. RMSE^* and Pearson's correlation (PC) values between measured and predicted angular velocity of regression module before and after Quick training.

5. Discussion

The presented study shows a new framework to use real ADL task data to train a multi-DoF prosthe tic wrist using sEMG signals. The “Quick training” shows the utilization of a large data pool for creating a generic model but applies to a new user by using only a small amount of data for improving the model performance. Four tasks were tested to create the ML models by recruiting a total of 24 participants and tested on eight participants, which showed comparable performance with other models using a larger data set or training only simple motions.

Comparing between tasks, the Screw and Bulb tasks showed higher Pearson‘s correlation than the Cup and Hammer tasks. This is presumably because high variation was found in the movement in Cup and Hammer tasks for various reasons. First, participants chose different movement strategy to perform the Cup and Hammer tasks. Some participants preferred moving only their wrists when lifting the hammer, while other participants preferred only moving their wrists when dropping the hammer. Participants chose different movement coordination between the wrist, elbow, and shoulder to perform the Cup and Hammer task. Second, the end-effector (tool) movement to fulfill the task had different kinematic redundancy. The Bulb and Screw tasks required to rotate the screw or bulb exactly along the screw thread. However, the cup or hammer task was not performed with restricted end-effector as Bulb and Screw tasks. Lastly, participants had different fluency to perform the hammer task. Even though 5-min practice session was provided for each task, there were participants who never used a hammer before. This could be another factor to create deviation in the movement, resulting different sEMG patterns among participants. Even though higher Pearson's Correlation was observed in the Cup and Hammer tasks, it should be noted that the Cup and Hammer tasks had larger RMSE. This was due to the different range of motion of the pronation-supination and the dart-throwing-motion tasks. Pronation-supination tasks (Bulb and Screw) had a significantly larger range of motion than dart-throwing-motion tasks (Hammer and Cup), which naturally led to larger RMSE despite higher Pearson's correlation.

A few other researchers also studied various regression models for controlling prosthetic wrist. Stival et al. (2018) combined sEMG and IMU features to control prosthetic systems, and tested their model on a publicly available database as shown in Table 2. The Pearson's correlation of our study in Table 1 was changed to correlation coefficient similar to the study in Stival et al. (2018). Our controller performed comparably to theirs on the Bulb and Hammer tasks, and significantly better on the Screw task, exceeding their sEMG and IMU data fusion methods. It should be noted that Stival et al.'s method only showed results for two tasks that performed the best (three-finger flexion and wrist flexion), while our method focused on more complex ADL movements. Our model was trained with data from 16 participants, with each of them performing three trials, whereas Stival et al.'s method was trained on 35 participants, with six trials each.

TABLE 2

Table 2. Correlation coefficient for the considered movements Stival et al. (2018) method and ADL tasks in our method.

Bao et al. (2021a) also proposed a CNN-LSTM model for wrist kinematics estimation. The data was collected from six participants with 12 sensors. Bao et al.'s method trained a model on 3/4 of the data and tested it on the remaining 1/4. The trained model was evaluated by using R², and the detailed numeric results for the model are listed in Table 3. Although our method showed less R² values, it is important to note that our study performed more complicated ADL movements with only eight sensors. Additionally, our “Quick Training” process required much less training data, and the performance of LSTM models would decrease substantially over time due to its natural instincts that the model itself depends on its previous predictions, which means minor turbulence could cause large deviation. Moreover, the way they combined CNN and LSTM required separate tuning, which would affect the efficiency of the proposed method significantly.

TABLE 3

Table 3. Best R² of the hybrid CNN-LSTM model (Bao et al., 2021a) on single-Dof tasks and our method on ADL tasks.

Another study proposed the regression Supervised Domain Adaptation (SDA) for estimation of the wrist angle of flexion/extension through sEMG data (Bao et al., 2021b). Domain shifting problem was applied to the model to increase the performance on new subjects. Eight participants were recruited in total, trained on 7, and tested on the last one. The model was evaluated by Normalized root mean square error (NRMSE) and the RMSE of our result in Table 1 was changed to NRMSE for selected models. Detailed information is shown in the Table 4. The study showed that the model had NRMSE of 0.181 on designated simple flexion/extension movements. Our method had slightly worse NRMSE on Bulb (0.191) and Screw (0.185) tasks but achieved further improvements on overall more complicated movements with the introduction of the “Quick Train” process (0.133 on Bulb task, 0.120 on Screw task, 0.138 on Hammer task, and 0.191 on Cup task, respectively).

TABLE 4

Table 4. Average NMSE of regression SDA (Bao et al., 2021b) on the selected movements and our method on ADL tasks.

Our future studies will focus on addressing the current limitation of the study. First, we performed four different ADL tasks in the present work, thus, more diverse ADL tasks could be explored, and taking extra data into consideration would potentially improve the performance, such as including elbow angles as additional data when predicting wrist angles for tasks that showed different coordination between wrist and elbow joint movements among participants. Secondly, we used MSE as loss function in our model. The model could be presumably improved by modifying the loss function by introducing functions related to Pearson's correlation. Thirdly, the current model was designed for each task. Future models will classify motions into DTM or PS movements and then performing regression could allow our method to be used more generically, similar to previous work (Swami et al., 2021). Some other promising aspects of model generalization including associating not only types of ADL tasks, but also grasp types (Masiero et al., 2023), or arm positions (Gloumakov et al., 2022), could also be utilized to improve the performance. Lastly, complex ADLs that include three dimensional wrist motion will be trained in the model as well in the future. The current study uses ADLs that focus on majorly one dimensional rotation. In the future, the suggested controller will be implemented in the UBArm (Kim, 2022) featuring all three dimensional rotation of the prosthetic wrist with power grasping. With the UBArm, the tasks that were used to train in the presented paper and new tasks will be evaluated in real-time. To test the controller on amputee participants, the protocol will be further optimized and tested. For example, the number of sensors with less importance will be reduced by computing feature importance. Local surrogate models for identifying feature importance will be used such as SHAP (Lundberg and Lee, 2017) and LIME (Ribeiro et al., 2016a,b) to determine the important sensors. For the amputee participants, the sEMG signals can be inconsistent depending on the location of the amputation. We will test 20% or 30% MVC (Maximum Voluntary Contraction) test and check which position of the muscle shows the most consistent sEMG signals for the controller.

6. Conclusion

This study employed a data collection approach that included activities of daily living to ensure the datasets reflect realistic wrist motions used in day-to-day scenarios. A CNN model based on the Inception-Time architecture was implemented to train the models using a specific method that allows the designed wrist controller to perform on new subjects. The Quick training process improved the performance of the controller when facing new subjects, while significantly decreasing on-site training time. We believe our method will provide a practical solution for new participants using the model as well as handling situations where sensors may displace during daily living, muscles can become fatigued, or sensors can become contaminated (e.g., due to sweat).

Ethics statement

The studies involving humans were approved by Institutional Review Board of University at Buffalo. The studies were conducted in accordance with the local legislation and institutional requirements. The participants provided their written informed consent to participate in this study. Written informed consent was obtained from the individual(s) for the publication of any identifiable images or data included in this article.

Author contributions

JK: conceptualization and supervision. ZM: data collection and analysis. ZM and JK: visualization, writing the manuscript, and editing. All authors have read and agreed to the published version of the manuscript.

Funding

This research was supported by Disability and Rehabilitation Engineering (DARE) Program under National Science Foundation Award (CBET-2221979) and by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. RS-2023-00240085). JK also acknowledge the support by Institute of Information and Communications Technology Planning and Evaluation (IITP) grant funded by the Korea government [MSIT; No. 2019-0-01842, Artificial Intelligence Graduate School Program (GIST)].

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

Ahmad, S., Ishak, A., Ali, S. H., and Chappell, P. H. (2011). Review of electromyography control systems based on pattern recognition for prosthesis control application. Aust. J. Basic Appl. Sci. 5, 1512–1518. doi: 10.1007/978-3-642-21729-6_139

CrossRef Full Text | Google Scholar

Al-Timemy, A. H., Bugmann, G., Escudero, J., and Outram, N. (2013). Classification of finger movements for the dexterous hand prosthesis control with surface electromyography. IEEE J. Biomed. Health Inform. 17, 608–618. doi: 10.1109/JBHI.2013.2249590

PubMed Abstract | CrossRef Full Text | Google Scholar

Ameri, A., Akhaee, M. A., Scheme, E., and Englehart, K. (2020). A deep transfer learning approach to reducing the effect of electrode shift in EMG pattern recognition-based control. IEEE Trans. Neural Syst. Rehabil. Eng. 28, 370–379. doi: 10.1109/TNSRE.2019.2962189

PubMed Abstract | CrossRef Full Text | Google Scholar

Atzori, M., and Müller, H. (2015). Control capabilities of myoelectric robotic prostheses by hand amputees: a scientific research and market overview. Front. Syst. Neurosci. 9, 162. doi: 10.3389/fnsys.2015.00162

PubMed Abstract | CrossRef Full Text | Google Scholar

Bao, T., Zaidi, S. A. R., Xie, S., Yang, P., and Zhang, Z.-Q. (2021a). A CNN-LSTM hybrid model for wrist kinematics estimation using surface electromyography. IEEE Trans. Instrument. Measure. 70, 1–9. doi: 10.1109/TIM.2020.3036654

PubMed Abstract | CrossRef Full Text | Google Scholar

Bao, T., Zaidi, S. A. R., Xie, S., Yang, P., and Zhang, Z.-Q. (2021b). Inter-subject domain adaptation for CNN-based wrist kinematics estimation using SEMG. IEEE Trans. Neural Syst. Rehabil. Eng. 29, 1068–1078. doi: 10.1109/TNSRE.2021.3086401

PubMed Abstract | CrossRef Full Text | Google Scholar

Controzzi, M., Clemente, F., Barone, D., Ghionzoli, A., and Cipriani, C. (2017). The SSSA-myhand: a dexterous lightweight myoelectric hand prosthesis. IEEE Trans. Neural Syst. Rehabil. Eng. 25, 459–468. doi: 10.1109/TNSRE.2016.2578980

PubMed Abstract | CrossRef Full Text | Google Scholar

Fazil, M., Meng, Z., and Kang, J. (2022). “CNN-based controller for multi-DoF prosthetic wrist using SEMG data during activities of daily living,” in 2022 9th IEEE RAS/EMBS International Conference for Biomedical Robotics and Biomechatronics (BioRob) (New York, NY), 1–6. doi: 10.1109/BioRob52689.2022.9925506

CrossRef Full Text | Google Scholar

Fougner, A., Stavdahl, Ø., Kyberd, P. J., Losier, Y. G., and Parker, P. A. (2012). Control of upper limb prostheses: terminology and proportional myoelectric control-a review. IEEE Trans. Neural Syst. Rehabil. Eng. 20, 663–677. doi: 10.1109/TNSRE.2012.2196711

PubMed Abstract | CrossRef Full Text | Google Scholar

Gloumakov, Y., Bimbo, J., and Dollar, A. M. (2022). Trajectory control-an effective strategy for controlling multi-DoF upper limb prosthetic devices. IEEE Trans. Neural Syst. Rehabil. Eng. 30, 420–430. doi: 10.1109/TNSRE.2022.3151055

PubMed Abstract | CrossRef Full Text | Google Scholar

Hargrove, L. J., Englehart, K., and Hudgins, B. (2007). A comparison of surface and intramuscular myoelectric signal classification. IEEE Trans. Biomed. Eng. 54, 847–853. doi: 10.1109/TBME.2006.889192

PubMed Abstract | CrossRef Full Text | Google Scholar

Ismail Fawaz, H., Lucas, B., Forestier, G., Pelletier, C., Schmidt, D. F., Weber, J., et al. (2020). Inceptiontime: Finding alexnet for time series classification. Data Mining Knowl. Discov. 34, 1936–1962. doi: 10.1007/s10618-020-00710-y

CrossRef Full Text | Google Scholar

Kawasaki, H., Kayukawa, M., Sakaeda, H., and Mouri, T. (2014). “Learning system for myoelectric prosthetic hand control by forearm amputees,” in The 23rd IEEE International Symposium on Robot and Human Interactive Communication (New York, NY), 899–904. doi: 10.1109/ROMAN.2014.6926367

CrossRef Full Text | Google Scholar

Kim, M. (2022). A transradial prosthesis with a high-functional wrist for various daily living task (Master's thesis). University at Buffalo, Buffalo, NY, United States. doi: 10.1109/UR57808.2023.10202316

CrossRef Full Text | Google Scholar

Kobrinskiy, A. (1960). Bioelectrical control of prosthetic devices. Her. Acad. Sci. 30, 58–61.

Google Scholar

Kuiken, T. A., Li, G., Lock, B. A., Lipschutz, R. D., Miller, L. A., Stubblefield, K. A., et al. (2009). Targeted muscle reinnervation for real-time myoelectric control of multifunction artificial arms. JAMA 301, 619–628. doi: 10.1001/jama.2009.116

PubMed Abstract | CrossRef Full Text | Google Scholar

Lines, J., Taylor, S., and Bagnall, A. (2016). “Hive-cote: the hierarchical vote collective of transformation-based ensembles for time series classification,” in 2016 IEEE 16th International Conference on Data Mining (ICDM) (New York, NY), 1041–1046. doi: 10.1109/ICDM.2016.0133

CrossRef Full Text | Google Scholar

Lundberg, S., and Lee, S. (2017). “A unified approach to interpreting model predictions,” in Advances in Neural Information Processing Systems (Cambridge, MA), 4765–4774.

Google Scholar

Masiero, F., Fagioli, I., Truppa, L., Mannini, A., Cappello, L., and Controzzi, M. (2023). Looking for synergies in healthy upper limb motion: a focus on the wrist. IEEE Trans. Neural Syst. Rehabil. Eng. 31, 1248–1257. doi: 10.1109/TNSRE.2023.3243785

PubMed Abstract | CrossRef Full Text | Google Scholar

Parajuli, N., Sreenivasan, N., Bifulco, P., Cesarelli, M., Savino, S., Niola, V., et al. (2019). Real-time EMG based pattern recognition control for hand prostheses: a review on existing methods, challenges and future implementation. Sensors 19, 4596. doi: 10.3390/s19204596

PubMed Abstract | CrossRef Full Text | Google Scholar

Poddar, S., Cummiskey, D., and Kang, J. (2021). “A cable-actuated prosthetic emulator for transradial amputees,” in 2021 43rd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC) (New York, NY), 4529–4532. doi: 10.1109/EMBC46164.2021.9631108

PubMed Abstract | CrossRef Full Text | Google Scholar

Poddar, S., and Kang, J. (2022). “A lightweight transradial prosthetic emulator for optimizing prosthetic wrist design,” in 2022 9th IEEE RAS/EMBS International Conference for Biomedical Robotics and Biomechatronics (BioRob) (New York, NY), 1–6. doi: 10.1109/BioRob52689.2022.9925286

CrossRef Full Text | Google Scholar

Popov, B. (1965). The bio-electrically controlled prosthesis. J. Bone Joint Surg. 47, 421–424. doi: 10.1302/0301-620X.47B3.421

PubMed Abstract | CrossRef Full Text | Google Scholar

Resnik, L., Huang, H. H., Winslow, A., Crouch, D. L., Zhang, F., and Wolk, N. (2018). Evaluation of EMG pattern recognition for upper limb prosthesis control: a case study in comparison with direct myoelectric control. J. Neuroeng. Rehabil. 15, 1–13. doi: 10.1186/s12984-018-0361-3

PubMed Abstract | CrossRef Full Text | Google Scholar

Ribeiro, M., Singh, S., and Guestrin, C. (2016a). “Why should i trust you?” Explaining the predictions of any classifier,” in Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (New York, NY), 1135–1144. doi: 10.1145/2939672.2939778

CrossRef Full Text | Google Scholar

Ribeiro, M., Singh, S., and Guestrin, C. (2016b). “Model-agnostic interpretability of machine learning,” in ICML Workshop on Human Interpretability in Machine Learning (WHI) (New York, NY), 91–95.

Google Scholar

Scheme, E., and Englehart, K. (2011). Electromyogram pattern recognition for control of powered upper-limb prostheses: state of the art and challenges for clinical use. J. Rehabil. Res. Dev. 48, 643–659. doi: 10.1682/JRRD.2010.09.0177

PubMed Abstract | CrossRef Full Text | Google Scholar

Semasinghe, C. L., Madusanka, D. G. K., Ranaweera, R. K. P. S., and Gopura, R. A. R. C. (2019). Transradial prostheses: trends in development of hardware and control systems. Int. J. Med. Robot. Comput. Assist. Surg. 15, e1960. doi: 10.1002/rcs.1960

PubMed Abstract | CrossRef Full Text | Google Scholar

Stango, A., Negro, F., and Farina, D. (2014). Spatial correlation of high density EMG signals provides features robust to electrode number and shift in pattern recognition for myocontrol. IEEE Trans. Neural Syst. Rehabil. Eng. 23, 189–198. doi: 10.1109/TNSRE.2014.2366752

PubMed Abstract | CrossRef Full Text | Google Scholar

Stival, F., Michieletto, S., De Agnoi, A., and Pagello, E. (2018). “Toward a better robotic hand prosthesis control: using EMG and IMU features for a subject independent multi joint regression model,” in 2018 7th IEEE International Conference on Biomedical Robotics and Biomechatronics (Biorob) (New York, NY), 185–192. doi: 10.1109/BIOROB.2018.8487188

CrossRef Full Text | Google Scholar

Swami, C. P., Lenhard, N., and Kang, J. (2021). A novel framework for designing a multi-DoF prosthetic wrist control using machine learning. Sci. Rep. 11, 1–13. doi: 10.1038/s41598-021-94449-1

PubMed Abstract | CrossRef Full Text | Google Scholar

Tam, S., Boukadoum, M., Campeau-Lecours, A., and Gosselin, B. (2021). Intuitive real-time control strategy for high-density myoelectric hand prosthesis using deep and transfer learning. Sci. Rep. 11, 1–14. doi: 10.1038/s41598-021-90688-4

PubMed Abstract | CrossRef Full Text | Google Scholar

Vujaklija, I., Farina, D., and Aszmann, O. C. (2016). New developments in prosthetic arm systems. Orthop. Res. Rev. 8, 31–39. doi: 10.2147/ORR.S71468

PubMed Abstract | CrossRef Full Text | Google Scholar

Yang, Z., Clark, A. B., Chappell, D., and Rojas, N. (2022). “Instinctive real-time SEMG-based control of prosthetic hand with reduced data acquisition and embedded deep learning training,” in 2022 International Conference on Robotics and Automation (ICRA) (New York, NY), 5666–5672. doi: 10.1109/ICRA46639.2022.9811741

CrossRef Full Text | Google Scholar