Finger Gesture Recognition Using Sensing and Classification of Surface Electromyography Signals With High-Precision Wireless Surface Electromyography Sensors

Finger gesture recognition (FGR) plays a crucial role in achieving, for example, artificial limb control and human-computer interaction. Currently, the most common methods of FGR are visual-based, voice-based, and surface electromyography (EMG)-based ones. Among them, surface EMG-based FGR is very popular and successful because surface EMG is a cumulative bioelectric signal from the surface of the skin that can accurately and intuitively represent the force of the fingers. However, existing surface EMG-based methods still cannot fully satisfy the required recognition accuracy for artificial limb control as the lack of high-precision sensor and high-accurate recognition model. To address this issue, this study proposes a novel FGR model that consists of sensing and classification of surface EMG signals (SC-FGR). In the proposed SC-FGR model, wireless sensors with high-precision surface EMG are first developed for acquiring multichannel surface EMG signals from the forearm. Its resolution is 16 Bits, the sampling rate is 2 kHz, the common-mode rejection ratio (CMRR) is less than 70 dB, and the short-circuit noise (SCN) is less than 1.5 μV. In addition, a convolution neural network (CNN)-based classification algorithm is proposed to achieve FGR based on acquired surface EMG signals. The CNN is trained on a spectrum map transformed from the time-domain surface EMG by continuous wavelet transform (CWT). To evaluate the proposed SC-FGR model, we compared it with seven state-of-the-art models. The experimental results demonstrate that SC-FGR achieves 97.5% recognition accuracy on eight kinds of finger gestures with five subjects, which is much higher than that of comparable models.

Introduction

Comparing to traditional peripheral devices such as a mouse or a keyboard, finger gesture recognition (FGR) is much more convenient and natural for users to control an artificial limb and to interact with a computer (Rechy-Ramirez and Hu, 2015). As a result, FGR becomes more and more important during the past few years (Rechy-Ramirez and Hu, 2015). Currently, the most common methods of FGR are visual-based, voice-based, and surface electromyography (EMG)-based ones. Among them, surface EMG is the comprehensive photoelectrical signal of potential muscle action on the surface of the skin (Botros et al., 2020). It is a kind of non-stationary signal, and its strength is sensitively proportional to the degree of muscle activity, which makes it can accurately represent the gesture of fingers (Botros et al., 2020). Therefore, surface EMG-based is widely adopted to achieve FGR.

Surface EMG-based FGR has been researched for many years. Among existing approaches, machine learning-based approach is very popular and successful (Qi et al., 2020; Wong et al., 2021). For example, Phinyomark et al. (2011) applied the critical index analysis and fractal dimension to extract the characteristics of surface EMG signals, and seven kinds of gestures were recognized from eight-channel EMG signals. Ishii et al. (2012) divided hand motions into six movements and classified finger motions using two types of characteristics. Khushaba et al. (2016) proposed the mutual component analysis (MCA) by improving the principal component analysis (PCA) to deduct the noise and redundant features. The recognition accuracy reached 95% for 15 kinds of gestures by combining the feature selection and MCA from eight channels of the surface EMG signals. Ngeo et al. (2014) used the multi-output convolution Gaussian process to analyze the dependence of multi-joint gesture and to estimate the finger joint motion. Through the correlation between knuckles, the regression model was modified to improve the recognition rate of finger posture. AlOmari and Liu (2015) constructed a model by combining genetic algorithm, particle swarm optimization, and support vector machine (SVM). Arozi et al. (2020) identified the hand gesture through the single channel of the surface EMG signal with the time-domain feature extraction, PCA, feature dimensionality reduction, and neural network. The recognition accuracy is 86.7% for nine kinds of gestures.

Recently, since convolution neural network (CNN) was proposed by Krizhevsky et al. in 2012 (Atzori et al., 2016), it has achieved great success in many fields of image recognition, natural language processing, and language translation (Wu et al., 2019b; Yao et al., 2019). As it has much better performance of feature extraction and non-linear fitting than traditional machine learning models, many researchers employed CNN to classify hand gestures from surface EMG signals. For example, Atzori et al. (2016) and Geng et al. (2016) selected CNN to classify hand gestures using the original surface EMG signals as the input signal. A spectral map that was obtained by the short-time Fourier transform (STFT) from the original surface EMG signal was put into the convolution network (Du et al., 2017; Côté-Allard et al., 2019a). Zia Ur Rehman et al. (2018) constructed a simple network model consisting of one convolutional layer, one pooling layer, and two fully connected layers. Then, the original surface EMG was directly used as the input of the CNN. Wu et al. (2018) proposed a model based on long short-term memory (LSTM) and CNN, where LSTM reserves time information and CNN extract features. Its performance was better than the model proposed in the study by Santello et al. (2016). Chen L. et al. (2020) designed a compact CNN with a small number of parameters to improve the classification accuracy of EMG signals. However, all these approaches mainly focus on developing a CNN-based recognition model while ignoring to acquire the high-precision surface EMG. Hence, they still cannot fully satisfy the required recognition accuracy for real applications of artificial limb control and human-computer interaction.

To address this issue, this study proposes a novel FGR model that consists of two parts, namely, sensing and classification of surface EMG signal (SC-FGR). First, wireless sensors with high-precision surface EMG are developed for acquiring multichannel surface EMG signals from the forearm. Second, a CNN-based classification algorithm is proposed to classify the acquired surface EMG signals for FGR, where we named it CNN-FGR. A general chart of FGR with the proposed SC-FGR model is shown in Figure 1. The surface EMG signals of each channel are segmented by a moving window. A spectrum map is generated by continuous wavelet transform (CWT) from the segmented signals of each channel. Then, the spectrum maps of multiple channels are put into the CNN-FGR for classifying.

FIGURE 1

Figure 1. General chart of finger gesture recognition (FGR) using sensing and classification of surface electromyography (EMG) signals (SC-FGR).

The main research contents and contributions of this study are as follows:

(1) The wireless sensors are specially developed to acquire surface EMG from the forearm with high precision. Its resolution is 16 Bits, the sampling rate is 2 kHz, the common-mode rejection ratio (CMRR) is less than 70 dB, and the short-circuit noise (SCN) is less than 1.5 μV.

(2) A new CNN-FGR algorithm is proposed to accurately classify the surface EMG signals acquired by the developed wireless sensors. It consists of a 5-layer CNN that is trained on a spectrum map transformed from the time-domain signals of surface EMG by CWT.

(3) A novel SC-FGR model is proposed for highly accurate FGR. It comprises two parts of the developed wireless sensors and the proposed CNN-FGR algorithm.

(4) A surface EMG dataset is collected and shared online. It contains eight kinds of finger gestures with five subjects collected by the developed wireless sensors.

In the experiments, we evaluated the proposed SC-FGR model on the collected surface EMG dataset. The results demonstrate that the proposed SC-FGR model achieves 97.5% recognition accuracy, which is much higher than that of comparable models.

The rest of this article is organized as follows: A wireless surface EMG acquisition system is designed in section “A Wireless Surface EMG Acquisition System”; The data processing and CNN-FGR algorithm are described in detail in section “Data Processing and Network Architecture”; The proposed SC-FGR model is compared with several related models in Section “Experiment and Results”; and finally, section “Conclusion” concludes this study.

A Wireless Surface EMG Acquisition System

The EMG is a weak electrophysiological signal of a muscle fiber group. It can be detected by sensors placed on the surface of skin or needle sensors implanted in muscle tissue (De Luca et al., 2006). The EMG signal is closely related to neuron muscular activity information so that the surface EMG signals of the forearm can be used to analyze and recognize the finger gestures.

De Luca (1997) showed that the amplitude of the EMG signal was random and could be expressed by the arithmetic mean value of zero Gaussian distribution function. The surface EMG signal is a weak signal whose amplitude ranges from 0 to 10 mV (Peak-to-Peak) or 0 to 1.5 mV [root mean square (RMS)]. The frequency range of the available energy signal is limited from 0 to 1,000 Hz, and the dominant energy is distributed in the range from 50 to 150 Hz. In the same state of muscle motion, the amplitude-frequency characteristic curve of the EMG signal is similar, and the EMG signal has a certain regularity in the muscle motion state of different detection points. According to the characteristics of surface EMG, the frame of the acquisition module is designed as shown in Figure 2.

FIGURE 2

Figure 2. Multichannel surface EMG acquisition.

Inspired by the surface EMG sensor on the market, the surface EMG sensor consists of the surface EMG electrode and the signal conditioning circuit. This surface EMG sensor uses three parallel silver electrodes with a spacing of 10 mm, including two measuring electrodes and one reference electrode, which prevent saturation caused by the common-mode signals. The silver electrode is put close to the skin for complete polarization, forming a capacitor by surface skin and electrode. To improve the accuracy, the front analog amplifier circuit is designed as close as possible to the silver electrode. This measure is beneficial to weaken the disturbance of white noise for the acquisition of surface EMG signals. Then, the potential difference between the two measuring electrodes is detected by the differential amplifier circuit and converted into a digital signal for signal preprocessing. Finally, the digital signal is transformed into a computer by the Bluetooth data acquisition module.

The signal conditioning circuit plays a key role in amplifying the weak signal to improve the performance of the whole acquisition system. The expected conditioning circuit is with high input impedance, high gain, wide frequency band, low noise, and high CMRR. It should amplify surface EMG signals while suppressing other noise signals (Khokhar et al., 2010). The signal conditioning circuit uses instrument amplifier AD8220 with the JFET as the input of the preamplifier. The rail-to-rail amplifier OPA364 constitutes the band-pass amplifier. The instrument amplifier AD8220 plays the role of first-order high-pass filtering, while the amplifier OPA364 plays the role of second-order band-pass filtering. All in all, the function of the analog conditioning circuit is to amplify the original EMG signal 1,000 times and then signal processing by the second-order band-pass filtering with the range of 5–1,000 Hz. The schematic diagram of the signal conditioning circuit (Fu et al., 2013) is shown in Figure 3. The theoretical gain of the signal conditioning circuit is shown as follows:

$$\text{G}=\frac{V_o}{V_{i2}-V_{i1}}$$

$$=(\frac{49.4e^3}{R_G+R_{c_1}}+1)(\frac{R_3{R}_{c_3}}{R_{c_2}{R}_{c_3}+R_2{R}_{c_2}+R_2{R}_{c_3}+R_2{R}_3})(1)$$

FIGURE 3

Figure 3. Conditioning circuit for the analog signal.

where G represents amplifier gain; R_c1,R_c2, and R_c3 represent the impedance of the capacitance C₁,C₂, and C₃, respectively;V_i1,V_i2, and V_i3represent the input of the detection points; and V_o is the output of the signal conditioning circuit. The core design principles of the surface EMG acquisition system are anti-noise treatment, such as co-ground and anti-electromagnetic interference. This EMG acquisition system uses a Bluetooth module for physical isolation and anti-interference, avoiding 50-Hz interference from a wired connection with the computer. This data acquisition system contains a 16-bit AD conversion, an ARM processor, and a Bluetooth communication module, as shown in Figure 2. The output of the surface EMG sensor is connected to the input port of the AD converter by shielding line. It adopts the common ground technology between the analog signal and the digital signal. There is photoelectric isolation between the AD converter and the ARM microprocessor to reduce the crosstalk from digital signals to analog signals. On the one hand, the ARM controller stores the eigenvalues of the collected signal and stresses it in the local SD card. On the other hand, it transfers the collected signal to the HC-05 Bluetooth module through the USRT serial communication protocol. Bluetooth communication realizes the information interaction function between sensors and the computer. The Bluetooth communication module uses low-energy radio communication technology to realize data transmission, with the maximum rate of 1 Mb/s (Song et al., 2020) and the effective communication of 15 m. The multichannel wireless surface EMG module is designed with a highly extending function and could be extended to 4–8 channels. The surface EMG device is shown in Figure 4.

FIGURE 4

Figure 4. Multichannel wireless surface EMG acquisition device.

The parameter comparison between the high-precision wireless surface EMG acquisition system and the other surface EMG acquisition systems on the market is shown in Table 1.

TABLE 1

Table 1. Characterization of different surface electromyography (EMG) acquisition systems.

Data Processing and Network Architecture

Signal Feature Extraction of Surface EMG

Since the surface EMG signal is non-stationary, it is limited to analysis the signal with Fourier transform. The STFT, which divides the signal into smaller segments by sliding windows and calculates the Fourier transform of each segment separately, is an effective method to solve that problem. A frequency spectrogram can be obtained from the transformation of STFT. When the signal x(t) and window function w(t) are designed, the spectra can be calculated as follows:

$$spectrogram(x(t),w(t))={\left|STF{T}_x(t,f)\right|}^2(2)$$

$$STF{T}_x(t,f)={\int }_{-\mathrm{\infty }}^{+\mathrm{\infty }}\left[x(u)w(u-t)\right]e^{-j2\mathrm{\pi }fu}\mathit{d}u(3)$$

where frepresents the frequency. The wavelet transform (WT) is similar to STFT, while it overcomes the disadvantage that the window does not change with frequency in STFT. By adjusting the width of the window, the WT adapts to the frequency changes in the signal. When the frequency of the processed signal increases, the WT improves the resolution by narrowing the time window. Furthermore, WT is an ideal analysis tool, which can obtain the amplitude and frequency of mutations in the signal.

$$X(a,b)=\frac{1}{\sqrt{b}}{\int }_{-\mathrm{\infty }}^{\mathrm{\infty }}x\left(t\right)\mathrm{\varphi }\left(\frac{t-a}{b}\right)\mathit{d}t(4)$$

$${\int }_{-\mathrm{\infty }}^{+\mathrm{\infty }}\frac{{\left|\mathrm{\varphi }(\mathrm{\omega })\right|}^2}{\mathrm{\omega }}\mathit{d}\mathrm{\omega }<\mathrm{\infty }(5)$$

where the Fourier transform φ(w) must satisfy Equation 5. φ(t)is named as the parent wavelet function, which is a signal with limited duration, frequency change, and zero mean value. The scaling factor b and the translation factor a control the scaling and transform of the wavelet function, respectively. There are many kinds of parent wavelet functions for the transform, such as Mexican hat wavelet (MEXH), Gaussian wavelet (GAUS), complex Morlet wavelet (CMOR), Shannon wavelet (SHAN), frequency B-spline wavelet (FBSP), and complex Gaussian wavelet (CGAU). MEXH function is defined by Equation 6 as follows:

$$\mathrm{\psi }(t)=c(1-t^2)e^{-t^2/2}(6)$$

where $c=\frac{2}{\sqrt{3}}{\mathrm{\pi }}^{1/4}$. GAUS is the differential form derived from the Gaussian function. It is defined by Equation 7 as follows:

$$\mathrm{\psi }(t)=C_{p1}t{e}^{-t^2}(7)$$

where $C_{p1}=\sqrt[4]{2/\text{π}}$. CMOR is defined by Equation 8 in the time-domain and by Equation 9 in the frequency domain as follows:

$$\mathrm{\psi }(t)=\frac{1}{\sqrt{\mathrm{\pi }{f}_b}}\bullet e^{j2\mathrm{\pi }{f}_{c}t-(t^2/f_b)}(8)$$

$$\mathrm{\Psi }(f)=e^{{\mathrm{\pi }}^2{f}_b{(f-f_c)}^2}(9)$$

where f_c is the center frequency and f_b is the bandwidth. SHAN is defined by Equation 10 as follows:

$$\mathrm{\psi }(t)=\sqrt{f_b}\sin c(f_{b}x)e^{2i\mathrm{\pi }{f}_{c}x}(10)$$

where f_c is the center frequency and f_b is the bandwidth. FBSP is defined by Equation 11 as follows:

$$\mathrm{\psi }(t)=\sqrt{f_b}{\left[\sin (\frac{f_{b}t}{m})\right]}^m{e}^{2j\mathrm{\pi }{f}_{c}t}(11)$$

where m is an integer parameter, f_c is the center frequency, and f_b is the bandwidth. CGAU is defined by Equation 12 as follows:

$$\mathrm{\psi }(t)=C_p{e}^{-it}{e}^{-x^2}(12)$$

where C_p is constant.

After the CWT of the surface EMG signals, the corresponding spectrum map is similar to the image on the scale and also contains the frequency domain information of the timing sequence data. The six-channel surface EMG signals of the forearm were collected by the high-precision wireless surface EMG sensors, and the data of each channel were separated by applying a sliding window of 264 samples (132 ms). The parent wavelet of the CWT adopts the optimal wavelet function, calculating the CWTs with 64 scales to obtain the 64 × 264 matrix of spectral information. The matrix is set as input to the CNN-FGR algorithm. Thus, the input of the CNN-FGR algorithm has six channels, each consisting of a matrix with the size of 64 × 264. Figure 5 is the spectrum maps of the spectral information transformed from 264 EMG data with different kinds of parent wavelet functions, such as MEXH, GAUS, CMOR, SHAN, FBSP, and CGAU.

FIGURE 5

Figure 5. Spectrum maps transformed from surface EMG with different kinds of parent wavelet functions.

CNN-FGR Algorithm

Chen L. et al. (2020) used a compact CNN to improve the hand gesture recognition by surface EMG. Inspired from that model, the CNN-FGR algorithm consists of four convolutional layers and one mean pool layer as shown in Figure 6, and its design details are listed in Table 2.

FIGURE 6

Figure 6. The block diagram of the CNN-FGR algorithm.

TABLE 2

Table 2. Configuration of CNN of CNN-FGR algorithm.

The loss function is calculated as follows:

$$Loss=-\sum _{i=1}^n{y}_i\log \left(y_i^{\prime }\right)(13)$$

where y_i is the true value of the first class, n is the number of categories, y_i′ is the first-class prediction value of the output. Since one-hot coding was adopted, the true value of one class is 1, while the true value of the other classes is 0.

The three quantities where accuracy rate (AR) is used to evaluate the performance of the SC-FGR model, such as AR, the mean AR (MAR), and the SD of AR (SD-AR), are, respectively, computed as Equations 14–16. A test set composed of t number of instances x_i with ω known is used for the test stage.

$$AR=\frac{1}{t}\sum _{i=1}^t\mathrm{\Psi }(w,f(x_i)),\mathrm{\Psi }(w,f(x_i))$$

$$=\{\begin{array}{cc}1,ifw=f(x_i)\hfill & \\ 0,else\hfill & \end{array}(14)$$

$$MAR=\frac{1}{n}\sum _{k=1}^{n}A{R}_k(15)$$

$$SD-AR=\sqrt{\frac{1}{n}\sum _{k=1}^n{(A{R}_k-MAR)}^2}(16)$$

where f(x_i) represents the calculated label of x_i, and n is the repeated times of computing AR. MAR represents the classification ability of the algorithm, and SD-AR represents the robustness of the algorithm.

Advanced optimization methods were used for the backpropagation of the CNN-FGR algorithm with the ultimate goal to minimize the function loss. In the field of image recognition, the common size of the convolutional kernel is selected as 3 × 3, 5 × 5, or 7 × 7 (Krizhevsky et al., 2012; Simonyan and Zisserman, 2014). Therefore, the different sizes of the convolutional kernel in the CNN-FGR algorithm model are evaluated to get a better experimental result. Meanwhile, the various layer feature maps of the model are also set smaller to minimize the parameters of the model. The step length of the convolution is set to 2, for reducing the feature parameters by half. To further reduce the number of network parameters, the output of the model used the convolutional layer with adaptive mean sampling for classification, instead of the full connection layer.

Experiment and Results

Finger Gestures

Before the experiment, the collection points of the surface EMG from the forearm must be disinfected and cleaned to reduce skin contact interference. In the experiment, the subject sat on a chair with his left arm lying flat on the table and relaxed. In each group of experiments, as shown in Figure 7, each subject completed eight types of gestures, namely, Thumb Flection (TF), Thumb Extension (TE), Thumb Swing (TS), Index-finger Flection (IF), Index-finger Extension (IE), Index-finger Swing (IS), Middle-finger Flection (MF), and Middle-finger Extension (ME).

FIGURE 7

Figure 7. Eight kinds of finger gestures: (A) Thumb Flection (TF), (B) Thumb Extension (TE), (C) Thumb Swing (TS), (D) Index-finger Flection (IF), (E) Index-finger Extension (IE), (F) Index-finger Swing (IS), (G) Middle-finger Flection (MF), and (H) Middle-finger Extension (ME).

Number of Sensors and Layout of Detection Points

The surface EMG signal is closely related not only to the objective factors such as human physical state and movement state but also to the form and location of the detection electrode. The number of electrodes also has a great impact on the accuracy of surface EMG signal recognition. Extensive research and experiments showed that the acquisition of surface EMG signals with six channels can not only effectively identify single and multi-finger movement information but also avoid the waste of resources with over-channel detection. It was found that the electrodes were placed on the nerve-dominated region, and the EMG signals collected in the 10-tendon head or muscle edge area were usually weak. When sensors were placed vertically on the muscle fibers, the surface EMG signals were strongest. Since the front group muscles of the forearm cover the flexor, it mainly controls the bending movement of the elbow, wrist, and knuckles. The muscles of the back group cover the stretched muscles, which mainly control the stretching movement of each joint. In this experiment, six surface electrodes were placed on the corresponding muscle abs, and the electrodes were radially perpendicular to the muscle fibers. The sensors were fixed on the forearm with a bandage in moderate tension. Three sensors were placed on the corresponding muscle abs at the front of the forearm, mainly for detecting the bending movement of the finger, while the other sensors were placed at the back of the forearm for detecting the stretching movement of the fingers. The raw EMG signals detected by six sensors on the forearm are shown in Figure 8.

FIGURE 8

Figure 8. Six channels of the raw EMG signals from the high-precision sensors.

Classification Results

This experiment used the high-precision wireless surface EMG sensors and DELSYS data acquisition system to collect six channels of the surface EMG signal, with a frequency of 2 kHz. Before classification, the collected surface EMG signal must be pretreated and feature extracted. The original EMG signal is preprocessed with a 264-sample-point (132 ms) sliding window and a 100-sample-point incremental step. After the data segment processing, each experiment of each gesture obtains 12 samples, and 300 samples are collated after 25 repeating times. The total datasets of eight gestures of five subjects (i.e., S1, S2, S3, S4, and S5) are 12,000 samples. Each subject has 2,400 samples, where 1,920 samples are adopted as training set and 480 samples are adopted as testing set.

To evaluate the effects of CWT in transforming the surface EMG from time-domain to spectrum map, we, respectively, trained the CNN-FGR algorithm on the time-domain and the spectrum map of surface EMG. The comparison results on the testing set are shown in Table 3, where we observed that the CNN-FGR algorithm trained on the spectrum map of surface EMG achieves much higher accuracy than that trained on the time-domain of surface EMG. This observation demonstrates that transforming the surface EMG from time-domain to spectrum map by CWT is beneficial for the CNN-FGR algorithm to achieve a better performance of FGR.

TABLE 3

Table 3. The effects of continuous wavelet transform (CWT) on the accuracy of gesture classification of five subjects (S1, S2, S3, S4, and S5).

There are two factors affecting the identification accuracy in the SC-FGR algorithm model. One is the size of the convolutional kernel, and the other is the parent wavelet function. Using the same parent wavelet function “CGAU” for CWT transform, the different sizes of the convolutional kernel are compared to get a better recognition accuracy. The training accuracy curve, loss curve during training, and testing accuracy curve are used to analyze the results of FGR. The accuracy of the CNN-FGR algorithm with the convolutional kernel size of 3 × 3 is shown in Table 4.

TABLE 4

Table 4. The accuracy of the CNN-FGR algorithm with the convolutional kernel size of 3 × 3 on five subjects.

From Table 4, we found that the training accuracy keeps increasing and loss keeps decreasing with more epochs until reaching convergence. Similarly, testing accuracy also keeps increasing with more epochs until reaching convergence. These findings verify that the CNN-FGR algorithm can be well applied to classify these samples for FGR. In the experiment, we compared the accuracy of the CNN-FGR algorithm with the kernel size of 3 × 3, 5 × 5, 7 × 7, and 9 × 9 on collected datasets.

From Table 5, it can be observed that the classification ability of the algorithm is improved, but the robustness of the algorithm becomes worse, while the size of the convolution kernel increases. The size of 5 × 5 is a better selection as the convolution kernel, because not only the accuracy is high, but also the robustness performed well.

TABLE 5

Table 5. The comparison of accuracy with different convolution kernel sizes.

To choose the suitable parent wavelet function for CNN-FGR, the experiments are carried out on different parent wavelet functions, such as MEXH, SHAN, GAUS, FBSP, CGAU, and CMOR. For dataset S3, the comparison results of accuracy of various parent wavelet functions with the same convolutional kernel size of 5 × 5 are shown in Table 6.

TABLE 6

Table 6. The comparison results of accuracy of various parent wavelet functions on the dataset S3.

On all collected datasets, the comparison results of accuracy of various parent wavelet functions with the same convolutional kernel size of 5 × 5 are shown in Table 7.

TABLE 7

Table 7. The comparison results of accuracy of various parent wavelet functions on the collected datasets.

From Table 7, it is easy to get the results that the accuracy of GAUS is higher than that of other wavelet functions, but the robustness is worse. Considering the classification ability and the robustness, the algorithm with the parent wavelet MEXH performs better.

Finally, to evaluate the proposed SC-FGR model, we compared it with several related models. Especially, enhanced time-domain (EnhancedTD) (Khushaba et al., 2016; Fournelle and Bost, 2019), time-domain cycle (TDC) (Tang et al., 2010), autoregression (AR) (Soares et al., 2003), sample entropy (SampEn) (Delgado-Bonal and Marshak, 2019), and wavelet package coefficient (WPC) (Zhao et al., 2006) are selected as feature extractors. The classical classifiers [e.g., probabilistic neural network (PNN) (Zeinali and Story, 2017), linear discriminant analysis (LDA) (Zhang et al., 2012), and SVM (Varatharajan et al., 2018)], CNN (Chen H.F. et al., 2020), and CWT-EMGNet (Chen L. et al., 2020) are adopted as classifiers. The comparison results are recorded in Table 8, where we clearly observed that the SC-FGR model achieves 97.5% accuracy, which is the best among all the models. Hence, we concluded that the proposed SC-FGR model is powerful for FGR.

TABLE 8

Table 8. The comparison results of accuracy of various models on the collected datasets.

Conclusion

This study proposes a novel SC-FGR model that consists of two parts, namely, sensing and classification of the surface EMG signal. First, wireless sensors are developed for acquiring multichannel surface EMG signals from the forearm according to the characteristics of the surface EMG signal. These sensors can provide a high-precision signal source of surface EMG for FGR. In addition, a CNN-based classification algorithm, i.e., CNN-FGR, is proposed for FGR based on the acquired surface EMG by the developed wireless sensors. The CNN-FGR is trained on a spectrum map transformed from the time-domain of surface EMG by CWT. The experimental results demonstrate that the proposed SC-FGR model achieves 97.5% recognition accuracy on eight kinds of finger gestures with five subjects, which is much higher than that of comparable models. In the future, we plan to adopt the techniques of latent factor analysis (Wu et al., 2019a, 2020, 2021a,b), cognitive computing (Wu et al., 2021c), and attention mechanism (Zheng and Chen, 2021) to simultaneously recognize the gesture and strength of the fingers based on the surface EMG of the forearm.

Data Availability Statement

The datasets presented in this study can be found in online repositories. The name of the repository and accession number can be found below: Baidu Netdisk, https://pan.baidu.com/s/1wXT_i2kPMRALvfI17bP1YA (access code: f6wu).

Author Contributions

JF contributed to writing—original draft, conceptualization, and methodology. SC contributed to the experiment design. LC contributed to the data curation. LY contributed to writing—review and editing. All authors contributed to the article and approved the submitted version.

Funding

This work was supported by the Surface Project of Chongqing Natural Science Fund, Grant No. cstc2021jcyj-msxmX0144.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

ADInstituments (2020). Delsys for research available from ADInstruments. Available Online at: https://www.adinstruments.com/partners/delsys (accessed September 04, 2021).

Google Scholar

AlOmari, F., and Liu, G. (2015). Novel hybrid soft computing pattern recognition system SVM–GAPSO for classification of eight different hand motions. Optik Int. J. Light Electron Opt. 126, 4757–4762. doi: 10.1016/j.ijleo.2015.08.170

CrossRef Full Text | Google Scholar

Arozi, M., Caesarendra, W., Ariyanto, M., Munadi, M., Setiawan, J. D., and Glowacz, A. (2020). Pattern Recognition of Single-Channel sEMG Signal Using PCA and ANN Method to Classify Nine Hand Movements. Symmetry 12, 541–549. doi: 10.3390/sym12040541

CrossRef Full Text | Google Scholar

Atzori, M., Cognolato, M., and Müller, H. (2016). Deep learning with convolutional neural networks applied to electromyography data: a resource for the classification of movements for prosthetic hands. Front. Neurorobot. 10:9. doi: 10.3389/fnbot.2016.00009

PubMed Abstract | CrossRef Full Text | Google Scholar

Botros, F., Phinyomark, A., and Scheme, E. (2020). EMG-Based Gesture Recognition: is It Time to Change Focus from the Forearm to the Wrist? IEEE Trans. Industr. Inform. 99:1. doi: 10.1109/TMC.2020.3045635

CrossRef Full Text | Google Scholar

Chen, H. F., Zhang, Y., Li, G. F., Fang, Y., and Liu, H. (2020). Surface electromyography feature extraction via convolutional neural network. Int. J. Mach. Learn. Cybern. 11, 185–196. doi: 10.1007/s13042-019-00966-x

CrossRef Full Text | Google Scholar

Chen, L., Fu, J., Wu, Y., Li, H., and Zheng, B. (2020). Hand Gesture Recognition Using Compact CNN Via Surface Electromyography Signals. Sensors 20, 672–680. doi: 10.3390/s20030672

PubMed Abstract | CrossRef Full Text | Google Scholar

Côté-Allard, U., Fall, C. L., Drouin, A., Campeau-Lecours, A., Gosselin, C., Glette, K., et al. (2019a). Deep learning for electromyographic hand gesture signal classification using transfer learning. IEEE Trans. Neural Syst. Rehabil. Eng. 27, 760–771. doi: 10.1109/TNSRE.2019.2896269

PubMed Abstract | CrossRef Full Text | Google Scholar

Côté-Allard, U., Gagnon-Turcotte, G., Laviolette, F., and Gosselin, B. (2019b). A low-cost, wireless, 3-d-printed custom armband for semg hand gesture recognition. Sensors 19, 2811–2820. doi: 10.3390/s19122811

PubMed Abstract | CrossRef Full Text | Google Scholar

De Luca, C. J. (1997). The use of surface electromyography in biomechanics. J. Appl. Biomech. 13, 135–163. doi: 10.1123/jab.13.2.135

CrossRef Full Text | Google Scholar

De Luca, C. J., Adam, A., Wotiz, R., Gilmore, L. D., and Nawab, S. H. (2006). Decomposition of surface EMG signals. J. Neurophysiol. 9, 1646–1657. doi: 10.1152/jn.00009.2006

PubMed Abstract | CrossRef Full Text | Google Scholar

Delgado-Bonal, A., and Marshak, A. (2019). Approximate entropy and sample entropy: a comprehensive tutorial. Entropy 21:541. doi: 10.3390/e21060541

PubMed Abstract | CrossRef Full Text | Google Scholar

Du, Y., Jin, W., Wei, W., Hu, Y., and Geng, W. (2017). Surface EMG-based inter-session gesture recognition enhanced by deep domain adaptation. Sensors 17, 458–466. doi: 10.3390/s17030458

PubMed Abstract | CrossRef Full Text | Google Scholar

Fournelle, M., and Bost, W. (2019). Wave front analysis for enhanced time-domain beamforming of point-like targets in optoacoustic imaging using a linear array. Photoacoustics 14, 67–76. doi: 10.1016/j.pacs.2019.04.002

PubMed Abstract | CrossRef Full Text | Google Scholar

Fu, J., Jian, C., and Shi, Y. (2013). “Design of a low-cost wireless surface EMG acquisition system,” in The 6th International IEEE EMBS Conference on Neural Engineering? California, 699–702.

Google Scholar

Fu, J., Xiong, L., Song, X., Yan, Z., and Xie, Y. (2017). “Identification of finger movements from forearm surface EMG using an augmented probabilistic neural network,” in 2017 IEEE/SICE International Symposium on System Integration (SII), (Taipei, Taiwan: IEEE). doi: 10.1109/SII.2017.8279278

CrossRef Full Text | Google Scholar

Geng, W., Du, Y., Jin, W., Wei, W., Hu, Y., and Li, J. (2016). Gesture recognition by instantaneous surface EMG images. Sci. Rep. 6:36571. doi: 10.1038/srep36571

PubMed Abstract | CrossRef Full Text | Google Scholar

Ishii, C., Saitou, S., Sasaki, A., and Hashimoto, H. (2012). “Distinction of finger operation for myoelectric prosthetic hand on the basis of surface EMG,” in 16th Csi International Symposium on Artificial Intelligence & Signal Processing, (Shiraz, Iran: IEEE). doi: 10.1109/AISP.2012.6313786

CrossRef Full Text | Google Scholar

Khokhar, Z. O., Xiao, Z. G., and Menon, C. (2010). Surface EMG pattern recognition for real-time control of a wrist exoskeleton. Biomed. Eng. Online 9, 41–47. doi: 10.1186/1475-925X-9-41

PubMed Abstract | CrossRef Full Text | Google Scholar

Khushaba, R. N., Al-Timemy, A., Kodagoda, S., and Nazarpour, K. (2016). Combined influence of forearm orientation and muscular contraction on EMG pattern recognition. Expert Syst. Appl. 61, 154–161. doi: 10.1016/j.eswa.2016.05.031

CrossRef Full Text | Google Scholar

Krizhevsky, A., Sutskever, I., and Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. Adv. Neural Inform. Process. Syst. 03, 1097–1105.

Google Scholar

Ngeo, J., Tamei, T., and Shibata, T. (2014). Estimation of continuous multi-DOF finger joint kinematics from surface EMG using a multi-output Gaussian process. Eng. Med. Biol. Soc. 08, 3537–3540. doi: 10.1109/EMBC.2014.6944386

PubMed Abstract | CrossRef Full Text | Google Scholar

Phinyomark, A., Phothisonothai, M., Phukpattaranont, P., and Limsakul, C. (2011). Critical Exponent Analysis Applied to Surface EMG Signals for Gesture Recognition. Metrol. Meas. Syst. 18, 645–658. doi: 10.2478/v10178-011-0061-9

CrossRef Full Text | Google Scholar

Qi, J., Jiang, G., Li, G., Sun, Y., and Tao, B. (2020). Surface EMG hand gesture recognition system based on PCA and GRNN. Neural Comput. Appl. 32, 6343–6351. doi: 10.1007/s00521-019-04142-8

CrossRef Full Text | Google Scholar

Rechy-Ramirez, E. J., and Hu, H. (2015). Bio-signal based control in assistive robots: a survey. Digit. Commun. Netw. 1, 85–101. doi: 10.1016/j.dcan.2015.02.004

CrossRef Full Text | Google Scholar

Santello, M., Bianchi, M., Gabiccini, M., Ricciardi, E., Salvietti, G., Prattichizzo, D., et al. (2016). Hand synergies: integration of robotics and neuroscience for understanding the control of biological and artificial hands. Phys. Life Rev. 06, 1–23. doi: 10.1016/j.plrev.2016.02.001

PubMed Abstract | CrossRef Full Text | Google Scholar

Simonyan, K., and Zisserman, A. (2014). Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv. Available online at: https://arxiv.org/abs/1409.1556 (accessed September 04, 2021).

Google Scholar

Soares, A., Andrade, A., Lamounier, E., and Carrijo, R. (2003). The development of a virtual myoelectric prosthesis controlled by an EMG pattern recognition system based on neural networks. J. Intell. Inf. Syst. 21, 127–141. doi: 10.1023/A:1024758415877

CrossRef Full Text | Google Scholar

Song, E., Kim, S., Han, J., and Kwon, K. (2020). A 2.4-GHz Quadrature Local Oscillator Buffer Insensitive to Frequency-Dependent Loads for Bluetooth Low Energy Applications. IEEE Microw. Wirel. Compon. Lett. 30, 961–964. doi: 10.1109/LMWC.2020.3016733

CrossRef Full Text | Google Scholar

Tang, M., Li, K., and Ma, X. T. (2010). Research on pulse wave signal and time-domain feature extraction algorithm. Comput. Modernization 176, 16–22.

Google Scholar

Varatharajan, R., Manogaran, G., and Priyan, M. K. (2018). A big data classification approach using LDA with an enhanced SVM method for ECG signals in cloud computing. Multimed. Tools Appl. 77, 10195–10215. doi: 10.1007/s11042-017-5318-1

CrossRef Full Text | Google Scholar

Wong, W. K., Juwono, F. H., and Khoo, B. T. T. (2021). Multi-features capacitive hand gesture recognition sensor: a machine learning approach. IEEE Sens. J. 21, 8441–8450. doi: 10.1109/JSEN.2021.3049273

CrossRef Full Text | Google Scholar

Wu, D., Luo, X., Shang, M. S., He, Y., Wang, G., Zhou, M., et al. (2019b). A Deep Latent Factor Model for High-Dimensional and Sparse Matrices in Recommender Systems. IEEE Trans. Syst. Man Cybern. Syst. 99, 1–12.

Google Scholar

Wu, D., He, Q., Luo, X., Shang, M., He, Y., Wang, G., et al. (2019a). A posteriorneighborhood-regularized latent factor model for highly accurate web service QoS prediction. IEEE Trans. Serv. Comput. 99:1. doi: 10.1109/TSC.2019.2961895

CrossRef Full Text | Google Scholar

Wu, D., Luo, X., Shang, M., He, Y., Wang, G., Wu, X., et al. (2020). A Data-Characteristic-Aware Latent Factor Model for Web Service QoS Prediction. IEEE Trans. Knowl. Data Eng. 32:1.

Google Scholar

Wu, D., He, Y., Luo, X., and Zhou, M. (2021a). A Latent Factor Analysis-Based Approach to Online Sparse Streaming Feature Selection. IEEE Trans. Syst. Man Cybern. Syst. 1–15. doi: 10.1109/TSMC.2021.3096065

CrossRef Full Text | Google Scholar

Wu, D., Shang, M., Luo, X., and Wang, Z. (2021b). An L1-and-L2-Norm-Oriented Latent Factor Model for Recommender Systems. IEEE Trans. Neural Netw. Learn. Syst. 1–14. doi: 10.1109/TNNLS.2021.3071392

PubMed Abstract | CrossRef Full Text | Google Scholar

Wu, E. Q., Lin, C. T., Zhu, L. M., Tang, Z. R., Jie, Y. W., Zhou, G. R., et al. (2021c). Fatigue Detection of Pilots’ Brain Through Brain Cognitive Map and Multi-Layer Latent Incremental Learning Model. IEEE Trans. Cybern. [Epub Online ahead of print].

Google Scholar

Wu, Y., Zheng, B., and Zhao, Y. (2018). “Dynamic gesture recognition based on LSTM-CNN,” in 2018 Chinese Automation Congress (CAC), (Xi’an, China: IEEE). doi: 10.1109/TNNLS.2021.3071392

PubMed Abstract | CrossRef Full Text | Google Scholar

Yao, G., Lei, T., and Zhong, J. (2019). A review of convolutional-neural-network-based action recognition. Pattern Recognit. Lett. 118, 14–22. doi: 10.1109/TCYB.2021.3068300

PubMed Abstract | CrossRef Full Text | Google Scholar

Zeinali, Y., and Story, B. A. (2017). Competitive probabilistic neural network. Integr. Comput. Aided Eng. 24, 105–118. doi: 10.1109/CAC.2018.8623035

CrossRef Full Text | Google Scholar

Zhang, D., Xiong, A., Zhao, X., and Han, J. (2012). “PCA and LDA for EMG-based control of bionic mechanical hand,” in 2012 IEEE International Conference on Information and Automation, (Shenyang, China: IEEE). doi: 10.1016/j.patrec.2018.05.018

CrossRef Full Text | Google Scholar

Zhao, J., Xie, Z., Jiang, L., Cai, H., Liu, H., Hirzinger, G., et al. (2006). “EMG control for a five-fingered underactuated prosthetic hand based on wavelet transform and sample entropy,” in 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems, (Beijing, China: IEEE). doi: 10.3233/ICA-170540

CrossRef Full Text | Google Scholar

Zheng, X., and Chen, W. (2021). An Attention-based Bi-LSTM Method for Visual Object Classification via EEG. Biomed. Signal Process. Control 63:102174. doi: 10.1109/ICInfA.2012.6246955

CrossRef Full Text | Google Scholar

Zia Ur Rehman, M., Waris, A., Gilani, S. O., Jochumsen, M., Niazi, I. K., Jamil, M., et al. (2018). Multiday EMG-based classification of hand motions with deep learning techniques. Sensors 18, 2497–2505. doi: 10.1109/IROS.2006.282425

CrossRef Full Text | Google Scholar