Decoding finger flexion from band-specific ECoG signals in humans

This article presents the method that won the brain-computer interface (BCI) competition IV addressed to the prediction of the finger flexion from electrocorticogram (ECoG) signals. ECoG-based BCIs have recently drawn the attention from the community. Indeed, ECoG can provide higher spatial resolution and better signal quality than classical EEG recordings. It is also more suitable for long-term use. These characteristics allow to decode precise brain activities and to realize efficient ECoG-based neuroprostheses. Signal processing is a very important task in BCIs research for translating brain signals into commands. Here, we present a linear regression method based on the amplitude modulation of band-specific ECoG including a short-term memory for individual finger flexion prediction. The effectiveness of the method was proven by achieving the highest value of correlation coefficient between the predicted and recorded finger flexion values on data set 4 during the BCI competition IV.

1. Introduction

The goal of brain-computer interface (BCI) research is to reinstall control and communication capabilities for people with severe motor disabilities by translating brain signals into commands for a computer application or a neuroprosthesis (Wolpaw et al., 2002).

The neural electrophysiological signals currently being studied in the BCI domain ranges from electroencephalogram (EEG), electrocorticogram (ECoG), to local field potential (LFP) and single-unit activity/multi-unit activity (SUA/MUA). These different types of brain signals have their own characteristics and there is still controversy on the type of signals which is the most suitable for the BCI applications.

Electrocorticogram electrodes are placed over the surface of the cortex with typically 1 cm inter-electrode distance (Asano et al., 2005). On the one hand, ECoG provides a higher spatial resolution, a higher signal quality, and is more suitable for long-term use than the classical scalp EEG recordings. On the other hand, ECoG is less invasive than intracortical recordings like LFP, SUA/MUA which by far are used in a few BCI systems with human beings (Kennedy et al., 2000, 2004; Hochberg et al., 2006; Chadwick et al., 2011). By recognizing the merit of ECoG recordings, several groups of BCI researchers have carried out tests on the efficiency of using ECoG as control signals for human BCIs (Chin et al., 2007; Schalk et al., 2007; Pistohl et al., 2008; Sanchez et al., 2008).

Spatial resolution plays an important role in BCI (Sanchez et al., 2008). The fine spatial resolution of ECoG provides a better opportunity for directly decoding brain activities. Therefore, it is possible to implement direct neural interfaces which are difficult to be accomplished through EEG-based BCIs.

To study the usability of ECoG in BCIs, several research groups had recorded ECoG signals from the participants while they are performing certain kind of tasks related to the brain functional areas where the electrode arrays had been implanted. The tasks include center-out reaching or pointing task (Sanchez et al., 2008), grasping (Acharya et al., 2010), individual finger flexion [(Kubánek et al., 2009; Wang et al., 2010; Flamary and Rakotomamonjy, 2012) or (Acharya et al., 2010; Wang et al., 2011b)], and cursor trajectory (Schalk et al., 2007; Pistohl et al., 2008).

This paper describes the method we proposed to contribute to the ECoG data set from the BCI competition IV, which was dedicated to the task of decoding individual finger flexion in 2008. More precisely, for decoding individual finger flexion from ECoG, we noticed that a simple linear regression model of amplitude modulation (AM) of band-specific ECoG signals was efficient (Sanchez et al., 2008). Moreover, we made contribution to this method in two ways: firstly, we replaced the inverse operator in the solution of the linear model by the pseudo-inverse operator that should improve the stability of the model; secondly, we proposed to use a forward feature selection procedure to select the relevant frequency bands and electrodes (Langley, 1994). This method won the competition.

2. Materials and Methods

2.1. Brain-Computer Interface Competition IV – Data Set 4

The task for data set 4 in BCI competition IV was to predict the finger flexion from ECoG recordings.¹ Detail description about this data set can be found in Miller and Schalk (2008). Here, we only provide a brief summary.

This data set contains data for three subjects who were epileptic patients under surgical planning. Each subject had an electrode array placed subdurally on the surface of the brain in order to identify the epileptic focus. Each subject gave consent to participate in the recording experiments. While he/she performed a finger flexion task, the corresponding ECoG signals and the finger flexion time courses were recorded simultaneously. The electrode array was arranged in 8 × 6 or 8 × 8 grid (n.b., the exact location of the electrodes was unknown to the competitors because the electrode order had been scrambled during the preparation of this data set). There were 62, 48, and 64 channels for subject 1, 2, and 3, respectively.

Subjects were asked to flex a particular finger according to a visual cue (e.g., “index”) on a computer monitor. Typically for each cue, the subjects flexed the finger 3–5 times lasting 2 s followed by a rest period of 2 s. There were 30 movement for each finger resulting in 600-s recordings for each subject. The first 400-s recording were used as training set and the last 200-s recording used as testing set. Off-line analysis of the finger flexion time courses revealed that the movements of the last three fingers (i.e., middle, ring, and little fingers) were correlated in a considerable way.

The ECoG signals were recorded through the general-purpose BCI system BCI2000 (Schalk et al., 2004), bandpass filtered between 0.15 and 200 Hz and sampled at 1000 Hz. The finger movements were recorded using a dataglove and sampled at 25 Hz. Figure 1 provides an example of the visualization of the ECoG signals and the corresponding finger movement time course from subject 1. Due to space limitation, only a subset of ECoG electrodes is displayed. The correlation coefficient between the predicted and actual finger flexion time course has been used as the evaluation criterion for this data set in the competition.

FIGURE 1

Figure 1. For illustration purposes only, ECoG signals from channel 46 to 50 (at the top) and the corresponding movement time courses for each finger (at the bottom) for the first 60 s of the training data set from subject 1 are displayed.

2.2. Methods

2.2.1. Pre-processing

2.2.1.1. Band decomposition. The evidence of sensorimotor ECoG dynamics has been reported in several specific frequency bands including slow potentials, sub-bands (1–60 Hz), gamma band (60–100 Hz), fast gamma band (100–300 Hz), and ensemble depolarization (300–6k Hz) (Sanchez et al., 2008). Therefore, for this data set, the band-specific ECoG signals were generated using equiripple finite impulse response (FIR) filters by setting their band-pass specifications as: sub-bands (1–60 Hz), gamma band (60–100 Hz), and fast gamma band (100–200 Hz; n.b., considering the frequency content available in this data set, the fast gamma band was defined up to 200 Hz and the ensemble depolarization frequency band had not been taken into account). Therefore for each channel, raw ECoG signals were decomposed into three sets of band-specific ECoG signals.

2.2.1.2. Amplitude modulation. Being inspired by the rate coding approach used in spike train decoding, Sanchez proposed a band-specific AM as the descriptor for ECoG signal decoding, which is defined as the sum of the square voltage of the band-specific ECoG signals v in a time window Δt:

$x\left(t_n\right)=\sum _{t=0}^{\Delta t}{\upsilon }^2\left(t_n+t\right)$ (1)

where Δt = t_n+1 − t_n. We simply let Δt = 40 ms such that the resulting band-specific AM features have the same sampling rate (i.e., 25 Hz) as that of the dataglove position measurements. For each set of band-specific ECoG signals v, we applied equation (1) to estimate the band-specific AM features.

2.2.1.3. Feature selection. Since the ECoG electrode array covered quite a large zone of cortical area, only a subset of electrodes was correlated to the task. Moreover, we had no prior information about which frequency band contributed more than the other. Therefore, for each finger and each subject, we use a forward feature selection procedure in a wrapper approach (Langley, 1994) to identify relevant AM features (i.e., good channel/frequency band couples). The whole set of AM features equals to 186, 144, and 192, respectively, for subject 1, 2, and 3 whose had 62, 48, and 64 channels (for three frequency bands). According to the forward selection procedure, we started from the empty set adding one by one the feature that improves the most the correlation among the remaining set of features. So, this procedure is suboptimal but the exhaustive procedure which tests all possible subsets is very time consuming here. Feature selection has been achieved and evaluated by splitting the original training set into a training set (3/5) and a validation set (2/5). The stopping criterion is satisfied when the correlation coefficient for the validation set does not increase or when a user predefined maximum number of features is reached (e.g., 10 for the results presented in Figures 2 and 3).

FIGURE 2

Figure 2. Evolution of the performance for the index finger of subject 1 according to the number of features given by the forward feature selection procedure. For each step, the three bars from left to right represent the correlation coefficient for the training, validation, and test set, respectively.

FIGURE 3

Figure 3. The X-axis shows the first 10 most relevant selected features, from left to right. Each feature is indicated by two elements: channel index and frequency band (1, sub-bands; 2, gamma band; and 3, fast gamma band). The Y-axis indicates the correlation coefficients of training, validation, and test set, respectively, regarding each feature individually.

2.2.2. Linear regressor model

The relationship between the features and the target signals or the interaction between features was not clear for this case. We simply applied a linear model as a decoder for its robustness property. Although, we noticed that other advanced methods have been used for ECoG signals decoding, for example, the Kalman filter (Pistohl et al., 2008), this method is not suited for our case because it needs a finger model which we do not have. The linear model we used here takes the following form:

$d\left(t_n\right)=W^T\overrightarrow{x}\left(t_n\right)$ (2)

where d is the finger position as measured by the dataglove. $\overrightarrow{x}\left(t_n\right)$ is the short-term memory AM feature vector $\overrightarrow{x}\left(t_n\right)={\left[x\left(t_n\right)x\left(t_{n-1}\right)\dots x\left(t_{n-k}\right)\right]}^T.$ k is the number of values stored. The best results have achieved when k = 25. The coefficients W of the model are trained with the Wiener solution:

$W=E{\left({\overrightarrow{x}}^T\overrightarrow{x}\right)}^{-1}E\left({\overrightarrow{x}}^{T}d\right)$ (3)

where E is the expected mean. In order to improve the stability for estimating the coefficients of the Wiener model, we replaced the inverse operator in equation (3) by the pseudo-inverse operator.

3. Results

First, we present the feature selection results. For the feature selection procedure as described in Section 1, we stop the forward selection when the number of cycles is equal to 10 or when the correlation coefficient for the validation set does not increase.

Figure 2 gives an example on the evolution of the feature selection procedure for the index finger of subject 1. We found that there is no evident increment of the testing correlation coefficient by increasing the number of features after the first four features have been selected.

Figure 3 gives another point of view on the feature selection for the same subject and finger. This Figure emphasizes the individual prediction power of each feature selected during the first 10 cycles. Looking at the first three features, we learn that channel 1 was very useful (with two bands selected) and channel 40 was complementary. Last three features had lesser contribution to the task of decoding finger flexion.

Observing the 10 most relevant selected features over the 15 possible studies (three subjects × five fingers), we noted that they mainly characterized information in the gamma band (1–60 Hz: 27%, 60–100 Hz: 44%, and 100–200 Hz: 29%).

Next, we summarized the prediction performance of this method using the testing dataset in terms of correlation coefficient between the predicted and recorded finger movement in Table 1.² In order to highlight the effect of frequency-specific decomposition, the results based on the original ECoG signals (i.e., without band-pass filtering) are also provided for comparison.

TABLE 1

Table 1. The prediction performance of the methods (with and without band-specific features) is provided in terms of correlation coefficient between the predicted and recorded finger movement for each finger and subject.

From Table 1, we observed that the method based on band-specific AM features obtained better performance than the method using the original ECoG signals (i.e., without band-specific filtering). It explains that the decoding square voltage of brain signals lies in certain frequency bands and other frequency bands are more likely to be background noise.

We also provided in Figure 4 an example for the predicted finger movement for subject 3 using the method based on band-specific ECoG AM features. For comparison, the corresponding time course of the recorded finger movement is plotted in the same figure.

FIGURE 4

Figure 4. Time course of the predicted (solid red) and actual (dash blue) finger flexion for the first 60 s of the testing data set from subject 3.

4. Discussion

This article presents the method that won the BCI competition IV addressed to the prediction of the finger flexion from ECoG signals. This method used a linear decoding scheme based on band-specific AM for decoding individual finger flexion from ECoG signals in humans. The correlation between the predicted and recorded finger flexion shows that ECoG-based BCIs was a promising solution for implementing a practical and apt neuroprosthesis. In particular, we can confirm from the experimental results that the sensitivity profile of ECoG signals is band-specific. However it is not clear if the frequency selection scheme used here is optimal. In the forward feature selection procedure, we found that some features did not contribute too much to the prediction of finger flexion alone but ranked high in the sequence of the feature selection procedure. This inspires us to consider the correlation between band-specific ECoG signals. It suggests that incorporating the feature correlation into feature selection, for example, using correlation feature selection (CFS) method (Hall, 2000), may produce an optimal compact feature set. Furthermore in a recent study (Wang et al., 2010), a sparse Gaussian process (SPGP) has been applied for decoding finger flexion and a set of important features has been deduced from the length scale parameters in the trained SPGP.

We noticed that our method failed in some cases, especially for the middle finger of subject 1. It might be due to the considerable correlation between middle, ring, and little fingers. This draws our attention to the natural constraints that governs the movements of fingers (Wang et al., 2011a). In Wang et al. (2011a), it incorporates the prior knowledge about constraints that govern finger flexion through a prior model to improve the prediction accuracy. Their work achieved the higher correlation coefficient for this problem.

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Footnotes

1. ^The dataset is accessible through http://www.bbci.de/competition/iv/
2. ^The last element in Table 1 indicates the correlation coefficient value averaged over all fingers and subjects for the method based on band-specific ECoG, which is slightly different from the result of value 0.46 announced in the competition (http://www.bbci.de/competition/iv/results/index.html#dataset4) because ring finger was removed from the evaluation in the competition due to the finger movements of ring finger, through off-line inspection, were quite correlated with middle finger and little finger.

References

Acharya, S., Fifer, M. S., Benz, H. L., Crone, N. E., and Thakor, N. V. (2010). Electrocorticographic amplitude predicts finger positions during slow grasping motions of the hand. J. Neural Eng. 7, 046002.

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Asano, E., Juhász, C., Shah, A., Muzik, O., Chugani, D. C., Shah, J., Sood, S., and Chugani, H. T. (2005). Origin and propagation of epileptic spasms delineated on electrocorticography. Epilepsia 46, 1086–1097.

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Chadwick, E. K., Blana, D., Simeral, J. D., Lambrecht, J., Kim, S. P., Cornwell, A. S., Taylor, D. M., Hochberg, L. R., Donoghue, J. P., and Kirsch, R. F. (2011). Continuous neuronal ensemble control of simulated arm reaching by a human with tetraplegia. J. Neural Eng. 8, 034003.

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Chin, C., Popovic, M., Cameron, T., Lozano, A., and Chen, R. (2007). “Identification of arm movements using electrocorticographic signals,” in 3rd International IEEE/EMBS Conference on Neural Engineering, Kohala Coast, Hawaii, 196–199.

Flamary, R., and Rakotomamonjy, A. (2012). Decoding finger movements from ECoG signals using switching linear models. Front. Neurosci. 6:29. doi:10.3389/fnins.2012.00029

CrossRef Full Text

Hall, M. A. (2000). “Correlation-based feature selection for discrete and numeric class machine learning,” in 7th International Conference on Machine Learning (Morgan Kaufmann), 359–366.

Hochberg, L. R., Serruya, M. D., Friehs, G. M., Mukand, J. A., Saleh, M., Caplan, A. H., Branner, A., Chen, D., Penn, R. D., and Donoghue, J. P. (2006). Neuronal ensemble control of prosthetic devices by a human with tetraplegia. Nature 442, 164–171.

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Kennedy, P., Bakay, R., Moore, M., Adams, K., and Goldwaithe, J. (2000). Direct control of a computer from the human central nervous system. IEEE Trans. Rehabil. Eng. 8, 198–202.

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Kennedy, P., Kirby, M., Moore, M., King, B., and Mallory, A. (2004). Computer control using human intracortical local field potentials. IEEE Trans. Neural Syst. Rehabil. Eng. 12, 339–344.

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Kubánek, J., Miller, K. J., Ojemann, J. G., Wolpaw, J. R., and Schalk, G. (2009). Decoding flexion of individual fingers using electrocorticographic signals in humans. J. Neural Eng. 6, 066001.

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Langley, P. (1994). “Selection of relevant features in machine learning,” in Proceedings of the AAAI Fall Symposium on Relevance (AAAI Press), New Orleans, 140–144.

Miller, K. J., and Schalk, G. (2008). Prediction of Finger Flexion 4th Brain-Computer Interface Data Competition. Available at: http://www.bbci.de/competition/iv/desc-4.pdf

Pistohl, T., Ball, T., Schulze-Bonhage, A., Aertsen, A., and Mehring, C. (2008). Prediction of arm movement trajectories from ECoG-recordings in humans. J. Neurosci. Methods 167, 105–114.

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Sanchez, J. C., Gunduz, A., Carney, P. R., and Principe, J. C. (2008). Extraction and localization of mesoscopic motor control signals for human ECoG neuroprosthetics. J. Neurosci. Methods 167, 63–81.

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Schalk, G., Kubánek, J., Miller, K. J., Anderson, N. R., Leuthardt, E. C., Ojemann, J. G., Limbrick, D., Moran, D., Gerhardt, L. A., and Wolpaw, J. R. (2007). Decoding two-dimensional movement trajectories using electrocorticographic signals in humans. J. Neural Eng. 4, 264–275.

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Schalk, G., McFarland, D., Hinterberger, T., Birbaumer, N., and Wolpaw, J. (2004). BCI2000: a general-purpose brain-computer interface (BCI) system. IEEE Trans. Biomed. Eng. 51, 1034–1043.

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Wang, Z., Ji, Q., Miller, K., and Schalk, G. (2010). “Decoding finger flexion from electrocorticographic signals using a sparse Gaussian process,” in 20th International Conference on Pattern Recognition (ICPR), 2010, Istanbul, 3756–3759.

Wang, Z., Ji, Q., Miller, K. J., and Schalk, G. (2011a). Prior knowledge improves decoding of finger flexion from electrocorticographic (ECoG) signals. Front. Neurosci. 5:127. doi:10.3389/fnins.2011.00127

CrossRef Full Text

Wang, Z., Schalk, G., and Ji, Q. (2011b). “Anatomically constrained decoding of finger flexion from electrocorticographic signals,” in Annual Conference on Neural Information Processing Systems (NIPS), Granada.

Wolpaw, J. R., Birbaumer, N., McFarland, D. J., Pfurtscheller, G., and Vaughan, T. M. (2002). Brain-computer interfaces for communication and control. Clin. Neurophysiol. 113, 767–791.

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text