Christopher Wirth
Jake Toth
Mahnaz Arvaneh- Automatic Control and Systems Engineering Department, University of Sheffield, Sheffield, United Kingdom
Studies have established that it is possible to differentiate between the brain's responses to observing correct and incorrect movements in navigation tasks. Furthermore, these classifications can be used as feedback for a learning-based BCI, to allow real or virtual robots to find quasi-optimal routes to a target. However, when navigating it is important not only to know we are moving in the right direction toward a target, but also to know when we have reached it. We asked participants to observe a virtual robot performing a 1-dimensional navigation task. We recorded EEG and then performed neurophysiological analysis on the responses to two classes of correct movements: those that moved closer to the target but did not reach it, and those that did reach the target. Further, we used a stepwise linear classifier on time-domain features to differentiate the classes on a single-trial basis. A second data set was also used to further test this single-trial classification. We found that the amplitude of the P300 was significantly greater in cases where the movement reached the target. Interestingly, we were able to classify the EEG signals evoked when observing the two classes of correct movements against each other with mean overall accuracy of 66.5 and 68.0% for the two data sets, with greater than chance levels of accuracy achieved for all participants. As a proof of concept, we have shown that it is possible to classify the EEG responses in observing these different correct movements against each other using single-trial EEG. This could be used as part of a learning-based BCI and opens a new door toward a more autonomous BCI navigation system.
1. Introduction
Studies concerning robotic movement and navigation tasks have previously used electroencephalography (EEG) to investigate the brain's responses to observing correct and erroneous movements. These studies have shown that it is possible to classify the responses to correct movements against erroneous ones on a single-trial basis (Chavarriaga et al., 2014; Iturrate et al., 2015; Zander et al., 2016; Kim et al., 2017). Furthermore, a few recent studies have demonstrated the feasibility of using such correct-vs-error classification as feedback for reinforcement-learning-based Brain-Computer Interfaces (BCI) (Iturrate et al., 2015; Zander et al., 2016; Kim et al., 2017). Additionally, some studies have shown that different erroneous conditions can be classified against each other (Iturrate et al., 2010; Spüler and Niethammer, 2015; Wirth et al., 2019). These interesting advances have created the possibility of systems in which machines can control the low-level action decisions in order to navigate semi-autonomously toward a target, with feedback provided via implicit communication with a user through brain signals spontaneously generated while observing the task (Iturrate et al., 2015; Zander et al., 2016).
However, none of these previous studies have investigated whether it is possible to classify EEG responses to different types of correct actions against each other. In most navigation tasks, it is crucial not only to know that you are moving in the correct direction, but also to recognize when you have reached your destination. As such, it is highly important to consider whether there are significant neurophysiological differences between the brain's responses to observing different correct movements: those that get closer to a target, compared to those that actually reach it.
To address this question, we evaluated data from a virtual robotic navigation task. Participants were asked to observe a virtual robot, represented by a cursor, navigating in a 1-dimensional space and attempting to reach a target. We then investigated the EEG responses to movements that reached the target (hereafter referred to as the “TR condition,” short for “target reached”), in contrast to the responses to movements toward the target, but not reaching it (hereafter referred to as the “TT condition,” short for “toward target”).
To explore neurophysiologicial distinctions between the TT condition and the TR condition, we used time domain features to compare the latency and amplitude of key features of the event related potentials (ERPs). We also examined the spatial distribution of EEG responses to each condition, using topographical maps.
Our main focus was on the P300: a positive peak in an ERP at ~300 ms (Smith et al., 1970; Picton, 1992), known to be elicited in the brain when a subject recognizes a target stimulus in a sequence containing both target and non-target stimuli (Polich et al., 1991, 1996; Picton, 1992). The P300 has been successfully utilized in BCI, notably in spelling devices (Farwell and Donchin, 1988; Sellers and Donchin, 2006; Krusienski et al., 2008; Gugera et al., 2009; Fazel-Rezai et al., 2012). In these cases, the “target” stimulus is the specific character the user wishes to type. Each potentially desired character is typically highlighted a number of times, with each time being referred to as a “subtrial.” These subtrials are then averaged to increase the robustness of classification (Farwell and Donchin, 1988; Sellers and Donchin, 2006; Lotte et al., 2007; Krusienski et al., 2008; Gugera et al., 2009; Fazel-Rezai et al., 2012). Similar systems have also been developed for the control of robots (Lüth et al., 2007; Bell et al., 2008; Johnson et al., 2010; Bhattacharyya et al., 2014), cursors (Polikoff et al., 1995; Li et al., 2010; Kanoh et al., 2011), and wheelchairs (Rebsamen et al., 2006; Iturrate et al., 2009).
Unlike these previous studies utilizing the P300 for robotic control, and similar applications, in our study each stimulus (i.e., each movement) was only presented once, and so our classification phase required single-trial classification. Single-trial P300 classification is challenging, due in part to the low signal-to-noise ratio of EEG data (Jansen et al., 2004; Lotte et al., 2007), hence many systems presenting a number of subtrials. One study investigated the effects of different numbers of subtrials, and, while high accuracy was achieved with many subtrials, classification accuracy of <50% was reported based on a single subtrial, and 3 subtrials were required to achieve over 60% accuracy (Lenhardt et al., 2008). More recently, studies focusing on single-trial P300 classification have shown success, with some reporting accuracies over 80% (Finke et al., 2009; Korczowski et al., 2015; Lin et al., 2017). These studies were classifying the presence of a P300 against its absence. Our goal was to differentiate the P300s elicited in response to two slightly different desired actions. This presents an extra challenge, as we can expect the signals of the conditions to be more similar to each other.
In one previous study, one version of a task presented 80% standard stimuli and 20% target stimuli with all targets being identical to each other, while another version presented 80% standard stimuli and 20% target stimuli, with a pool of 25 different target stimuli; the latter case was found to elicit a broader P300 (Breton, 1988). While the responses to the different target stimuli were not compared to one another, this finding suggests that the P300 is affected by how often a specific stimulus appears in a task. Indeed, other literature has reported that P300 amplitude increases for larger target-to-target intervals (Gonsalvez and Polich, 2002). As well as this, the P300 has been shown to be associated with positive outcomes (Hajcak et al., 2005), and its amplitude has been shown to be affected by reward magnitude (Yeung and Sanfey, 2004; Sato et al., 2005; Wu and Zhou, 2009).
In this study, the desired stimulus is either a movement toward the target or, in cases when the virtual robot is adjacent to the target location, a movement that reaches the target. We hoped to identify and exploit differences between responses to these stimuli, arising from both the experimental differences (i.e., reaching the target occurs less frequently than other correct moves) and the participants' cognitive response to the two conditions (i.e., reaching the target may be considered more important than other correct moves). We then aimed to use the identified neurophysiological differences in order to classify the EEG responses to the two conditions against each other on a single-trial basis.
In order to classify responses to the conditions against each other, we implemented a stepwise linear discriminant analysis strategy, using time domain features from six electrode sites to generate subject-specific classification models. A second publicly available data set (Chavarriaga and Millán, 2010), gathered from participants observing a similar 1-dimensional navigation paradigm, was used to further validate the efficacy of the classification strategy. We tested our approach using data from 10 healthy young adults from the first task, and a further five healthy young adults from the second task.
2. Methods
This study uses data from two tasks. Neurophysiological analysis and single-trial classification were performed on data from Task 1. These data were recorded at the University of Sheffield, UK. Data from a Task 2 were used in order to further validate the single-trial classification section of the study. This was an open access data set, obtained under a Creative Commons Attribution—Non Commercial—No Derivatives 4.0 International license, based on a study by Chavarriaga and Millán (2010).
2.1. Participants
Ten healthy adults (4 female, 6 male, mean age 27.30 ± 8.31) were recruited to participate in Task 1. All of these participants were included in all aspects of the study. All participants had normal or corrected-to-normal vision. They reported no history of psychiatric illness, head injury, or photosensitive epilepsy. Written informed consent was provided by all participants before testing began. All procedures were in accordance with the Declaration of Helsinki, and were approved by the University of Sheffield Ethics Committee in the Automatic Control and Systems Engineering Department.
Six healthy adults (1 female, 5 male, mean age 27.83 ± 2.23) performed Task 2. 1 participant was excluded from this study as too few trials were available after artifact rejection.
2.2. Experimental Setup
2.2.1. EEG Setup
For Task 1, eight channels of EEG were recorded at 500 Hz using an Enobio 8 headset. The electrode sites recorded were Fz, Cz, Pz, Oz, C3, C4, P07, and PO8. A further reference electrode was placed on the earlobe.
For Task 2, 64 channels of EEG were recorded at 512 Hz using a BioSemi ActiveTwo system, and were referenced to the common average. Electrodes were placed using the 10–20 system.
2.2.2. Task 1
In Task 1, participants were seated in front of a screen and asked to observe a computer controlled cursor. Participants were presented with nine squares, arranged in a horizontal line, on a black background, as seen in Figure 1. The cursor's current square was colored blue. The target square was identified by a red bullseye symbol on a white background. All other squares were plain white.
Figure 1. Task 1 paradigm. Participants were asked to observe as a blue cursor attempted to navigate toward, and select, a marked target square. If the cursor was on the target, possible actions were either to select it by drawing a yellow box around the square, or take 1 step away from the target. If the cursor was not on the target, possible actions were either to move 1 step toward the target, move 1 step further away from the target, or erroneously select the current square as the target by drawing a yellow box around it. “TT” condition refers to “toward target,” i.e., movements toward, but not reaching, the target. “TR” condition refers to “target reached,” i.e., movements that did reach the target.
At the beginning of each run, the cursor appeared 2 or 3 squares away from the target location, either to the left or the right. Every 2 s, either the cursor would move to an adjacent square, or a yellow box would be drawn around the cursor's current position in order to identify that the computer believed that it had reached the target. Such target identification could occur correctly or erroneously. Actions occurred with preset probabilities, which depended on whether or not the cursor was on the target. These probabilities are shown in Table 1.
Table 1. Action probabilities for Tasks 1 and 2.
After the target was identified, either correctly or erroneously, the run finished and the screen was cleared. After 5 s, the next run began. A beep sounded 1 s before the start of each run. Participants were asked to refrain from movement and blinking during each run, but told that they could move and blink freely between runs, while the screen was blank. This process repeated until the end of the block, with each block lasting ~4 min.
Each participant performed a single session of observations. Participants were asked to observe blocks, with breaks of as long as they wished between blocks, until they reported their concentration levels beginning to decrease. Most participants observed six blocks of trials. However, two participants observed only 2 blocks. On average, Task 1 participants observed a total of 149.2 ± 40.0 (mean ± standard deviation) TT condition trials, and 82.3 ± 20.0 TR condition trials.
2.2.3. Task 2
In Task 2, participants were similarly asked to observe the 1-dimensional movement of a computer-controlled cursor. Twenty locations were arranged in a horizontal line across a screen. The cursor was displayed as a green square. The target was displayed as a blue square when it appeared to the left of the cursor, or a red square when it appeared to the right of the cursor.
At the beginning of a run, the target was drawn no more than three positions away from the cursor. Every 2 s, the cursor would move either toward or away from the target with preset probabilities, shown in Table 1. Unlike Task 1, no target identification was required by the computer. Instead, each run ended when the cursor reached the target. After this, the cursor stayed in its existing location, and a new target was drawn, again no more than three positions away from the cursor. This process repeated until the end of the block, with each block lasting 3 min.
Participants each performed two sessions of observations. Each session consisted of 10 blocks. The number of days between sessions varied between participants, from a minimum of 50 days to a maximum of more than 600 days. On average, Task 2 participants observed a total of 620.2 ± 10.6 TT condition trials, and 277.7 ± 14.1 TR condition trials.
2.3. Neurophysiological Analysis
Data from Task 1 were used for neurophysiological analysis. As we did not have control over the experimental paradigm for Task 2, and so did not have a precisely detailed picture of how the stimuli were presented, we opted not to perform neurophysiological analysis on Task 2 data, instead using these only to further validate the classification phase of this study.
Raw data from Task 1 were resampled to 64 Hz, and then band-pass filtered from 1 to 10 Hz, using a zero-phase Butterworth filter. TT and TR Trials were extracted from a time window of 0 to 1,000 ms, relative to the movement of the cursor. All extracted trials were baseline corrected relative to a period of 200 ms immediately before the movement of the cursor. Artifact rejection was performed by discarding any trials in which the range between the highest and lowest amplitudes, in any channel, was >100μV.
Grand average time domain event related potential (ERP) data were plotted using the extracted trials, showing the mean voltage ± 1 standard error, comparing responses to the TT condition with those to the TR condition.
Peak analysis was performed in order to identify the latencies at which the P300 occurred in the ERP data. Visual inspection of time domain ERP and topographical plots indicated that the highest P300 amplitude in this study occurred at electrode site Cz, and that there was a difference in P300 amplitudes in response to the two conditions at this site. As such, Cz was chosen as the most suitable channel for peak analysis. This peak analysis was carried out on the grand average ERP for responses to each condition. Subsequently, the P300 was identified as the highest positive peak, occurring between 200 and 500 ms. This time window was selected based on a visual inspection of the grand average time-domain data. To check for statistically significant differences in peak latencies, the same analysis was carried out to find the P300 peak in the average responses of each individual participant, for both conditions. According to one-sample Kolmogorov-Smirnov tests, we could not assume the data to be normally distributed. Therefore, a Wilcoxon signed-rank test was performed to compare the peak latencies identified for the two conditions.
To check whether there was a statistically significant difference in peak amplitude between responses to the two conditions, the mean amplitude was calculated in the responses the average responses of each individual participant, in a time window from 200 to 500 ms in order to encapsulate the full breadth of the P300. According to one-sample Kolmogorov-Smirnov tests, we could not assume the data to be normally distributed. Therefore, a Wilcoxon signed-rank test was performed to compare the amplitudes identified for the two conditions.
Topographical maps were then plotted for responses to each condition, using a 50 ms window surrounding the P300 latency (from peak −25 ms to peak +25 ms) as identified in the pooled data from all trials of both conditions combined. All topographical maps used the same scale, from the minimum value to the maximum values across all grand averages.
2.4. Single-Trial Classification
Single-trial classification was performed on data from both tasks. The same classification protocol was followed for both data sets, and is described in this section.
2.4.1. Pre-processing and Feature Extraction
Data from six electrode sites were used for single-trial classification: Fz, Cz, Pz, Oz, PO7, and PO8. These channels were selected based on visual inspection of grand average time domain ERPs, and considering prior knowledge related to these sites. The P300 has shown to peak in midline electrodes (Polich et al., 1997), and posterior sites, such as PO7 and PO8 are associated with visual processing (Deutsch et al., 1988; Wolber and Wascher, 2005; Schneider et al., 2012). As with the neurophysiological analysis, data were resampled at 64 Hz, trials were baseline corrected to a period of 200 ms immediately before presentation of the stimulus, and artifact rejection was performed to remove any trials with a range of >100μV between the highest and lowest amplitude in any channel. For the classification phase, data were band-pass filtered between 1 and 32 Hz. This band was selected after visual inspection of event-related spectral perturbation (ERSP) data which showed that, while most activity occurred at low frequencies, some potentially useful activity was also present in higher frequencies (see Supplementary Figure 1). Trials were extracted from 200 to 700 ms relative to the movement of the cursor. This window was selected based on visual inspection of grand average time domain ERPs. Selecting this window results in 33 samples per channel. Thus, in total, each trial was represented by 198 (6 × 33) features.
Previous literature has suggested that a minimum of 20 trials are required to provide stability in the P300 (Cohen and Polich, 1997). As such, we implemented a minimum cut-off of 20 artifact-free trials per class, in order to ensure we had enough data to produce a reliable training set. One participant was excluded from the single-trial classification phase of this study due to this cutoff.
2.4.2. Classification With Stepwise Linear Discriminant Analysis
In order to classify the data based on the most relevant subset of features, stepwise linear discriminant analysis was chosen as our classification approach, as previous literature has shown this strategy to be effective at both feature selection and classification of both P300 (Donchin et al., 2000; Krusienski et al., 2006, 2008; Sellers and Donchin, 2006; Lotte et al., 2018) and motion-onset visual evoked potential (mVEP) EEG data (Guo et al., 2008). An individual classification model was generated for each participant, using only the data from that individual participant's responses to the task. Firstly, for a given participant, an initial subset of features was selected. The amplitudes of the training trials for each condition were compared in each feature (i.e., each combination of channel and time point) using an unequal variances t-test. Features whose p-value was < 0.05 were included in the initial feature set. The stepwise procedure was then performed to select which features would be included in the final model. At each step, a regression analysis was performed on models with and without each feature, producing an F-statistic with a p-value for each feature. If the p-value of any feature was < 0.05, the feature with the smallest p-value would be added. Otherwise, if the p-value of any features already in the model had risen to >0.10 at the current step, the feature with the largest p-value would be removed from the model. This process continued until no feature's p-value reached the thresholds for being added to, or removed from, the model. If no features were added to the model at all, a single feature with the smallest p-value would be selected. Training and test trials were then reduced to the selected features.
The training set for the condition with the fewest training trials was oversampled in order to ensure that training occurred with an equal number of trials per condition. A linear classification model was then trained and tested. All classifiers were trained and tested using leave-one-out cross validation. To test statistical significance of the classification, a right-tailed Fisher's exact test was performed on the confusion matrix of each participant's results. In order to test whether the classification was significant at a group level, individual p-values were combined into a group p-value using Fisher's method (Loughin, 2004; Heard and Rubin-Delanchy, 2018).
3. Results
3.1. Neurophysiological Distinctions
In the responses to both conditions, grand average time domain ERPs showed a broad P300 peak, as can be seen in Figure 2A. Figures 2B,C show examples of time domain ERPs from individual participants (1 and 10, respectively). In both conditions, the shape of the broad P300 featured a peak shortly prior to 300 ms, followed by a slight drop in amplitude, and then a secondary peak, shortly after 400 ms. In responses to the TR condition, the earlier peak was found to have the highest amplitude, at a latency of 265 ms. The secondary peak marked the highest amplitude in grand average responses to the TT condition, with a latency of 420 ms. However, the Wilcoxon signed-rank test did not find a significant difference between the P300 peak latencies of responses to the two conditions (p = 0.81).
Figure 2. Time domain ERPs at electrode site Cz, from Task 1. Time shown is relative to movement of the cursor. Central lines represent mean signals. Shaded areas cover 1 standard error. Blue lines show TT condition data. Green lines show TR condition data. (A) Shows grand average data from all Task 1 participants, (B) shows data from participant 1, and (C) shows data from participant 10.
A distinction was seen between the P300 amplitudes of responses to the two conditions. The TR condition was observed to elicit a P300 with a greater amplitude than that generated in response to the TT condition. The Wilcoxon signed-rank test comparing the amplitudes of the two conditions, based on a time window from 200 to 500 ms in order to encapsulate the breadth of the P300, found this difference in amplitude to be statistically significant (p = 0.004).
Grand average time domain data for all eight electrode sites recorded for Task 1 are shown in Supplementary Figure 2.
Topographical maps plotted at the P300 peak latency showed the main activation to occur in the central midline, in response to both conditions, as can be seen in Figure 3.
Figure 3. Grand average topographical maps of Task 1 data. Maps were plotted based on a 50 ms window surrounding the peaks identified as P300 from grand average data across all participants, and both conditions. Plots shown represent (A) responses to the TT condition, and (B) responses to the TR condition.
We observed some features in the ERP responses to both conditions which may be related to motion-onset visual evoked potentials (mVEP). Such mVEPs occur when users percieve the beginning of movement of an object or symbol on a screen (Kuba et al., 2007; Guo et al., 2008; Marshall et al., 2013; Beveridge et al., 2019). Three main peaks have been identified in mVEP: a positive peak (P1), followed by a negative deflection (N2), then another positive peak with a latency of 240–500 ms (Kuba et al., 2007; Guo et al., 2008; Marshall et al., 2013; Beveridge et al., 2019), which has been described as a P2 (Kuba et al., 2007; Guo et al., 2008; Marshall et al., 2013) or P300 (Beveridge et al., 2019). The movements considered in this study were instantaneous steps from one location to the next. However, along with the P300, small P1 and N2 peaks were visible, with latencies of 78 and 125 ms, respectively, relative to the movement of the cursor. These peaks did not appear to differ between responses to the two conditions.
3.2. Classification
3.2.1. Classification of Task 1
The classification accuracies of each individual participant of Task 1 are shown in Table 2. The mean overall accuracy for all Task 1 participants was 66.5%. The mean accuracy for the TT condition was 68.8%, and the mean accuracy for the TR condition was 62.4%. Statistically significant separation of the conditions (p < 0.05) was found for all Task 1 participants. At a group level, the classification results for Task 1 were found to be statistically significant (p = 2.8 × 10−54).
Table 2. Single-trial classification results of Task 1 data.
3.2.2. Classification of Task 2
The classification accuracies of each individual participant of Task 2 are shown in Table 3. The mean overall accuracy for all Task 2 participants was 68.0%. The mean accuracy for the TT condition was 70.5%, and the mean accuracy for the TR condition was 61.0%. As with Task 1, statistically significant separation of the conditions (p < 0.05) was found for all Task 2 participants. At a group level, the classification results for Task 2 were found to be statistically significant (p = 9.6 × 10−62).
Table 3. Single-trial classification results of Task 2 data.
4. Discussion and Conclusion
4.1. Neurophysiological Distinctions Between the Conditions
In this study, the key neurophysiological difference that we identified between the two conditions was in the amplitude of the P300. The amplitude of the P300 was found to be greater in response to the TR condition (i.e., movements that reached the target) than the TT condition (i.e., movements that were correct, but did not reach the target). This distinction was found to be statistically significant (p = 0.004).
As discussed in section 1, a number of studies have reported that P300 amplitude is affected by reward magnitude (Yeung and Sanfey, 2004; Sato et al., 2005; Wu and Zhou, 2009). It should be noted that, in this study, participants were not directly rewarded based on the virtual robot's performance. However, it is certainly feasible that they regarded moves that reached the target as being more important than moves that did not reach it, which could be considered analogous to the TR condition having a higher reward magnitude. Reports have been mixed regarding the effects of valence on the P300. Some studies have reported amplitude being affected by positive valence (Cano et al., 2009; Wu and Zhou, 2009), while others have reported valence either having no effect (Carretié et al., 1997; Yeung and Sanfey, 2004), or an effect only in the case of negative valence (Conroy and Polich, 2007). P300 amplitude has also been shown to be dependent on whether feedback was expected or unexpected (Hajcak et al., 2005), and on target-to-target interval, with amplitude increasing when targets appeared less frequently (Gonsalvez and Polich, 2002).
Taking into consideration previous findings on the P300, and the experimental setup of our task, there are a number of potential causes of this increase in amplitude for responses to the TR condition, compared to the TT condition. It may represent a cognitive response recognizing that a move that reaches the target is a more important step than other correct moves. Alternatively, while this study was designed as a navigation observation task, it could also conceptually be considered as an oddball paradigm. That is to say, the TR condition occurs less frequently than the TT condition. Therefore, it is possible that the increased P300 amplitude is due to the relative rarity of the TR condition. It is quite possible that the difference in amplitude may be the result of a combination of these factors.
We also briefly investigated frontal theta power, and asymmetry in alpha power, as these have been reported to vary with regard to valence (Reuderink et al., 2013). However, no significant differences in these markers were identified between the conditions. It is certainly feasible that participants would not have had a strong emotional reaction to reaching the target. In Task 1, the goal was not fully achieved until the target was not only reached but also identified. Furthermore, users knew they were not controlling the virtual robot, and were not rewarded if it performed well. It may be interesting to investigate whether these valence markers indicate different reactions in future on-line experiments, in which participants' responses affect the actions of the virtual robot.
4.2. Single-Trial Classification
Previous studies have successfully classified the brain's responses to correct movements against responses to erroneous movements in navigation tasks, such as the ones explored in this study. The original study for which the data of Task 2 were generated reported classification accuracy of 75.8 and 63.2% for the correct and erroneous movement classes, respectively (Chavarriaga and Millán, 2010). Another study reported correct vs. erroneous movement classification accuracy, in three similar navigation tasks, of 73.8, 72.5, and 74.3% (Iturrate et al., 2015). It is reasonable to expect that the classification of two different correct movements against each other would be more challenging than the classification of correct movements against erroneous ones; we would expect to see more pronounced differences in the brain's responses in the latter case.
In this study, classifying EEG responses to correct movements toward the target (but not reaching it) against responses to movements that reached the target, we achieved mean overall classification accuracy of 66.5 and 68.0% for the two tasks. Indeed, these were only slightly below the levels previously reported for erroneous vs. correct movements in similar tasks. Interestingly, overall accuracy reached a high of 83.7% in the best case. Crucially, statistically significant separation of the two conditions (p < 0.05) was achieved for all participants from both tasks, and highly significant separation of the classes was shown at the group level (p = 2.8 × 10−54 and p = 9.6 × 10−62 for the Task 1 and Task 2, respectively).
As a proof of concept, we have shown that it is possible to classify responses to these two classes of correct movement against each other using single-trial EEG. As discussed in section 2.4.2, we chose to apply stepwise linear discriminant analysis in this study, as it has previously been shown to be successful in classifying similar data types (Donchin et al., 2000; Krusienski et al., 2006, 2008; Sellers and Donchin, 2006; Guo et al., 2008; Lotte et al., 2018). However, it is possible that other methodologies, which could be explored in future, may be able to provide further increases in classification accuracy. In potential future systems, classifications of the human observer's EEG responses could be used to guide the movement of a real or virtual robot, with the user being explicitly rewarded for good performance of the robot. In such systems, adding information from more frontal electrodes may be able to provide an increase in classification accuracy, as the frontal cortex has been shown to code prediction and reward (Schultz et al., 1997; Schultz, 2001; McClure et al., 2004).
4.3. Implications for BCI
The P300 has a history of successful use in BCI, as discussed in section 1. In particular, there have been many studies, dating back over 30 years, regarding the use of P300 signals in BCI spelling devices (Farwell and Donchin, 1988; Sellers and Donchin, 2006; Krusienski et al., 2008; Gugera et al., 2009; Fazel-Rezai et al., 2012). These systems have often been able to improve the robustness and accuracy of their classifications by using paradigms that allowed each stimulus to be presented multiple times, and the responses to be averaged. P300-based BCIs have also been created for other applications, such as video games (Finke et al., 2009; Kaplan et al., 2013), virtual reality (Bayliss, 2003), and control of robots (Lüth et al., 2007; Bell et al., 2008; Johnson et al., 2010; Bhattacharyya et al., 2014), cursors (Polikoff et al., 1995; Li et al., 2010; Kanoh et al., 2011) and wheelchairs (Rebsamen et al., 2006; Iturrate et al., 2009). Furthermore, the P300 has been utilized alongside other modalities, such as motor imagery (Su et al., 2011) and steady-state visual evoked potentials (SSVEP) (Yin et al., 2013) to create hybrid BCIs (Pfurtscheller et al., 2010; Müller-Putz et al., 2011; Amiri et al., 2013). The navigation scenarios presented in this study provided a further challenge compared to many previous P300-related systems, as each stimulus (i.e., movement) was only presented once. This was an important aspect of the paradigm, as we wished to simulate the observation of real navigation, with a view to future applications in which classifications could be made solely based on users' responses to the actions they observe. In such real navigation, each action occurs only once. While accurate single-trial P300 classification is challenging due to the low signal-to-noise ratio of EEG (Jansen et al., 2004; Lotte et al., 2007), some recent studies have shown that it can be achieved. One study using a video game context reported mean offline classification accuracy of 85%, and online accuracy of 66% (Finke et al., 2009). Another study reported single trial P300 classification accuracy of 70% (Jansen et al., 2004). In other cases, the area under the receiver operating characteristic curve (AUC) was reported for various possible classifier parameters, rather than the classification accuracy for a specific trained and optimized model. An AUC of over 0.8 has been reported for many participants (Korczowski et al., 2015; Lin et al., 2017). In this study, rather than classifying a condition eliciting a P300 against a condition that did not elicit a P300, we were classifying two P300-generating conditions against each other. As such the fact that statistically significant separation of two different correct conditions was achieved for all participants is encouraging for the use of the P300 in single-trial BCI scenarios.
In recent years, there have been interesting advances in BCIs based on signals that are generated spontaneously in the brain, without the need of a conscious effort to generate them on the part of the user. These systems, making use of implicit communication, have been described in two groups, referred to as “reactive BCI,” in which a spontaneous response is triggered by a stimulus, and “passive BCI,” whereby arbitrary mental states are measured (Zander et al., 2010, 2014; Zander and Köthe, 2011). Some particularly interesting recent studies have been those exploring reactive BCI in robotic movement and navigation tasks. Classification of error-related potentials (ErrP) in order to differentiate correct movements from erroneous ones has been combined with reinforcement learning in order to allow machines to perform a desired action (Kim et al., 2017) or navigate toward a desired target (Chavarriaga and Millán, 2010; Iturrate et al., 2015; Zander et al., 2016). By obtaining more detailed information from spontaneously generated signals, we can provide these systems with more context, and allow them to learn more efficiently and act more appropriately. The ability to classify when a target has been reached specifically and separately from other correct movements, as has been demonstrated in this study, would be an important aspect of a navigation system, and thus could enhance the usability and effectiveness of navigation-based BCI.
4.4. Conclusion
In this study, we compared the ERPs generated in EEG data, in response to observing two types of correct movements by a virtual robot: those that moved the robot closer to the target without reaching it, and those in which the robot reached the target. We were able to show that both correct movement conditions elicited a P300, and we identified a significantly higher P300 amplitude in cases in which the target was reached.
Interestingly, we were able to classify the responses to these two types of correct actions against each other with mean overall accuracies of 66.5 and 68.0% for two tasks, achieving statistically significant separation of the conditions for all participants. This single-trial classification could be used as part of a learning-based BCI, and opens a new door toward a more autonomous BCI navigation system.
Data Availability Statement
The raw data supporting the conclusions of this article will be made available by the authors, within 3 years, once further experiments have concluded, to any qualified researcher.
Ethics Statement
The studies involving human participants were in accordance with the Declaration of Helsinki, and were reviewed and approved by University of Sheffield Ethics Committee in the Automatic Control and Systems Engineering Department. The participants provided their written informed consent to participate in this study.
Author Contributions
MA conceived the study, supervised the work, and contributed to writing the paper. JT performed the data collection for Task 1. CW designed Task 1, performed the neurophysiological analysis and single-trial analysis, and wrote the paper.
Funding
This research was supported by University of Sheffield, United Kingdom, and a Doctoral Training Partnership Scholarship from the Engineering and Physical Sciences Research Council (EPSRC).
Conflict of Interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Supplementary Material
The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fnins.2020.00066/full#supplementary-material
References
Amiri, S., Fazel-Rezai, R., and Asadpour, V. (2013). A review of hybrid brain-computer interface systems. Adv. Hum. Comput. Interact. 2013, 1–8. doi: 10.1155/2013/187024
Bayliss, J. (2003). Use of the evoked potential p3 component for control in a virtual apartment. IEEE Trans. Neural Syst. Rehabil. Eng. 11, 113–116. doi: 10.1109/TNSRE.2003.814438
Bell, C. J., Shenoy, P., Chalodhorn, R., and Rao, R. P. N. (2008). Control of a humanoid robot by a noninvasive brain-computer interface in humans. J. Neural Eng. 5, 214–220. doi: 10.1088/1741-2560/5/2/012
Beveridge, R., Wilson, S., Callaghan, M., and Coyle, D. (2019). Neurogaming with motion-onset visual evoked potentials (mVEPs): adults versus teenagers. IEEE Trans. Neural Syst. Rehabil. Eng. 27, 572–581. doi: 10.1109/TNSRE.2019.2904260
Bhattacharyya, S., Konar, A., and Tibarewala, D. N. (2014). Motor imagery, P300 and error-related EEG-based robot arm movement control for rehabilitation purpose. Med. Biol. Eng. Comput. 52, 1007–1017. doi: 10.1007/s11517-014-1204-4
Breton, F., Ritter, W., Simson, R., and Vaughan, H. G. Jr. (1988). The N2 component elicited by stimulus matches and multiple targets. Biol. Psychol. 27, 23–44. doi: 10.1016/0301-0511(88)90003-8
Cano, M. E., Class, Q. A., and Polich, J. (2009). Affective valence, stimulus attributes, and p300: color vs. black/white and normal vs. scrambled images. Int. J. Psychophysiol. 71, 17–24. doi: 10.1016/j.ijpsycho.2008.07.016
Carretié, L., Iglesias, J., García, T., and Ballesteros, M. (1997). N300, P300 and the emotional processing of visual stimuli. Electroencephalogr. Clin. Neurophysiol. 103, 298–303. doi: 10.1016/S0013-4694(96)96565-7
Chavarriaga, R., and Millán, J. D. R. (2010). Learning from EEG error-related potentials in noninvasive brain-computer interfaces. IEEE Trans. Neural Syst. Rehabil. Eng. 18, 381–388. doi: 10.1109/TNSRE.2010.2053387
Chavarriaga, R., Sobolewski, A., and Millán, J. D. R. (2014). Errare machinale EST: the use of error-related potentials in brain-machine interfaces. Front. Neurosci. 8:208. doi: 10.3389/fnins.2014.00208
Cohen, J., and Polich, J. (1997). On the number of trials needed for P300. Int. J. Psychol. 25, 249–255. doi: 10.1016/S0167-8760(96)00743-X
Conroy, M. A., and Polich, J. (2007). Affective valence and P300 when stimulus arousal level is controlled. Cogn. Emot. 21, 891–901. doi: 10.1080/02699930600926752
Deutsch, G., Bourbon, W. T., Papanicolaou, A. C., and Eisenberg, H. M. (1988). Visuospatial tasks compared via activation of regional cerebral blood flow. Neuropsychologia 26, 445–452. doi: 10.1016/0028-3932(88)90097-8
Donchin, E., Spencer, K., and Wijesinghe, R. (2000). The mental prosthesis: assessing the speed of a p300-based brain-computer interface. IEEE Trans. Rehabil. Eng. 8, 174–179. doi: 10.1109/86.847808
Farwell, L., and Donchin, E. (1988). Talking off the top of your head: toward a mental prosthesis utilizing event-related brain potentials. Electroencephalogr. Clin. Neurophysiol. 70, 510–523. doi: 10.1016/0013-4694(88)90149-6
Fazel-Rezai, R., Allison, B. Z., Guger, C., Sellers, E. W., Kleich, S. C., and Kübler, A. (2012). P300 brain computer interface: current challenges and emerging trends. Front. Neuroeng. 5:14. doi: 10.3389/fneng.2012.00014
Finke, A., Lenhardt, A., and Ritter, H. (2009). The mindgame: a P300-based brain-computer interface game. Neural Netw. 22, 1329–1333. doi: 10.1016/j.neunet.2009.07.003
Gonsalvez, C. J., and Polich, J. (2002). P300 amplitude is determined by target-to-target interval. Psychophysiology 39, 388–396. doi: 10.1017/S0048577201393137
Gugera, C., Dabana, S., Eric Sellers, C. H., Krausza, G., Carabalonac, R., Gramaticac, F., et al. (2009). How many people are able to control a P300-based brain-computer interface (BCI)? Neurosci. Lett. 462, 94–98. doi: 10.1016/j.neulet.2009.06.045
Guo, F., Hong, B., Gao, X., and Gao, S. (2008). A brain-computer interface using motion-onset visual evoked potential. J. Neural Eng. 5, 477–485. doi: 10.1088/1741-2560/5/4/011
Hajcak, G., Holroyd, C. B., Moser, J. S., and Simons, R. F. (2005). Brain potentials associated with expected and unexpected good and bad outcomes. Psychophysiology 42, 161–170. doi: 10.1111/j.1469-8986.2005.00278.x
Heard, N. A., and Rubin-Delanchy, P. (2018). Choosing between methods of combining p-values. Biometrika 105, 239–246. doi: 10.1093/biomet/asx076
Iturrate, I, Montesano, L, and Minguez, J. (2010). “Robot reinforcement learning using eeg-based reward signals,” in Proceedings IEEE International Conference on Robotics and Automation (ICRA'10) (Anchorage, AK), 4822–4829. doi: 10.1109/ROBOT.2010.5509734
Iturrate, I., Antelis, J. M., Kübler, A., and Minguez, J. (2009). A noninvasive brain-actuated wheelchair based on a P300 neurophysiological protocol and automated navigation. IEEE Trans. Robot. 25, 614–627. doi: 10.1109/TRO.2009.2020347
Iturrate, I., Chavarriaga, R., Montesano, L., Minguez, J., and Millán, J. D. R. (2015). Teaching brain-machine interfaces as an alternative paradigm to neuroprosthetics control. Sci. Rep. 5:13893. doi: 10.1038/srep13893
Jansen, B. H., Allam, A., Kota, P., Lachance, K., Osho, A., and Sundaresan, K. (2004). An exploratory study of factors affecting single trial P300 detection. IEEE Trans. Biomed. Eng. 51, 975–978. doi: 10.1109/TBME.2004.826684
Johnson, G. D., Waytowich, N. R., Cox, D. J., and Krusienski, D. J. (2010). “Extending the discrete selection capabilities of the P300 speller to goal-oriented robotic arm control,” in 3rd IEEE RAS and EMBS International Conference on Biomedical Robotics and Biomechatronics (Tokyo), 572–575.
Kanoh, S., Miyamoto, K., and Yoshinobu, T. (2011). “A P300-based BCI system for controlling computer cursor movement,” in Proceedings of 33th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) (Boston, MA), 6405–6408.
Kaplan, A., Shishkin, S., Ganin, T., Basyul, I., and Zhigalov, A. (2013). Adapting the P300-based brain-computer interface for gaming: a review. IEEE Trans. Comput. Intell. AI Games 5, 141–149. doi: 10.1109/TCIAIG.2012.2237517
Kim, S. K., Kirchner, E. A., Stefes, A., and Kirchner, F. (2017). Intrinsic interactive reinforcement learning–using error-related potentials for real world human–robot interaction. Sci. Rep. 7:17562. doi: 10.1038/s41598-017-17682-7
Korczowski, L., Congedo, M., and Jutten, C. (2015). “Single-trial classification of multi-user P300-based brain-computer interface using riemannian geometry,” in Proceedings of 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) (Milan), 1769–1772.
Krusienski, D. J., Sellers, E., Mcfarland, D., Vaughan, T., and Wolpaw, J. (2008). Toward enhanced P300 speller performance. J. Neurosci. Methods 167, 15–21. doi: 10.1016/j.jneumeth.2007.07.017
Krusienski, D. J., Sellers, E. W., Cabestaing, F., Bayoudh, S., McFarland, D. J., Vaughan, T. M., et al. (2006). A comparison of classification techniques for the P300 speller. J. Neural Eng. 3, 299–305. doi: 10.1088/1741-2560/3/4/007
Kuba, M., Kubová, Z., Kremláček, J., and Langrová, J. (2007). Motion-onset VEPs: Characteristics, methods, and diagnostic use. Vision Res. 47, 189–202. doi: 10.1016/j.visres.2006.09.020
Lenhardt, A., Kaper, M., and Ritter, H. (2008). An adaptive P300-based online brain-computer interface. IEEE Trans. Neural Syst. Rehabil. Eng. 16, 121–130. doi: 10.1109/TNSRE.2007.912816
Li, Y., Long, J., Yu, T., Yu, Z., Wang, C., Zhang, H., et al. (2010). An EEG-based BCI system for 2-D cursor control by combining mu/beta rhythm and P300 potential. IEEE Trans. Biomed. Eng. 57, 2495–2505. doi: 10.1109/TBME.2010.2055564
Lin, Z., Zeng, Y., Tong, L., Zhang, H., Zhang, C., and Yan, B. (2017). Method for enhancing single-trial P300 detection by introducing the complexity degree of image information in rapid serial visual presentation tasks. PLoS ONE 12:e0184713. doi: 10.1371/journal.pone.0184713
Lotte, F., Bougrain, L., Cichocki, A., Clerc, M., Congedo, M., Rakotomamonjy, A., et al. (2018). A review of classification algorithms for EEG-based brain-computer interfaces: a 10 year update. J. Neural Eng. 15:031005. doi: 10.1088/1741-2552/aab2f2
Lotte, F., Congedo, M., Lécuyer, A., Lamarche, F., and Arnaldi, B. (2007). A review of classification algorithms for EEG-based brain-computer interfaces. J. Neural Eng. 4, R1–R13. doi: 10.1088/1741-2560/4/2/R01
Loughin, T. M. (2004). A systematic comparison of methods for combining p-values from independent tests. Comput. Stat. Data Anal. 47, 467–485. doi: 10.1016/j.csda.2003.11.020
Lüth, T., Ojdanic̀, D., Ola Friman, O. P., and Gräser, A. (2007). “Low level control in a semi-autonomous rehabilitation robotic system via a brain-computer interface,” in IEEE 10th International Conference on Rehabilitation Robotics (Noordwijk), 721–728.
Marshall, D., Coyle, D., and Wilson, S. (2013). “Motion-onset visual evoked potentials for gaming: a pilot study,” in 24th IET Irish Signals and Systems Conference (Letterkenny), 56.
McClure, S. M., York, M. K., and Montague, P. R. (2004). The neural substrates of reward processing in humans: the modern role of fmri. Neuroscientist 10, 260–268. doi: 10.1177/1073858404263526
Müller-Putz, G. R., Breitwieser, C., Cincotti, F., Leeb, R., Schreuder, M., Leotta, F., et al. (2011). Tools for brain-computer interaction: a general concept for a hybrid bci. Front. Neuroinform. 5:30. doi: 10.3389/fninf.2011.00030
Pfurtscheller, G., Allison, B. Z., Brunner, C., Bauernfeind, G., Solis-Escalante, T., Scherer, R., et al. (2010). The hybrid BCI. Front. Neurosci. 4:30. doi: 10.3389/fnpro.2010.00003
Picton, T. W. (1992). The P300 wave of the human event-related potential. J. Clin. Neurophysiol. 9, 456–479. doi: 10.1097/00004691-199210000-00002
Polich, J., Alexander, J. E., Baue, L. O., Kuperman, S., Morzorati, S., O'Connor, S. J., et al. (1997). P300 topography of amplitude/latency correlations. Brain Topogr. 9, 275–282. doi: 10.1007/BF01464482
Polich, J., Brock, T., and W. Geisler, M. (1991). P300 from auditory and somatosensory stimuli: probability and inter-stimulus interval. Int. J. Psychophysiol. 11, 219–222. doi: 10.1016/0167-8760(91)90015-P
Polich, J., Ellerson, P. C., and Cohen, J. (1996). P300, stimulus intensity, modality, and probability. Int. J. Psychophysiol. 23, 55–62. doi: 10.1016/0167-8760(96)00028-1
Polikoff, J. B., Bunnell, H. T., and Borkowski, W. J. (1995). “Toward a P300-based computer interface,” in RESNA'95 Annual Conference (Vancouver, BC), 178–180.
Rebsamen, B., Burdet, E., Guan, C., Zhang, H., Teo, C. L., Zeng, Q., et al. (2006). “A brain-controlled wheelchair based on P300 and path guidance,” in 1st IEEE/RAS-EMBS International Conference on Biomedical Robotics and Biomechatronics (Pisa), 1101–1106.
Reuderink, B., Mühl, C., and Poel, M. (2013). Valence, arousal and dominance in the EEG during game play. Int. J. Auton. Adapt. Commun. Syst. 6, 45–62. doi: 10.1504/IJAACS.2013.050691
Sato, A., Yasuda, A., Ohira, H., Miyawaki, K., Nishikawa, M., Kumano, H., et al. (2005). Effects of value and reward magnitude on feedback negativity and P300. Neuroreport 16, 407–411. doi: 10.1097/00001756-200503150-00020
Schneider, D., Beste, C., and Wascher, E. (2012). On the time course of bottom-up and top-down processes in selective visual attention: an EEG study. Psychophysiology 49, 1660–1671. doi: 10.1111/j.1469-8986.2012.01462.x
Schultz, W. (2001). Book review: reward signaling by dopamine neurons. Neuroscientist 7, 293–302. doi: 10.1177/107385840100700406
Schultz, W., Dayan, P., and Montague, P. R. (1997). A neural substrate of prediction and reward. Science 275, 1593–1599. doi: 10.1126/science.275.5306.1593
Sellers, E. W., and Donchin, E. (2006). A P300-based brain-computer interface: initial tests by als patients. Clin. Neurophysiol. 117, 538–548. doi: 10.1016/j.clinph.2005.06.027
Smith, D., Donchin, E., Cohen, L., and Starr, A. (1970). Auditory averaged evoked potentials in man during selective binaural listening. Electroencephalogr. Clin. Neurophysiol. 28, 146–152. doi: 10.1016/0013-4694(70)90182-3
Spüler, M., and Niethammer, C. (2015). Error-related potentials during continuous feedback: using EEG to detect errors of different type and severity. Front. Hum. Neurosci. 9:155. doi: 10.3389/fnhum.2015.00155
Su, Y., Qi, Y., Xun Luo, J., Wu, B., Yang, F., Li, Y., et al. (2011). A hybrid brain-computer interface control strategy in a virtual environment. J. Zhejiang Univ. Sci. C 12, 351–361. doi: 10.1631/jzus.C1000208
Wirth, C., Dockree, P. M., Harty, S., Lacey, E., and Arvaneh, M. (2019). Towards error categorisation in BCI: single-trial EEG classification between different errors. J. Neural Eng. 17:016008. doi: 10.1088/1741-2552/ab53fe
Wolber, M., and Wascher, E. (2005). The posterior contralateral negativity as a temporal indicator of visuo-spatial processing. J. Psychophysiol. 19, 182–194. doi: 10.1027/0269-8803.19.3.182
Wu, Y., and Zhou, X. (2009). The P300 and reward valence, magnitude, and expectancy in outcome evaluation. Brain Res. 1286, 144–122. doi: 10.1016/j.brainres.2009.06.032
Yeung, N., and Sanfey, A. G. (2004). Independent coding of reward magnitude and valence in the human brain. J. Neurosci. 24, 6258–6264. doi: 10.1523/JNEUROSCI.4537-03.2004
Yin, E., Zhou, Z., Jiang, J., Chen, F., Liu, Y., and Hu, D. (2013). A novel hybrid BCI speller based on the incorporation of SSVEP into the P300 paradigm. J. Neural Eng. 10:026012. doi: 10.1088/1741-2560/10/2/026012
Zander, T. O., Brönstrup, J., Lorenz, R., and Krol, L. R. (2014). “Towards BCI-based implicit control in human-computer interaction,” in Advances in Physiological Computing, eds S. H. Fairclough and K. Gilleade (London: Springer London), 67–90.
Zander, T. O., and Köthe, C. (2011). Towards passive brain-computer interfaces: applying brain-computer interface technology to human-machine systems in general. J. Neural Eng. 8:025005. doi: 10.1088/1741-2560/8/2/025005
Zander, T. O., Kothe, C., Jatzev, S., and Gaertner, M. (2010). “Enhancing human-computer interaction with input from active and passive brain-computer interfaces,” in Brain-Computer Interfaces: Applying Our Minds to Human-Computer Interaction, eds D. S. Tan and A. Nijholt (London: Springer London), 181–199.
Keywords: EEG, classification, BCI, human machine interaction, neurophysiology, P300, navigation, target recognition
Citation: Wirth C, Toth J and Arvaneh M (2020) “You Have Reached Your Destination”: A Single Trial EEG Classification Study. Front. Neurosci. 14:66. doi: 10.3389/fnins.2020.00066
Received: 03 September 2019; Accepted: 16 January 2020;
Published: 11 February 2020.
Edited by:
Damien Coyle, Ulster University, United KingdomReviewed by:
Yongtian He, University of Houston, United StatesDidem Gokcay, Middle East Technical University, Turkey
Copyright © 2020 Wirth, Toth and Arvaneh. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Christopher Wirth, cwirth1@sheffield.ac.uk