TNO Human Factors, Soesterberg, Netherlands

In this study, we investigated a Brain-Computer Interface (BCI) based on EEG responses to vibro-tactile stimuli around the waist. P300 BCIs based on tactile stimuli have the advantage of not taxing the visual or auditory system and of being potentially unnoticeable to other people. A tactile BCI could be especially suitable for patients whose vision or eye movements are impaired. In Experiment 1, we investigated its feasibility and the effect of the number of equally spaced tactors. Whereas a large number of tactors is expected to enhance the P300 amplitude since the target will be less frequent, it could also negatively affect the P300 since it will be difficult to identify the target when tactor density increases. Participants were asked to attend to the vibrations of a target tactor, embedded within a stream of distracters. The number of tactors was two, four or six. We demonstrated the feasibility of a tactile P300 BCI. We did not find a difference in SWLDA classification performance between the different numbers of tactors. In a second set of experiments we reduced the stimulus onset asynchrony (SOA) by shortening the on- and/or off-time of the tactors. The SOA for an optimum performance as measured in our experiments turned out to be close to conventional SOAs of visual P300 BCIs.

Introduction

A brain-computer interface (BCI), the first of which was developed by Vidal (1973 , 1977) , enables a user to communicate with the world through her or his brain signals alone. One of the most well-known BCIs was based on the P300 event-related potential (Farwell and Donchin, 1988 ). The P300 is a positive deflection in EEG that occurs around 300 ms after a target stimulus has been presented. In the P300 speller by Farwell and Donchin (1988) , rows and columns of a matrix consisting of letters were sequentially flashed in random order. Every time the row or column was flashed that contained the target symbol that the user wanted to spell, a P300 occurred. In this way, users could spell words. A P300-speller could be especially helpful for, and has been tested on paralyzed users (Hoffmann et al., 2008 ; Nijboer et al., 2008 ). More than two decades later, P300s are still popular to drive BCIs since they are relatively robust and easy to detect (Donchin et al., 2000 ).

The vast majority of P300 based BCIs use visual stimuli to elicit P300s. Recently (off-line) auditory P300 BCIs have also been developed (Sellers and Donchin, 2006 ; Furdea et al., 2009 ; Schreuder et al., 2010 ). The fundamental research on P300s focuses on visual and auditory stimulation. For BCI purposes tactile stimuli can be an interesting alternative. Tactile stimuli can be delivered by tactors that can be hidden under the user’s clothes, making the device go unnoticed to others. Using tactile stimuli will also keep the eyes and ears of the user free. Another important advantage is that a tactile P300 BCI can be used by patients with impaired vision or eye movements. This is especially relevant because one of the main potential groups of BCI users, patients with ALS, experience problems with their eyes as the disease progresses. A tap on your shoulder seems to automatically draw attention into the direction of the person tapping. Consistent with this, tactors applied around the waist have proven to be successful as navigation display, both for pedestrians (van Erp et al., 2005 ) and vehicle drivers (van Erp and van Veen, 2004 ) and can even result in better navigation performance than comparable visual displays. Tactile stimuli applied around the waist may therefore be a good choice for navigation BCIs (van Erp, 2005b , see also Thurlings et al., in press ). Users can focus their attention on the tactile stimulus that corresponds to the direction in which they want to move, possibly eliciting a useful P300 signal. Although tactile P300 BCIs have not been studied before, the feasibility of a BCI based on steady-state somatosensory evoked potentials has been demonstrated (Müller-Putz et al., 2006 ) and the tactile modality has been used as a BCI feedback channel (Chatterjee et al., 2007 ; Cincotti et al., 2007 ).

Previous research has shown that tactile stimuli can elicit P300s (Satomi et al., 1995 ; Ito and Takamatsu, 1997 ; Nakajima and Imamura, 2000 ). In these studies, electrical and mechanical stimuli were delivered to the hands and wrists using relatively long inter stimulus intervals (around 2 s in the study by Nakajima and Imamura). Obviously, for BCI purposes, short stimulus durations and inter stimulus intervals are required. The finding that P300s can be elicited by tactile stimulation of the hands does not guarantee that tactile stimulation of the torso easily elicits P300s as well because the sensitivity threshold and the spatial resolution of the torso is lower than that of the hands (Wilska, 1954 ).

Recently, we investigated whether quickly presented vibro-tactile stimuli at different locations around the waist can elicit robust P300s (Brouwer et al., submitted). We presented participants bursts of vibration delivered by one of eight tactors around the participants’ waist. Vibration duration was 188 ms and vibrations were interleaved by 438 ms breaks. In other conditions, we presented analogous visual stimuli consisting of flashed circles on a monitor in a schematic drawing of the tactor layout. Participants attended to the vibrations and/or flashes of the target tactor presented in a rapid stream of stimuli that also contained the seven distracters. The target was always the front tactor. In addition, this tactor physically stood out from the distracter: the distance to the distracters was larger than the distance between the distracters themselves, and the stimulus intensity of the target was stronger than of the distracters (in the tactile condition, a tactor above and below the target tactor vibrated simultaneously with the target, whereas in the visual condition the target circle was enlarged when flashed). By standing out from the distracters, the target probably drew attention by itself. The results demonstrated that the amplitudes of P300s elicited by tactile and visual stimuli depended on electrode site, but on the whole, they were equally high. Classification accuracy (how well an algorithm can identify the target stimulus) as determined off-line was also similar for visual and tactile stimuli. Whereas one or the other modality could probably have been favored by choosing different experimental parameters, we concluded that it is possible to elicit reliable P300s with quickly presented vibro-tactile stimuli around the waist.

The present study aimed at extending these proof-of-concept findings to a more realistic BCI setting. Whereas the oddball task as used in the previous experiment more or less guaranteed the attention of the participant to the target, it could not be generalized to a BCI where the user can choose the target. Furthermore, reliable online classification of tactile P300s remained to be demonstrated.

Besides tackling the issues mentioned above, we also wanted to explore the effect of varying the number of stimuli (i.e. the number of tactors around the waist) and the effect of varying stimulus timing on classification accuracy and rate of information transfer.

With a large number of tactors, it will be difficult to identify the individual targets because they will be close to the distracters. It has been shown that a low discriminability reduces the P300 amplitude (Comercho and Polich, 1999 ). On the other hand, with a large number of tactors, the target tactor is activated relatively infrequent, which positively affects the P300 (Polich et al., 1991 ), possibly via a longer target-to-target interval (Gonsalvez and Polich, 2002 ; Croft et al., 2003 ; Martens et al., 2009 ). In case of few tactors, the discriminability is high but the probability of target presentation is also high. In order to guide the development of a tactile BCI for navigation, we are interested in the relative importance or tradeoff of these potential effects on type classification accuracy and rate of information transfer within the context of our vibro-tactile stimuli around the waist.

Presenting stimuli quickly after each other shortens the time needed to go through the several different options. This means that the BCI could reach its decision as to which option the user wants to select sooner. However, increasing presentation speed will also likely decrease the quality of the P300 and decrease classification performance. Firstly, refractory effects could result in a lower P300 (Gonsalvez and Polich, 2002 ; Martens et al., 2009 ). Secondly, the quicker stimuli are presented, the more stimuli will be reflected in the EEG of the epochs analyzed by the classification algorithm, probably making successful classification harder (Martens et al., 2009 ). Finally, there will be a limit to presentation speed due to human limits in perception and attention. Several studies investigated the effect of different stimulus timing parameters on the P300. Generally, more time between stimuli seem to result in larger P300 amplitudes. However, this is not always found (Sellers et al., 2006 ) and the exact timing variable(s) underlying possible P300 effects are still not clear. In most studies about timing, the stimulus onset asynchrony (SOA) was varied, i.e. the time interval in between the start of one (non) target stimulus and the next (Farwell and Donchin, 1988 ; Polich et al., 1991 ; Allison and Pineda, 2006 ). Increasing the SOA while holding everything else constant will also increase the time interval between target stimuli (target-to-target interval), which may be the key variable affecting the P300 amplitude (Gonsalvez and Polich, 2002 ; Croft et al., 2003 ; Martens et al., 2009 ). Studies that show an increasing P300 with decreasing relative frequency of target presentation amongst non-targets (Polich et al., 1991 ; Cahill and Polich, 1992 ; Allison and Pineda, 2003 ; Sellers et al., 2006 ) also affect the target-to-target interval. Regardless of the specific underlying cause, decreasing the SOA (which decreases the target-to-target interval if the target probability stays the same) could cause a decrease in the P300 amplitude. Nonetheless, with the resulting shorter required time to go through the different user options, the rate of information transfer may increase. Many P300 BCI studies successfully used visual stimuli with SOAs ranging from as little as 125–500 ms (Farwell and Donchin, 1988 ; Serby et al., 2005 ; Sellers et al., 2006 ; Mell et al., 2008 ; Martens et al., 2009 ). We are interested in whether similar SOAs work for tactile stimuli as well.

In sum, the present study further evaluated (the feasibility of) a tactile P300 BCI. The tactile stimuli were delivered by several tactors placed around the waist. Classification for all experiments in this study was performed online, and the target stimulus did not stand out from the distracters. In Experiment 1, we varied the number of tactors. In Experiment 2 we reduced the SOA with respect to Experiment 1, both by reducing the vibration duration and the breaks in between vibration. Since it turned out that this was possible without sacrificing classification accuracy, we further reduced the SOA in Experiment 3.

Materials and Methods

Experiment 1

Participants

For all experiments described, participants were recruited from a pool of potential participants maintained by TNO Human Factors. Most of them study at a nearby university. In addition, some TNO interns (also students) participated. For each experiment, participants gave informed consent before the experiment started. Approval of the experiments was acquired from the local ethical committee.

Eleven healthy participants (six female) between 20 and 27 years old volunteered to participate in Experiment 1. None of them had participated in any other BCI experiment before.

Stimuli

Participants wore an adjustable vest over their clothes lined with 62 vibrating elements called tactors. For this study, we used one row of 12 equally spaced tactors around the participant’s waist, approximately at the height of the navel. Different combinations of tactors were used in different experimental conditions. Tactors vibrated successively with a vibration time (on-time) of 188 ms and breaks in between vibrations (off-time) of 438 ms. The tactors were custom built. They consisted of plastic cases with a contact area of 1 × 2 cm, containing 160 Hz vibrating motors. (TNO, The Netherlands, model JHJ-3; see van Erp et al. (2007) for comparable equipment and tactor layout). During the experiment, participants viewed a dimmed LCD (Dell 20 inch flatpanel, refresh rate 75 Hz) displaying instructions and a fixation cross. To prevent participants from using auditory cues, we played pink noise from two speakers, one placed left and the other placed right on the desk in front of the participant.

EEG Recording

EEG activity was recorded at the Fz, Cz, Pz, Oz, P3, P4, PO7 and PO8 electrode sites of the 10-20 system (Jasper, 1958 ) using electrodes mounted in an EEG cap (g.tec medical engineering GmbH). A ground electrode was attached to the forehead. The EEG electrodes were referenced to linked mastoid electrodes. The impedance of each electrode was below 5 kΩ. Data were sampled with a frequency of 256 Hz and filtered before storage by a 0.1-Hz high pass-, a 60-Hz low pass- and a 50-Hz notch filter (USB Biosignal Amplifier, g.tec medical engineering GmbH). The experiment (stimulus presentation and data recording) was controlled by a combination of custom built software and BCI2000 (Schalk et al., 2004 ).

Conditions

We used four experimental conditions called Two, Four, Six and FourPlus. Conditions Two, Four and Six refer to the number of equally distributed tactors used (see Figure 1 for a schematic indication of their location). In these conditions, the number of distracters was respectively one, three and five. The probability of target presentation decreased over these conditions (expected to enhance the P300), whereas the distance between the tactors and therewith the spatial discriminability between target and distracters decreased (expected to reduce the P300). The FourPlus condition served as a comparison for condition Four and Six. In this condition, the same tactors were used as in condition Four (see Figure 1 ), that is, the discriminability in Four and FourPlus was the same. However, by having the front and back tactor vibrate twice as often as the other tactors and by only designating the left and right tactors as targets, we made the target probability equal to that in condition Six. This will allow us to directly evaluate the relative importance of the factors discriminability and target probability: when discriminability plays a more important role, condition Four and FourPlus will produce similar results; when target probability is more important, condition Six and FourPlus will produce similar results.

FIGURE 1

Figure 1. Schematic overview of the tactor layout (top view) in the different conditions of Experiment 1. The tactors used are colored gray whereas the inactive tactors are white. In the FourPlus condition, only the gray tactors could be targets. The black tactors vibrated twice as often as the gray tactors.

Design

The order of conditions was randomized for each participant. For every condition, there were three blocks: two training blocks followed by one test block. After the training blocks, a classification algorithm was applied to the collected data. The resulting model was used in the test block to give participants feedback about the tactor that the algorithm classified as the target. Aside from this feedback, the training and test blocks were the same (please note that training refers to training the classifier, not training the participant).

A block consisted of six sequences of stimuli. Each sequence started with one of the tactors being designated as the target. In condition Six, each of the tactors served as a target once in one block, starting with the front-left tactor and every time going to the next tactor counter-clockwise. In condition Four, the front target and the one counter clockwise served as targets again after the others had been a target. In condition Two and FourPlus, the left and right tactor were designated as a target three times each, alternating and starting with the tactor at the left.

Each of the six sequences consisted of activating each tactor in that condition (the target and the one, three or five distracters) 10 times, in random order. With a stimulus on-time of 188 ms and a stimulus off-time of 438 ms this resulted in mean target-to-target intervals of 1252 ms (condition Two), 2504 ms (condition Four) and 3756 ms (conditions Six and FourPlus).

Task and procedure

Before the experiment started, the complete procedure was explained to the participants, with their task being to concentrate on the target by counting the number of times it occurred, and to ignore the distracters. They were further asked to fixate the fixation cross displayed on the screen, to blink as little as possible and to limit any other movements during tactile stimulation.

Participants were seated comfortably in front of a monitor in a dimly lit, shielded room, wearing the tactile vest and an EEG electrode cap. During the recording, an analog noise generator produced pink noise in order to mask the sound of the tactors. The monitor always displayed a fixation cross. A sequence of stimuli started by the appearance of the word ‘focus’ on the monitor. Simultaneously, one of the tactors vibrated for 750 ms. This indicated the target for the upcoming sequence. Then, each of the tactors used in that condition vibrated 10 times in random order. The appearance of a dashed line (in the case of a training block) or the word ‘result’ (in the case of a test block) indicated the end of a sequence. In the latter case, one of the tactors vibrated for 750 ms simultaneously with the word ‘result’, to indicate to the participant which tactor the algorithm designated as being the target. Participants took 1- to 5-min breaks in between blocks.

Classification algorithm and analysis

Classification models were built during the experiment for each participant and each of the four conditions after two training blocks. Using step-wise linear discriminate analysis (SWLDA; Krusienski et al., 2006 ) a maximum of 60 features were extracted from the EEG data. The epochs used in the analysis started at stimulus onset and ended 797 ms afterwards. The data was down sampled with a factor of four. Each of the two training blocks yielded 60 target epochs (6 sequences × 10 repetitions) for the algorithm to be trained on. Since the number of distracters differed between conditions, the number of distracter epochs per block was 60 in condition Two, 180 in condition Four and 300 in condition Six and Four Plus.

For each participant and each condition, we determined classification accuracy as the proportion of targets that was correctly identified by the algorithm. However, the conditions differed with respect to chance level: 50% or three targets correct for each participant in condition Two, 25% or 1.5 target correct in condition Four, and 17% or one target correct in the other conditions. Therefore, we also present the results after correcting for chance by subtracting chance performance from the observed classification accuracy and normalizing the results.

Besides classification accuracy, BCI performance can also be expressed in terms of bitrate (Wolpaw et al., 2002 ; Serby et al., 2005 ). This is a measure of performance that represents the amount of information being communicated, taking into account the number of possible options, the classification accuracy and the time required to communicate the chosen option. The bitrate is calculated as follows (Serby et al., 2005 ):

Bitrate = BM

where M is the mean number of decisions per minute and B is the amount of bits communicated per decision:

B = log₂N + Plog₂P + (1 − P)log₂[(1 − P)/(N − 1)].

N is the number of possible decisions and P is the accuracy probability. In this study P is the classification performance, without the correction for chance. N is set to 2, 4, 6 and 6 in conditions Two, Four, Six and FourPlus condition respectively. We computed bitrate for each subject and each condition, where we set bitrate to the theoretically maximal achievable bitrate of log₂(N) × M if the target was correctly identified after each of the six sequences, and to 0 if performance was below chance level.

We used one-sample t-tests, repeated measures ANOVA and post-hoc Tukey tests to evaluate classification accuracy and bitrate.

Results

Figure 2 A shows the classification accuracy for each condition, together with an indication of chance performance. Figure 2 B presents the classification accuracy corrected for chance. The one-sample t-tests against zero on classification accuracy corrected for chance showed that classification accuracy was well above chance for all conditions (all p-values < 0.01). A repeated measures ANOVA showed no effect of condition [F_(3,30) = 0.19, p = 0.90]. Figure 2 C shows the achieved bitrate for each condition. Again, a repeated measures ANOVA indicated no effect of condition [F_(3,30) = 0.11, p = 0.95]. Table 1 displays timing characteristics of the conditions in the different experiments, the maximal achievable bitrate, the mean achieved bitrate, the number of participants that achieved the maximal achievable bitrate and finally the mean bitrate aligned to Experiment 2 (see General Discussion). In every condition of Experiment 1, at least one participant achieved the maximal achievable bitrate.

FIGURE 2

Figure 2. Mean performance of the classification model in the four conditions of Experiment 1 as expressed by classification accuracy, or proportion of targets correct (A), classification accuracy corrected for chance (B) and bitrate (C). The stars in (A) indicate chance performance. Error bars represent standard errors of the mean.

TABLE 1

Table 1. Timing characteristics and bitrate results for the different conditions in the different experiments. The maximal bitrate is the bitrate that would be achieved when classification would always be correct. The mean bitrate is the bitrate achieved in the corresponding condition, averaged across participants. Asterisks indicate a significant difference within an Experiment as indicated by Tukey post-hoc tests. In the next column, the number of participants that achieved the maximal achievable bitrate is indicated. The last column presents the mean bitrate aligned to Experiment 2 (see General Discussion). Equal conditions in different experiments that serve to compare results between experiments are highlighted in bold and italic font.

Figure 3 depicts the mean Pz EEG samples from 200 ms before stimulus onset until 800 ms after, for each condition, and separately for target and standard presentations (solid and dashed curves respectively). Pz is a location where the P300 is usually strongly displayed (e.g. Ravden and Polich, 1999 ; Srinivasan, 2007 ). Clearly, a P300 is present in all conditions, even in condition Two, although the amplitude seems to be smaller. The P300 occured later than 300 ms after the start of the stimulus. This corresponds to many P300 studies that report this as well (e.g. Ravden and Polich, 1999 ; Bledowski et al., 2004 ; Gerson et al., 2006 ).

FIGURE 3

Figure 3. EEG averaged across participants and presented separately for targets (solid lines) and distracters (dashed lines) in the four conditions of Experiment 1. Only data from electrode Pz is presented.

Discussion

In a proof-of-concept study (Brouwer et al., submitted) we showed that an oddball tactile stimulus presented within a stream of tactile distracters around the waist elicited P300s that could be classified offline. Experiment 1 demonstrated that a BCI in which a tactile target is chosen among similar non-targets in an array around the waist is also feasible. Using different tactors as targets, online classification performance was well above chance for all conditions; ranging from an average of 58% correct for six alternatives to an average of 73% correct for two alternatives.

We did not find significant differences in classification accuracy corrected for chance between the conditions that varied in target-distracter discriminability and target probability. We expected at least to find a difference between the FourPlus condition and either the Four or the Six condition, or both. The FourPlus condition was expected to produce clearer P300s than the Four condition because of a more favorable target probability, and than the Six condition because of a more favorable discriminability. An explanation could be that in all conditions, target probability was low enough (or target-to-target interval long enough) and that the distinction between target and distracter was clear enough (van Erp, 2005a ), even though some participants remarked that in the Six condition, they sometimes experienced difficulties in distinguishing the target from adjacent distracters.

Since at least under these experimental conditions, different numbers of tactors do not affect classification accuracy and bitrate, the number of choices in a tactile BCI (two, four or six) can be directly tailored to the user application. For instance, it makes no sense to use a six tactor BCI for a yes–no choice.

The mean EEG signals suggested a slightly different picture than the classification results. The amplitude in condition Two seemed smaller than in the other conditions. The fact that this was not reflected in the classification accuracy could be because even though the amplitude was smaller, it was sufficiently different from that of the distracters. When fewer trials would have been used to train the classification algorithm, or fewer repetitions of the stimuli before the BCI decided on the target, there may have been a difference. Another possible reason for discrepancies between classification results and P300 amplitudes is that the classification model used different features than P300 amplitude only. From classification results obtained in experiments that are supposed to manipulate the P300, one cannot directly conclude anything about P300s without further looking at the signals and the features that a model uses. So-called P300 BCIs (including our own) probably often rely on non-P300 components in differentiating between targets and standards (Allison and Pineda, 2006 ; Brouwer et al., submitted).

Our main concern for this study was to build an online tactile BCI and to compare the effects of tactor number between conditions; not to optimize classification performance. Even so, the bitrate that we achieved is modest (see for example bitrates of online visual P300 BCIs in Table V of Serby et al. (2005) where bitrates range from 2.35 to 10.88 b/m). There are several possible reasons for our relatively low bitrate. We used only eight electrodes and each model was trained on a limited dataset obtained in 4–10 min. Even though the P300 signal itself is often referred to and liked for being relatively independent on training, most studies mentioned by Serby et al. (2005) used training times from seven sessions up to a few months. We expect that acquiring more data to train the model and leaving out less representative blocks (e.g. at the start of the experiment) could increase classification performance. Furthermore, there may be a difference between P300s elicited by visual and tactile stimuli. In virtually all visual P300 BCI paradigms, participants are allowed to fixate their eyes on the target. This will likely result in a perceptual difference between (fixated) target and (non-fixated) distracters as well as the possibility for the classification algorithm to distinguish between target and distracter on the basis of low-level visual evoked potentials (Treder and Blankertz, submitted). A strategy analogous to visual fixation is not possible in the tactile modality. A third reason for our modest bitrate may be the motivation of our (naïve) participants (Nijboer et al., 2009 ; Kleih et al., in press ). In our experimental procedure, we did not check whether participants counted the correct number of targets and the financial reward did not depend on performance. Introducing performance checks and possibly boosting their motivation by rewarding them for every correctly classified target stimulus may be ways to increase performance. Finally, our SOA of 626 ms is short compared to classic P300 studies, but long compared to P300 BCI studies, leaving a high bitrate unlikely (see Table 1 for maximal achievable bitrates).

In Experiment 2 we increased the maximal possible bitrate of the tactile P300 BCI by shortening the SOA. This was done by adjusting the stimulus on- and off-times by different amounts. Even though the effects of SOA and target-to-target interval on P300s and classification performance have been studied before (e.g. Farwell and Donchin, 1988 ; Serby et al., 2005 ; Allison and Pineda, 2006 ; Sellers et al., 2006 ; Martens et al., 2009 ; Mell et al., 2008 ), we did not encounter studies on the specific effect of stimulus on- and off-time. However, in a psychophysical discrimination study, varying these timing parameters produced differential effects (van Erp, 2005a ). Participants judged relative location of tactile stimuli. Whereas for vibration durations between 50 and 350 ms, vibration duration did not have a large effect on the discriminability, discriminability increased with increasing inter stimulus intervals in the range from 0 to 200 ms.