Brain Painting: first evaluation of a new brain–computer interface application with ALS-patients and healthy volunteers

Brain–computer interfaces (BCIs) enable paralyzed patients to communicate; however, up to date, no creative expression was possible. The current study investigated the accuracy and user-friendliness of P300-Brain Painting, a new BCI application developed to paint pictures using brain activity only. Two different versions of the P300-Brain Painting application were tested: A colored matrix tested by a group of ALS-patients (n = 3) and healthy participants (n = 10), and a black and white matrix tested by healthy participants (n = 10). The three ALS-patients achieved high accuracies; two of them reaching above 89% accuracy. In healthy subjects, a comparison between the P300-Brain Painting application (colored matrix) and the P300-Spelling application revealed significantly lower accuracy and P300 amplitudes for the P300-Brain Painting application. This drop in accuracy and P300 amplitudes was not found when comparing the P300-Spelling application to an adapted, black and white matrix of the P300-Brain Painting application. By employing a black and white matrix, the accuracy of the P300-Brain Painting application was significantly enhanced and reached the accuracy of the P300-Spelling application. ALS-patients greatly enjoyed P300-Brain Painting and were able to use the application with the same accuracy as healthy subjects. P300-Brain Painting enables paralyzed patients to express themselves creatively and to participate in the prolific society through exhibitions.

Introduction

Brain–computer interfaces (BCIs) are useful and accurate tools to provide communication for paralyzed people without the need of any voluntary muscular control (Kübler and Neumann, 2005; Hoffmann et al., 2008; Kübler and Birbaumer, 2008; Nijboer et al., 2008).

One possible brain-signal for BCI-control is the P300 event-related potential. The P300 is a positive deflection in the EEG occurring approximately 300 ms after the presentation of a rare stimulus (oddball) in between frequently presented standard stimuli (oddball paradigm, Sutton et al., 1965). Farwell and Donchin (1988) developed the first P300 BCI application. A 6 × 6 matrix containing the letters of the alphabet and the numerals 0–9 was presented to the user. The rows and columns were flashed randomly and the participant was asked to concentrate exclusively on the letter to be selected and to ignore the other letters. Therefore, the desired stimulus served as the rare event or oddball, while all unattended stimuli were standard stimuli. The P300 elicited by the oddball was detected, translated into the desired character and presented on a computer screen.

Today, additionally to the visual P300-BCI, also auditory and tactile P300-BCIs have been developed and their feasibility was demonstrated (Furdea et al., 2009; Brouwer and van Erp, 2010). However, even if these applications are promising for patients with disabled vision, to date, the visual P300-BCI is still the most developed and researched P300-BCI application for patients diagnosed with amyotrophic lateral sclerosis (ALS) (Nijboer et al., 2008; Silvoni et al., 2009; Townsend et al., 2010), a degenerative and fatal disease characterized by progressive paralysis due to degeneration of the upper and lower motor neurons (Cudkowicz et al., 2004). For example, Nijboer et al. (2008) showed that ALS-patients could use the P300-Spelling application with an offline classification accuracy of above 80%. With a visual four choice P300 BCI for ALS-patients to answer simple questions, a classification accuracy of 83% could be attained (Silvoni et al., 2009). To even improve spelling performance, Townsend et al. (2010) developed a new “checkerboard matrix” for the P300-Spelling application. Using the same matrix but an advanced flashing pattern based on a checkerboard, the accuracy of ALS-patients improved by 24.6% compared to the classical P300-Spelling application. However, because the applications for communication controlled by all non-invasive EEG-driven BCIs aim to fulfill the need for verbal communication and are restricted to letter or word selection, other BCI applications are needed to allow patients to express their creativity and actively participate in leisure activities. One such BCI application, the P300-based “neural signal surfing interface (NESSI)”, enables paralyzed patients to use the internet with accuracies of about 72% (Bensch et al., 2007; Mugler et al., 2008, 2010). The P300 controlled painting application called “Brain Painting” serves as a new P300-BCI application providing the possibility of creative expression. The Brain Painting application was designed by the German artist Adi Hösle in cooperation with the Institute of Medical Psychology and Behavioural Neurobiology at the University of Tübingen (Kübler et al., 2008). It is based on the P300-Spelling application described above. The cells of a 6 × 8 matrix contain symbols indicating color (), objects (), grid size (3 7 15 63 127 255 511), object size (1 2 4 8 ), transparency (25 50 100), zoom (Z+, Z−), and cursor movement (), which enable the user to create pictures (Kübler et al., 2008) (for an example picture see Figure 1A, for the colored P300-Brain Painting matrix see Figure 2B).

FIGURE 1

Figure 1. Example pictures. Example picture painted by patient A (with permission) (A) original picture painted by the artist (B) and the reproduction of the same picture (yellow circle is missing because the artist did not finish copying) (C). For the original picture, 21 selections were needed, for the replication 27 selections (six corrections) were needed.

FIGURE 2

Figure 2. Matrices: matrix of the P300-Spelling application (A) and the P300-Brain Painting application (B).

A pilot study addressing the usability of the P300-Brain Painting application was conducted and showed positive results (Kübler et al., 2008). Two participants took part in the pilot study: the artist Adi Hösle and a paralyzed patient, close to the completely locked-in syndrome (CLIS), diagnosed with late stage ALS. Both participants were able to paint pictures by selecting the appropriate symbols. For the proof of principle excluding that Brain Painting was not just a summation of accidental selections (which is clearly revealed by the appearance of the pictures) the artist reproduced one of his pictures (Figures 1B,C). After the pilot study, an exhibition was hosted at the Künstlerbund Tübingen (Exhibition hall of a German artist association) at which both, the artist and the patient, were painting. The ALS-patient was enthusiastic and showed high interest in the new application (Kübler et al., 2008). The exhibition illustrated the possibility of minimizing the gap between “healthy subjects” and “patients” through collaborative work in the field of art by implementing the Brain Painting application. It also demonstrated that Brain Painting can be realized in a real world environment.

The current study investigates accuracy and user-friendliness of the P300-Brain Painting application in healthy subjects and ALS-patients. We hypothesized that the P300-Brain Painting application would be usable with the same accuracy as the highly functional and precise P300-Spelling application, which is known to be usable for ALS-patients with accuracies above 70% over a time period of 40 weeks (Nijboer et al., 2008) and by healthy subjects with an accuracy of 99% (Kleih et al., 2010). We also predicted that ALS-patients would be highly motivated to use the Brain Painting application.

The study consisted of two different experiments: In experiment I (in the further text referred to as Exp I) the P300-Spelling application and the P300-Brain Painting application were compared regarding accuracy and P300 amplitudes in healthy subjects and ALS-patients. On the basis of the results of Exp I, in experiment II (Exp II), a new P300-Brain Painting application matrix was developed and accuracy and P300 amplitude were compared with the P300-Spelling application. It was designed in black and white to avoid differences with regards to stimulus intensity and complexity and the cells of the matrix were organized in a more self-explanatory manner.

Materials and Methods

Participants

Exp I included 3 ALS-patients (two female, age 53 and 42 years; one male, age 54 years, see Table 1) and 10 healthy volunteers (eight female, age M = 22.6, SD ± 4.43). None of the healthy subjects previously participated in a BCI study while two of the three ALS-patients had prior experience with P300-BCIs, but not with the P300-Brain Painting application. The healthy subjects were psychology students and were compensated with credit hours applicable for their studies. All participants gave informed consent to the study, which was approved by the Ethical Board of the Medical Faculty, University of Tübingen. They were right handed and the groups of healthy subjects had no history of prior psychological or neurological disorders.

TABLE 1

Table 1. Age and ALS-related data of patients.

In Exp II, a group of 10 healthy volunteers (six female, age M = 21.70, SD ± 2.71) without prior BCI experience participated. All participants were psychology students and were compensated with credit hours applicable for their studies. They were right handed, had no history of psychological or neurological disorders and gave informed consent to the study. The study was approved by the Ethical Review Board of the Medical Faculty, University of Tübingen.

Data Acquisition

EEG was measured using a 16-electrode cap (tin, 8 mm, Electro-cap International, Inc.). The electrodes F3, Fz, F4, T7, C3, Cz, C4, T8, Cp3, Cp4, P3, Pz, P4, Po7, Po8, and Oz were arranged according to the 10–20 international electrode system (Jasper, 1958; Sharbrough et al., 1991) and the impedance was maintained below 5 kΩ. The left mastoid served as ground, while the right mastoid was used as reference. The EEG signals were amplified using a 16-channel amplifier (g.USBamp, g.tec, Graz, Austria) and recorded by the BCI2000 software (Schalk et al., 2004) on a Dell Latitude notebook (Intel Pentium M, 2 GHz, 1 GB RAM, Windows XP, SP2). The signals were band-pass filtered between 0.1 to 30 Hz. An additional notch filter (48–52 Hz) was applied to remove the power-line noise.

Individual parameters for each subject necessary to control the device were calculated using the P300-GUI (part of the BCI2000 software package), an application developed in Matlab (The MathWorks™, MA, USA).

Signal Classification

Stepwise linear discriminant analysis (SWLDA) was used for signal classification. Qualified predictor variables were selected and included in a multiple regression model. No initial model terms were provided at the beginning of the calculation. In multiple steps, the currently most significant predictor variable (with p < 0.1) was included in the model (forward stepwise regression) while the currently least significant variable (with p > 0.15) was excluded from the model (backward stepwise regression). This selection process was repeated until no further predictor variables were found which met the inclusion/exclusion criteria better than the ones already included in the model or until the maximum number of features (60) was reached (Krusienski et al., 2008).

The scalp distribution of the P300 is known to be centro-parietal with highest amplitudes over midline scalp sites (Sutton et al., 1965). Therefore, in the current study the P300 amplitudes for the copy-spelling and copy-painting conditions were calculated for the Cz electrode. BrainVisionAnalyzer version 2 (BrainProducts GmbH, Munich, Germany) was used for offline analysis.

Design and Procedure

The interest of the current study was focused on accuracy and P300 amplitudes as dependent variables. Type of application (P300-Spelling application vs. P300-Brain Painting application) was manipulated as independent variable. The influence of the application on accuracy and P300 amplitudes was tested using a within-subject design.

Due to their restricted mobility, ALS-patients were presented with Brain Painting in their home environment while healthy volunteers were tested in the laboratory of the Institute of Medical Psychology and Behavioural Neurobiology of the University Tübingen.

During the measurement, all participants were sitting in a comfortable chair or wheelchair at a distance of approximately 1 m from the screens. One screen displayed the matrix while another screen showed the developing picture (the “canvas” in the words of painting). After a screening necessary to adjust the BCI to the individual user, participants had to fulfill three different tasks: spelling a pre-set sentence (copy-spelling, Kübler et al., 2001), painting a pre-set picture (copy-painting), and painting an individual picture (free-painting). The order of the copy-spelling and the copy-painting task was randomized between subjects to prevent order effects.

During the screening, subjects had to spell the words BRAIN and POWER using the copy-spelling (Kübler et al., 2001) function of the P300-Spelling application (Nijboer et al., 2008), where the letter to be written was indicated on the upper left corner of the screen. After adjusting the application to the individual user, the subjects started with the copy-spelling or copy-painting tasks.

The copy-spelling task consisted of spelling the sentence “(UWE,27.BüRO_LINKS!)” (engl: “(UWE,27.OFFICE_LEFT!)” where UWE is a male first name). This sentence was printed on a DIN A4 sheet of paper and placed in front of the screens instead of using the copy-spelling function included in the P300-Spelling application (the word to spell usually appears in the top line of a computer screen) to create equal test conditions as with the P300-Brain Painting application which does not have the copy function. This particular sentence was chosen to cover one letter in each row and column. To make it easier for the subjects to maintain their level of concentration, the sentence was split into three runs allowing for short breaks after the fifth and 13th selection. In case of an error, subjects were instructed not to correct it but to proceed with the next selection. In the similar copy-painting condition, subjects had to paint a pre-set picture matched to the pre-set sentence in number of choices (n = 20) and number and time-points of breaks. For the group of healthy subjects one symbol selection consisted of 15 flashes per row and column in the screening period and 10 flashes per row and column for the copy-spelling and copy-painting task. The duration of each flash was 62.5 ms with an ISI between flashes of 125 ms. Before and after each block of flashes a pre-/post-set interval of 3750 ms was inserted for the copy-spelling task and during the screening and of 1250 ms pre-set and 11250 ms post-set for the copy-painting and free-painting tasks. The longer post-set interval was chosen to allow participants to search and focus on the next item to choose, which may be more difficult in the Brain Painting matrix as the order does not follow external rules such as in the spelling matrix in which letters are sorted according to the alphabet. Taken together, selection of a symbol required 46.88 s during screening, 33.75 s during copy-spelling and 38.75 s during copy-painting and free-painting.

For the patients, the number of flashes during screening was the same as for the healthy subjects and was individually adapted for the copy-spelling, copy-painting, and free-painting tasks. Patients A and C were presented with 15 flashes per row and column for all tasks resulting in a duration of 46.88 s per selection for the copy-spelling task and 51.88 s for the copy-painting task. Patient B was presented with five flashes per row and column resulting in a total duration of 20.63 s per selection during copy-spelling and 25.63 s during copy-painting.

After the screening, the copy-spelling and the copy-painting task, the subjects could paint an individual picture. There were no time constraints; subjects could indicate when they wanted to stop. The duration of the free-painting period was noted.

Before each session the matrices and task were explained to the participants who were asked to focus their attention on the item to be selected by counting the number of times the item was flashing.

Matrices

In Exp I, a 6 × 8 matrix was used for the P300-Spelling application and the P300-Brain Painting application. For the P300-Spelling application, the letters of the German alphabet, the numerals 0–9 and some additional punctuation marks were displayed in white on a black background (see Figure 2A). For the P300-Brain Painting application, the letters, numerals, and punctuation marks were replaced by symbols necessary to choose object size, grid size, transparency, object shape, cursor movement, and color. All symbols were indicated by letters or numbers displayed in black on a white background with two exceptions: the symbols for colors were displayed in color on a differently colored background and the symbols for shapes were displayed in color on a white background (see Figure 2B).

In Exp II, the P300 Spelling matrix was identical to Exp I (see Figure 2A). The matrix of the P300-Brain Painting application was adapted to that of the P300-Spelling application: all symbols were displayed in white on a black background. The symbol categories (e.g., shape, color) were arranged in groups rendering the matrix more sorted (see Figure 3).

FIGURE 3

Figure 3. Adapted black and white matrix of the P300-Brain Painting application. Items for selection were sorted according to their function (e.g. color, shape, cursor position, intensity, etc.).

The possibility of “memory usage” related differences in presentation timing between the icons of the P300-Brain Painting application and the letters of the P300-Spelling application could be ruled out because the BCI2000 software loaded all icons into a bitmap buffer where they were re-sampled to the resolution and color depth of the display used in the experiment. Therefore, any icon used the same amount of memory in the experiment and there was no size difference between grayscale, black and white or colored icons.

Statistical Analysis

The group of ALS-patients was too small for statistical analysis; therefore individual data will be reported descriptively. For healthy subjects, all data was tested for Gaussian distribution using the Kolmogorov–Smirnov test. In case of Gaussian distribution, a paired-sample t-test was computed to compare the means of the accuracy and P300 amplitude between the P300-Spelling application and P300-Brain Painting application. For non-Gaussian distributions, the non-parametric Wilcoxon test was used.

All EEG-data was analyzed using BrainVisionAnalyzer version 2 (BrainProducts GmbH, Munich, Germany) and filtered with a low cutoff of 0.1 Hz and a high cutoff of 20 Hz. Data segments were extracted reaching from the start markers of each “flashing” period until 500 ms after the end marker of each flashing period. An interval of 100 ms before the start marker was used for baseline correction. The average ERPs were calculated for targets and non-targets separately, and a grand average was calculated across all participants separately for each of the two conditions. Peak detection was executed for the window of 200–500 ms after stimulus. Peak amplitude values per subject were exported to SPSS (SPSS Inc., 2008).

To provide a measurement of accuracy under conditions matching the “use in daily life” as close as possible (in which an environment free of artifact producing sources is unlikely), the EEG-data was not corrected for eye-blinks and artifacts.

Information Transfer Rates

The information transfer rate was calculated to measure the speed of command selection. Most publications employ a method suggested by Wolpaw et al. (2000)

with N being the number of possible selections in the matrix and P being the accuracy of the participant. This method assumes the specific accuracy for each selection (class) and the probability of each undesired selection to be the same. In the case of the P300-Spelling application and the P300-Brain Painting application these requirements are not met, because neighbor cells of the desired cell and cells in the same row or column have a higher probability to be accidentally selected (Fazel-Rezai, 2007; Kleih et al., 2010). Therefore, in this case the formula of Wolpaw et al. (2000) can only be used as an approximation of the information transfer rate.

A more specific method accounting for different probabilities of undesired selection per cell is the formula for mutual information (Nykopp, 2001) suggested by (Schlögel et al., 2007).

with p(x_i) being the a priori probability of the ith class defined as

with n_i: being the sum over the ith row of the confusion matrix and N being the total number of all trials per group. The confusion matrix is the M× M matrix of all commands of the P300-Spelling/P300-Brain Painting application matrix and each row i contains the number of times class i was classified as the class in column j. Therefore element n_ij contains the number of times class i was classified correctly. The confusion matrix was calculated based on the selections made by the groups of the experiment. It follows that p(u_j|x_i) in equation (2) is n_ij/n_i:. Furthermore p(y_j) is defined as

However, this method estimates the probability of the undesired selection per cell which requires a large number of data points for every selection. In the current study, this high number of data points was not available for a single subject because every selection has to be made only once or twice. Therefore, to give a complete overview of the information transfer rates and a comparison between group means, the current work reports the information transfer rates calculated with both formulas. Considering the strengths and weaknesses of both methods, the formula of Wolpaw et al. (2000) was used for group mean comparisons and to show the results of single patients. The formula for mutual information (Nykopp, 2001) was used to report the information transfer rates on the group level.

Psychological Data

Prior to each part of the measurement (copy-spelling, copy-painting, free-painting) all participants completed questionnaires to collect psychological data concerning mood and motivation. A German version of the Questionnaire on Current Motivation for BCI2000 (Fragebogen zur Erfassung aktueller Motivation für BCI2000, FAM-BCI2000) (Rheinberg et al., 2001; Nijboer et al., 2008) was used to assess current motivation in situations of BCI use. It consisted of 18 items subdivided in four different scales measuring incompetence fear, confidence, challenge, and interest which have to be rated on a seven-point Likert scale. Moreover, current mood was measured with a subscale of the “Skalen zur Erfassung der Lebensqualität (SEL)” (Averbeck et al., 1997), a German questionnaire to assess quality of life comprising 10 items to be rated on a five-point Likert scale. Additionally, each participant rated mood and motivation on a visual analog scale (VAS) ranging from 1 (not motivated/bad mood) to 10 (highly motivated/good mood).

At the beginning of the experiment, the ALS-patients completed a questionnaire to assess quality of life in ALS-patients (schedule for the evaluation of individualized quality of life, SEIQoL) (O’Boyle et al., 1995). Participants firstly had to define the five most important domains for their quality of life. Then satisfaction with and importance of each domain had to be rated in percent (0–100). The SEIQol index indicating quality of life was calculated by multiplying the satisfaction and importance of each life domain and summing it over all five life domains that were named by the patient (Browne et al., 1997) where a maximum of 100 could be reached. Additionally, patients had to complete a questionnaire to assess depression in ALS-patients (12 items ALS Depression Questionnaire, ADI-12) (Kübler et al., 2005a,b; Hammer et al., 2008). In contrast to conventional questionnaires measuring depression, the ADI-12, which consists of 12 items to be rated on a five-point Likert scale, considers the specific circumstances accompanying ALS.

The healthy controls completed an evaluation form at the end of the measurement, which is described in the next paragraph.

Evaluation Form

After the EEG measurement, each healthy participant completed a custom-made evaluation form gathering information about user satisfaction with the P300-Brain Painting application, the easiness of use comparing P300-Spelling and P300-Brain Painting application and subjective details about the level of concentration in each individual subject. The form consisted of nine questions. Answers were provided on a five-point Likert scale ranging from 1 (easy) to 5 (difficult) for questions one and two and 1 (not appropriate at all) to 5 (completely appropriate) for questions three to nine. Additionally, the questionnaire included a request for helpful suggestions. To compare subjective differences in easiness of use between the two applications, averaged ratings were compared.

The intention of this evaluation form was to gather information concerning possible subjective differences between the spelling and painting application and to use the information gathered to improve the application.

Results

All calculations reported here (Exp I and Exp II) were based on 19 selections instead of the originally planned 20 selections due to impediments in the protocol. Additionally, in Exp II, EEG recordings of the first six selections of the painting task could not be included for one subject due to technical problems.

For all healthy subjects (Exp I and Exp II), the duration of the whole session including preparation of the measurement and removing the electrodes after the measurement varied between 1 and 2 h dependent on the time spent for free-painting. The duration of the copy-spelling and copy-painting tasks without breaks added up to 24.17 min.

On average, the time spent on painting a picture during free-painting was 29.10 min (SD ± 18.59) in the group of healthy subjects using the colored matrix (Exp I) and 17.30 min (SD ± 9.92) in the group of healthy subjects using the black and white matrix (Exp II).

For the ALS-patients, some extra time was needed to fulfill their medical needs (e.g., suction cleaning). The duration of one entire session varied between 2 and 3 h. The duration of copy-spelling and copy-painting without breaks added up to 32.91 min for patients A and C and 15.42 min for patient B.

Due to fatigue none of the patients completed the free-painting task after the copy-spelling and copy-painting tasks.

Individual and average accuracy, information transfer rates and P300 amplitudes per group are depicted in Table 2.

TABLE 2

Table 2. Individual and average (av) accuracies, information transfer rates and P300-amplitudes per group (SD in parentheses).

Accuracy

In Exp I, a significant drop in average accuracy between the copy-spelling task and the copy-painting task was found in the healthy group (Z = −2.52, p < 0.05) while in Exp II performance did not differ between these two tasks (t₍₉₎ = 0.18, p = 0.86).

P300 Amplitudes

Patients’ individual P300 amplitudes and average P300 amplitudes per group of healthy subjects are depicted in Figure 4.

FIGURE 4

Figure 4. Amplitude of EEG is depicted as a function of time during spelling and painting. 380 trials were averaged across all subjects for targets and 4560 trials for non-targets. (A) the EEG of patients ((a) patient A, (b) patient B and (c) patient C), (B) healthy subjects using the colored matrix (Exp I) and (C) healthy subjects using the black and white matrix. (Exp II).

In the group of healthy subjects using the colored matrix (Exp I), a significant drop in P300 target amplitudes from copy-spelling to copy-painting was found (t₍₉₎ = 2.76, p < 0.05) which was absent with the black and white matrix in Exp II (t₍₉₎ = −1.00, p = 0.35).

Target and non-target P300 amplitudes were significantly differentiable in the copy-spelling conditions (Exp I: t₍₉₎ = 9.67, p < 0.05, Exp II: t₍₉₎ = 6.62, p < 0.05) and the copy-painting conditions (Exp I: Z = −2.09, p < .05, Exp II: t₍₉₎ = 5.88, p < 0.05).

Information Transfer Rates

Additionally to the information transfer rates calculated using the formula of Wolpaw et al. (2000) (Table 2), average information transfer rates per group of healthy subjects were also calculated using the formula of Nykopp (2001). These averaged transfer information rates (Nykopp, 2001) for Exp I/Exp II were 6.34/6.23 Bits/min for the copy-spelling task and 5.33/5.28 Bits/min for the copy-painting task.

Psychological data

The SEIQol index scores were 87.1 for patient A, 58.8 for patient B and 82.7 for patient C. Patient A reported communication (importance, I = 25%; satisfaction, S = 100%), social contact (I = 25%, S = 100%), intellectual stimulation (I = 25%, S = 70%), health (I = 20%, S = 85%), and finances (I = 5%, S = 55%) as most important areas of life.

Patient B reported family (I = 10%, S = 85%), health (I = 10%, S = 85%), communication (I = 60%, S = 45%), financial covering (I = 10%, S = 85%), and friends (I = 10%, S = 85%) as most important areas of life.

For patient C, the five most important areas of life reported were family (I = 40%, S = 90%), establish well-being (I = 20%, S = 85%), friends (I = 15%, S = 80%), communication (I = 10%, S = 70%), and mobility (I = 15%, S = 70%).

Depression was measured using the ADI-12 with a score of 17 for patient A, 18 for patient B and 26 for patient C of a total of 48. All patients were below the cut off (30) for Major Depressive Disorder (sensitivity 100%, specificity 83%). However, with a value of 26 patient C fell above the cut off (23) for the possibility of any affective disorder including minor depression (sensitivity 100%, specificity 60%) (Hammer et al., 2008).

All data concerning mood and motivation collected with the VAS, SEL, and FAM-BCI2000 per group are depicted in Table 3. For the FAM-BCI2000 the four components of motivation (mastery confidence, incompetence fear, interest, and challenge) are listed.

TABLE 3

Table 3. Mood and motivation per patient and group of healthy subjects before each session (SD in parentheses, patient C was not able to answer the questionnaires before the second session due to health reasons).

In the group of healthy subjects using the colored matrix (Exp I), a significant correlation was found between challenge (FAM-BCI2000) before copy-painting and the accuracy of copy-painting (r_s = 0.706, p < 0.05). No other significant correlations were found between any of the psychological variables tested and the accuracy of copy-painting or the time participants spent with free-painting.

In the group of healthy subjects using the black and white matrix (Exp II), challenge before free-painting was significantly correlated to the duration of free-painting (r_s = 0.64, p < 0.05). No significant correlations were found between psychological variables and the accuracy of copy-painting.

Evaluation Form

Analyzing the evaluation form in the group of healthy subjects using the colored matrix (Exp I), a significant difference in clarity and easiness of use could be found between the matrices of the P300-Spelling (M = 4.50, SD ± 0.53) and P300-Brain Painting (M = 2.70, SD ± 0.95) applications. The matrix of the P300-Brain Painting application was rated as significantly more complex and harder to use than the matrix of the P300-Spelling application (Z = −2.57, p < 0.05).

In the group of healthy subjects using the black and white matrix (Exp II), no significant difference in clarity and easiness of use between the matrices of the P300-Spelling application (M = 3.90, SD ± 1.10) and P300-Brain Painting application (M = 3.30, SD ± 0.95) could be found (Z = −1.07, p = 0.29).

Discussion

The current study investigated whether the new P300-Brain Painting application would provide the same accuracy as the well established and highly accurate P300-Spelling application and to test the acceptance of the application in severely impaired patients. ALS-patients were able to use the P300-Brain Painting application with high accuracies. In the group of healthy subjects, a drop of accuracy between the P300-Spelling application and the P300-Brain Painting application which was found for the colored matrix could be prevented by adapting the painting matrix to that of the P300-Spelling application.

In the first experiment (Exp I), it was hypothesized that the P300-Brain Painting application is usable with the same accuracy and evokes P300 amplitudes comparable to the P300-Spelling application. The group of ALS-patients was too small for statistical analysis. However, the P300-Brain Painting application and the P300-Spelling application were usable with a high accuracy (above 89%) in two of the three patients, even if the shape of the their P300 differed from that of the classical P300 in healthy subjects (see Figure 4). This can be attributed to the neurodegeneration which accompanies the progression of the disease (Paulus et al., 2002), however, our results show that also the modified P300s of the ALS-patients can be detected by the SWLDA and P300 amplitudes were comparable with that of healthy controls.

Patient B with lower accuracy rates reported afterwards that she was suffering from pain during the entire day, which could have influenced her results. Moreover, the fewer number of sequences (n = 5) chosen for this patient might have been too few on this day. As the other ALS-patients in this study, patient B participated in the framework of a broader study concerning evaluation and improvement of the user-friendliness of the P300 Brain Painting application in multiple brain painting sessions. She was an experienced P300-BCI user who in the screening achieved accuracies above 90% with five flashes (see also Nijboer et al., 2008, 2010). Instead of adapting the number of flashes each day (which would always require a screening and thus, additional time), we always used the same parameters in all sessions. We admit that this procedure reduces internal validity which would be higher when adapting the BCI in each session. However, our procedure provided a higher external validity as it represented daily life procedure with patients where only a limited amount of time is available and patients do not wish to perform in each and every session a (boring) screening procedure. Thus, we consider it very important to also report on patient B, because it shows that performance and EEG of patients may largely vary, when using BCI in the field. Moreover, her performance in the current study was, quite in contrast to her usual performance (see Nijboer et al., 2008), extremely low also during copy-spelling which indicated that the inferior performance during brain painting was not specific to the P300-Brain Painting application, but rather attributable to her current state of the disease which was characterized by considerable pain and loss of communicative abilities.

In the group of healthy subjects, our hypothesis of equal performance in both applications could not be confirmed. Lower accuracy and reduced P300 amplitudes were found for the P300-Brain Painting application as compared to the P300-Spelling application. These results might be explained by the perceived higher complexity and lower user-friendliness of the P300-Brain Painting matrix. This may imply that the participants needed more cognitive resources to decide which symbol to select, leaving less cognitive resources to focus on the target item for selection which is consistent with a focus of attention within working memory (Cowan, 1988). Moreover, most participants commented on the evaluation form that they wished the matrix of the P300-Brain Painting application to be more sorted according to function and less colorful.

Colors are perceived as salient stimuli among other symbols which in the P300-Brain Painting application could have influenced selective attention needed to detect the target stimulus (Johnston and Dark, 1986). The model of selective attention described by Theeuwes (1993) states that it is possible to direct visual attention spatially in a voluntary top-down process. However, within the spatial field of attention, stimulus selection depends on the physical properties of the stimuli and cannot be directed voluntarily (bottom-up process). Therefore, the stimulus which differs most in color, shape, or brightness is automatically selected as “target stimulus” (Hickey et al., 2006). With regards to the colored P300-Brain Painting matrix, the participants could have been distracted by the colors which in turn could have led to more errors. To test this assumption, a new matrix was designed and tested in a second experiment (Exp II). It was adapted to the matrix of the P300-Spelling application by implementing a black and white design. Moreover, the symbols were ordered in a more self-explanatory manner according to the results of the evaluation form gathered in the first experiment.

Concerning different aspects of mood and motivation on a descriptive level, higher values of interest, challenge, mood (VAS), and motivation (VAS) and lower values of incompetence fear were found for the group of ALS-patients compared to the group of healthy subjects. Two of the three patients had already experience with other P300-BCI applications which may explain the lower values of incompetence fear (Nijboer et al., 2008, 2010). Moreover, higher interest and challenge in ALS-patients could be attributed to the novelty of the application and they may have perceived more challenge because they wanted to perform as well as they did when using other P300-BCI applications. The higher values of mood and motivation on the VAS for patients as compared to healthy controls may have been because training with the P300-Brain Painting application was more important for the patients than for healthy subjects. While healthy subjects are able to express themselves and be creative without BCI, for ALS-patients the P300-Brain Painting application presented a great possibility of creative expression. This higher motivation could have had impact on the high overall performance because motivation was found to be positively correlated with the P300 amplitude in healthy subjects (Kleih et al., 2010).

Similarly, for healthy subjects we found a significant positive correlation between challenge and accuracy of copy-painting indicating that perceived challenge may be accompanied by more effort to perform better.

As an aside we would like to point out that none of the patients had Major Depressive Disorder, but moderate depressive symptoms were found in patient C. Patient C was still included in the study because he showed a well differentiable P300 and the mild depression was found not to interfere with the study design.

In the second experiment (Exp II), it was hypothesized that the drop in accuracy and P300 amplitudes between the P300-Spelling and the P300-Brain Painting application found in Exp I would disappear when using a black and white and functionally ordered matrix. This hypothesis could be confirmed as no significant difference was found between accuracy and P300 amplitude in the copy-spelling and copy-painting tasks.

The averaged results for accuracy and P300-amplitudes for the copy-spelling task did not differ between both experiments which supported the assumption that the equal performance during copy-spelling and copy-painting in Exp II was indeed due to an improvement in painting with the newly designed black and white matrix and not due to the unlikely possibility that the subjects in Exp II were scoring worse on the copy-spelling task while achieving equal results in the copy-painting task. Moreover, in line with these results, also the difference between the subjective evaluation of complexity and easiness of use between the matrices of the P300-Spelling and the P300 Brain Painting application disappeared. This was also in line with our hypothesis that the optical differences (Theeuwes, 1993) between matrices and differences in the order of the symbols negatively influenced subjects’ performance as found subjectively (questionnaire) as well as objectively (accuracies and P300 amplitudes).

Taken together, the current studies showed that the new P300-Brain Painting application is usable with high accuracy both in healthy subjects and severely impaired patients with ALS. Moreover, accuracies and P300 amplitudes found for the P300-Brain Painting applications are in line with prior studies which found accuracies and P300 amplitudes of >90%/5.90 μV (Kleih et al., 2010) and 91% (Guger et al., 2009) in healthy subjects and >70%/4.06 μV in ALS-patients (Nijboer et al., 2008). Patients enjoyed using brain painting and are continuing to work on paintings. P300-Brain Painting constitutes an application which enables ALS-patients to express themselves creatively and offers and opens them a new possibility of communication. It is especially remarkable that two of the ALS-patients showed even higher accuracies than most of the healthy subjects with the colored matrix. This is quite in contrast to many other studies in which patients perform worse with BCI when compared to healthy subjects (Kübler et al., 2004; Piccione et al., 2006). A possible explanation might be a difference in type of motivation between the ALS-patients and healthy subjects. A task can be performed for the joy of the involvement in the task itself without any reward (intrinsic motivation) as most likely in the group of ALS-patients or a task can be performed in expectation of a reward (extrinsic motivation) as more likely in the group of healthy subjects, because they received credits for participating (Deci, 1972). Because the black and white matrix allowed for higher accuracies than the colored matrix in healthy subjects, it should be tested with ALS-patients if their accuracies could be enhanced even further.

One limitation for the user-friendliness of the P300-Brain Painting application is its low information transfer rate compared to spelling, where information transfer rates of up to 23 bits/min were reported (Kleih et al., 2010). To date, such a high information transfer rate cannot be achieved with P300-Brain Painting rendering painting more time consuming. Based on our experience with Brain Painting users, we speculate, however, that speed of painting is less important than accuracy. That is why the number of sequences should always be high enough to ensure high accuracy; how high “high enough” is can only be defined by the individual Brain Painting user. Another restriction in the sense of artistic freedom is that the cursor can only be moved in a predetermined grid. To allow for less restricted cursor movement, research with a BCI–Brain Painting application using sensorimotor rhythms (SMR) as input signal is currently conducted (Halder et al., 2009). In this SMR-approach the user controls the BCI by moving a cursor by means of motor imagery which allows the user more flexible cursor movement (Müller-Putz et al., 2006). The possibility of moving a cursor in eight different directions was already shown by Wolpaw and McFarland (2004). Preliminary results on the applicability of SMR-based brain painting are promising and indicate that more creative flexibility might be possible using SMR as an input signal.

Conclusion

We conclude that the P300-Brain Painting application is usable with high accuracy by healthy subjects and neurological, severely paralyzed patients alike and it allows the user creative expression, which is specifically enjoyed by patients. In the future, matrix designs should be restrained to black and white elements; at least at the beginning of training. As patients experienced no drop in performance, extensive training with the P300 matrix might reduce the effect of selective attention, which favors the most salient stimuli. Also, other input signals than event-related potentials (ERP) should be tested to further optimize functionality and user-friendliness of the Brain Painting application. However, already today the P300-Brain Painting application, when applied as leisure time activity, allows the patients to be productive and creative and to participate in the prolific society by means of exhibitions of their paintings.

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Acknowledgments

This work is supported by the European ICT Programme Project FP7-224631. This paper only reflects the authors’ views and funding agencies are not liable for any use that may be made of the information contained herein. Special thanks to the patients who participated in the study for their time, effort and enthusiasm.

References

Averbeck, M., Leiberich, P., Grote-Kusch, M. T., Olbrich, E., Schröder, A., Brieger, M., and Schumacher, K. (1997). Skalen zur Erfassung der Lebensqualität (SEL). Manual. Frankfurt: Swets & Zeilinger B.V., Swets Test Services.

Bensch, M., Karim, A. A., Mellinger, J., Hinterberger, T., Tangermann, M., Bogdan, M., Rosenstiel, W., and Birbaumer, N. (2007). Nessi: an EEG-controlled web browser for severely paralyzed patients. Comput. Intell. Neurosci. 71863.

Brouwer, A.-M., and van Erp, J. B. F. (2010). A tactile P300 brain computer interface. Front. Neurosci. 4, 1–11. doi: 10.3389/fnins.2010.00019.

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Browne, J. P., O’Boyle, C. A., McGee, H. M., McDonald, N. J., and Joyce, C. R. (1997). Development of a direct weighting procedure for quality of life domains. Qual. Life Res. 6, 301–309.

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Cowan, N. (1988). Evolving conceptions of memory storage, selective attention, and their mutual constraints within the human information-processing system. Psychol. Bull. 104, 163–191.

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Cudkowicz, M., Qureshi, M., and Shefner, J. (2004). Measures and markers in amyotrophic lateral sclerosis. NeuroRx 1, 273–283.

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Deci, E. L. (1972). Intrinsic motivation, extrinsic reinforcement, and inequity. J. Pers. Soc. Psychol. 22, 113–120.

CrossRef Full Text

Farwell, L. A., and Donchin, E. (1988). Talking off the top of your head: toward a mental prosthesis utilizing event-related brain potentials. Electroencephalogr. Clin. Neurophysiol. 70, 510–523.

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Fazel-Rezai, R. (2007). Human error in P300 speller paradigm for brain-computer interface. Conf. Proc. IEEE Eng. Med. Biol. Soc. 2516–2519.

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Furdea, A., Halder, S., Krusienski, D. J., Bross, D., Nijboer, F., Birbaumer, N., and Kübler, A. (2009). An auditory oddball (P300) spelling system for brain-computer interfaces. Psychophysiology 46, 617–625.

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Guger, C., Daban, S., Sellers, E., Holzner, C., Krausz, G., Carabalona, R., Gramatica, F., and Edlinger, G. (2009). How many people are able to control a P300-based brain-computer interface (BCI)? Neurosci. Lett. 462, 94–98.

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Halder, S., Furdea, A., Leeb, R., Müller-Putz, G., Hösle, A., and Kübler, A. (2009). “Implementation of SMR based brain painting”. in Poster at Neuromath Workshop. Leuven, Belgium.

Hammer, E. M., Hacker, S., Hautzinger, M., Meyer, T. D., and Kubler, A. (2008). Validity of the ALS-Depression-Inventory (ADI-12)–a new screening instrument for depressive disorders in patients with amyotrophic lateral sclerosis. J. Affect. Disord. 109, 213–219.

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Hickey, C., McDonald, J. J., and Theeuwes, J. (2006). Electrophysiological evidence of the capture of visual attention. J. Cogn. Neurosci. 18, 604–613.

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Hoffmann, U., Vesin, J. M., Ebrahimi, T., and Diserens, K. (2008). An efficient P300-based brain-computer interface for disabled subjects. J. Neurosci. Methods. 167, 115–125.

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Jasper, H. (1958). The ten-twenty electrode system of the International Federation. Electroencephalogr. Clin. Neurophysiol. 10, 371–375.

Johnston, W. A., and Dark, V. J. (1986). Selective attention. Annu. Rev. Psychol. 37, 43–75.

CrossRef Full Text

Kleih, S. C., Nijboer, F., Halder, S., and Kubler, A. (2010). Motivation modulates the P300 amplitude during brain-computer interface use. Clin. Neurophysiol. 121, 1023–1031.

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Krusienski, D. J., Sellers, E. W., McFarland, D. J., Vaughan, T. M., and Wolpaw, J. R. (2008). Toward enhanced P300 speller performance. J. Neurosci. Methods 167, 15–21.

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Kübler, A., and Birbaumer, N. (2008). Brain-computer interfaces and communication in paralysis: extinction of goal directed thinking in completely paralyzed patients? Clin. Neurophysiol. 119, 2658–2666.

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Kübler, A., Halder, S., Furdea, A., and Hösle, A. (2008). “Brain painting – BCI meets art,” in Proceedings of the 4th International Brain-Computer Interface Workshop and Training Course, 361–366.

Kübler, A., Kotchoubey, B., Kaiser, J., Wolpaw, J. R., and Birbaumer, N. (2001). Brain-computer communication: unlocking the locked in. Psychol. Bull. 127, 358–375.

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Kübler, A., and Neumann, N. (2005). Brain-computer interfaces – the key for the conscious brain to the paralyzed body. Prog. Brain Res. 150, 513–525.

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Kübler, A., Neumann, N., Wilhelm, B., Hinterberger, T., and Birbaumer, N. (2004). Predictability of brain-computer communication. J. Psychophysiol. 18, 121–129.

CrossRef Full Text

Kübler, A., Winter, S., Kaiser, J., Birbaumer, N., and Hautzinger, M. (2005a). Das ALS-depressionsinventar (ADI): Ein Fragebogen zur Messung von Depression bei degenerativen neurologischen Erkrankungen (amyotrophe Lateralsklerose) – a questionnaire to measure depression in degenerative neurological diseases. Z. Klin. Psych. Psychother. 34, 19–26.

Kübler, A., Winter, S., Ludolph, A. C., Hautzinger, M., and Birbaumer, N. (2005b). Severity of depressive symptoms and quality of life in patients with amyotrophica lateral sclerosis. Neurorehabil. Neural Repair 19, 182–193.

CrossRef Full Text

Mugler, E., Bensch, M., Halder, S., Rosenstiel, W., Bodgan, M., Birbaumer, N., and Kübler, A. (2008). Control of an Internet Browser using the P300 event-related potential. Int. J. Bioelectromag. 10, 56–63.

Mugler, E., Ruf, C., Halder, S., Bensch, M., and Kübler, A. (2010). Design and implementation of a P300-based brain-computer interface for controlling an internet browser. IEEE Trans. Neural Syst. Rehabil. Eng. doi: 10.1109/TNSRE.2010.2068059 [Epub ahead of print].

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Müller-Putz, G., Scherer, R., Pfurtscheller, G., and Rupp, R. (2006). Brain-computer interfaces for control of neuroprostheses: from synchronous to asynchronous mode of operation. Biomed. Tech. 51, 57–63.

CrossRef Full Text

Nijboer, F., Birbaumer, N., and Kubler, A. (2010). The influence of psychological state and motivation on brain-computer interface performance in patients with amyotrophic lateral sclerosis - a longitudial study. Front Neurosci. 4:55. doi: 10.3389/fnins.2010.00055.

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Nijboer, F., Sellers, E. W., Mellinger, J., Jordan, M. A., Matuz, T., Furdea, A., Halder, S., Mochty, U., Krusienski, D. J., Vaughan, T. M., Wolpaw, J. R., Birbaumer, N., and Kubler, A. (2008). A P300-based brain-computer interface for people with amyotrophic lateral sclerosis. Clin. Neurophysiol. 119, 1909–1916.

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Nykopp, T. (2001). Adaptive Brain Interface (M.Sc Diploma). Finland: Helsinki University of Technology, Dep. of Electrical and Comm Eng.

O’Boyle, C., Browne, J., Hickey, A., McGee, M., and Joyce, C. (1995). Schedule for the Evaluation of Individual Quality of Life (SEIQol): a Direct weighting procedure for Quality of Life Domains (SEIQol-DW). Administration Manual. Dublin: Department of Psychology, Royal College of Surgeons in Ireland.

Paulus, K. S., Magnano, I., Piras, M. R., Solinas, M. A., Solinas, G., Sau, G. F., and Aiello, I. (2002). Visual and auditory event-related potentials in sporadic amyotrophic lateral sclerosis. Clin. Neurophysiol. 113, 853–861.

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Piccione, F., Giorgi, F., Tonin, P., Priftis, K., Giove, S., Silvoni, S., Palmas, G., and Beverina, F. (2006). P300-based brain computer interface: reliability and performance in healthy and paralysed participants. Clin. Neurophysiol. 117, 531–537.

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Rheinberg, F., Vollmeyer, R., and Burns, B. (2001). FAM: Ein Fragebogen zur Erfassung aktueller Motivation in Lern- und Leistungssituationen. Diagnostica 47, 57–66.

CrossRef Full Text

Schalk, G., McFarland, D. J., Hinterberger, T., Birbaumer, N., and Wolpaw, J. R. (2004). BCI2000: a general-purpose brain-computer interface (BCI) system. IEEE Trans. Biomed. Eng. 51, 1034–1043.

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Schlögel, A., Kronegg, J., Huggins, J. E., and Mason, S. G. (2007). “Evaluation criteria for BCI research,” in Towards Brain-Computer Interfacing, eds C. Dornhege, J. R. Millan, T. Hinterberger, D. J. McFarland, and K. R. Müller (Cambridge, MA, MIT Press), 327–342.

Sharbrough, F., Chatrian, G. E., Lesser, R. P., Lüders, H., Nuwer, M., and Picton, T. W. (1991). American Electroencephalographic Society guidelines for standard electrode position nomenclature. J. Clin. Neurophysiol. 8, 200–202.

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Silvoni, S., Volpato, C., Cavinato, M., Marchetti, M., Priftis, K., Merico, A., Tonin, P., Koutsikos, K., Beverina, F., and Piccione, F. (2009). P300-based brain-computer interface communication: evaluation and follow-up in amyotrophic lateral sclerosis. Front. Neurosci. 3, 1–12. doi: 10.3389/neuro.20.001.2009.

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Sutton, S., Braren, M., Zubin, J., and John, E. R. (1965). Evoked-potential correlates of stimulus uncertainty. Science 150, 1187–1188.

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Theeuwes, J. (1993). Visual selective attention: a theoretical analysis. Acta Psychol. (Amst) 83, 93–154.

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Townsend G., LaPallo, B. K., Boulay, C. B., Krusienski, D. J., Frye, G. E., Hauser, C. K., Schwartz, N. E., Vaughan, T. M., Wolpaw, J. R., and Sellers, E. W. (2010). A novel P300-based brain-computer interface stimulus presentation paradigm: Moving beyond rows and columns. Clin. Neurophysiol. 121, 1109–1120.

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Wolpaw, J., Birbaumer, N., Heetderks, W., McFarland, D., Peckham, P., Schalk, G., Donchin, E., Quatrano, L., Robinson, C., and Vaughan, T. (2000). Brain-Computer interface technology: a review of the first International Meeting. IEEE Trans. Rehab. Eng. 8, 164–173.

Wolpaw, J. R., and McFarland, D. J. (2004). Control of a two-dimensional movement signal by a noninvasive brain-computer interface in humans. Proc. Natl. Acad. Sci. U.S.A. 101, 17849–17854.

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text