Artemiy Berkmush-Antipova1*
Nikolay Syrov1
Lev Yakovlev1
Andrei Miroshnikov1,2
Frol Golovanov1
Natalia Shusharina1
Alexander Kaplan1,2- 1Baltic Center for Neurotechnology and Artificial Intelligence, Immanuel Kant Baltic Federal University, Kaliningrad, Russia
- 2Laboratory for Neurophysiology and Neuro-Computer Interfaces, Department of Human and Animal Physiology, Faculty of Biology, Lomonosov Moscow State University, Moscow, Russia
Error-related potentials (ErrPs) have attracted attention in part because of their practical potential for building brain-computer interface (BCI) paradigms. BCIs, facilitating direct communication between the brain and machines, hold great promise for brain-AI interaction. Therefore, a comprehensive understanding of ErrPs is crucial to ensure reliable BCI outcomes. In this study, we investigated ErrPs in the context of the “guess what I am thinking” paradigm. 23 healthy participants were instructed to imagine an object from a predetermined set, while an algorithm randomly selected another object that was either the same as or different from the imagined object. We recorded and analyzed the participants’ EEG activity to capture their mental responses to the algorithm’s “predictions”. The study identified components distinguishing correct from incorrect responses. It discusses their nature and how they differ from ErrPs extensively studied in other BCI paradigms. We observed pronounced variations in the shape of ErrPs across different stimulus sets, underscoring the significant influence of visual stimulus appearance on ErrP peaks. These findings have implications for designing effective BCI systems, especially considering the less conventional BCI paradigm employed. They emphasize the necessity of accounting for stimulus factors in BCI development.
1 Introduction
Non-invasive brain-computer interfaces (BCIs) translate user intentions, detected as changes in brain activity (e.g., electroencephalography (EEG) signal), directly into control signals for external devices without requiring muscle activity (Wolpaw et al., 2002; Abiri et al., 2019; Wolpaw et al., 2020). In this approach, the user’s mental effort generates specific patterns in brain signals that are recorded, classified, and translated into actionable commands (Wolpaw et al., 2020). Despite significant advances in BCIs, only a few mental strategies have been demonstrated to produce detectable brain signal perturbations for effective control (Abiri et al., 2019).
In addition, BCIs face certain limitations that lead to errors in recognizing user intentions. These errors can be caused by external factors, such as noise signals, insufficient spatial resolution of the EEG, and suboptimal feature extraction and classification algorithms. Internal factors, such as low level of attention and difficulties in performing mental tasks, also contribute to these errors (Moore, 2003; Zhang et al., 2021). To address this challenge, researchers have incorporated verification procedures that monitor the user’s mental responses to BCI output. This includes the detection of specific EEG responses, namely error-related potentials (ErrPs) or feedback-related potentials, which are neural signatures indicating mental agreement/disagreement with the presented BCI output (Chavarriaga et al., 2014). Detection of error-related activity makes it possible to correct the BCI feedback, thereby improving BCI performance (Chavarriaga et al., 2014; Yasemin et al., 2023).
Although ErrPs are useful for improving BCI performance, validation mechanisms have their own limitations. First, the reliance on decoding mental responses to validate BCI feedback significantly increases the cognitive load, resulting in a more complicated interaction paradigm (Plass-Oude Bos et al., 2011; Lotte, 2012; Ke et al., 2015). Users must shift their focus between different tasks, such as visualizing movements and the task of cognitive confirmation of BCI output. In addition, the current BCI paradigm often exhibits a mismatch between the intended mental commands and the actual output of the BCI system (Wang et al., 2022). For example, scenarios where users control robot movements with a P300-based BCI by mentally counting visual stimuli flashes (Gillini et al., 2022) or by imagining tongue movements (Guo et al., 2020) highlight this discrepancy. Such discrepancies demonstrate the lack of naturalness in brain-machine interaction.
To address these limitations, a new approach similar to reinforcement learning has been proposed in recent studies (Ferrez and Millán, 2005; Choi and Kim, 2019; Batzianoulis et al., 2021). In this paradigm, a user observes a robot’s behavior and controls it through mental approval or disapproval, which is detected by analyzing ErrPs. Consequently, this real-time feedback iteratively improves the robot’s performance. Unlike classical approaches that measure brain signals directly related to mental intent, in this new scenario, the computer presents various output options (e.g., different robot motion trajectories) and analyzes the neural responses to verify the validity of these potential outcomes (Batzianoulis et al., 2021). The validation process aims to determine whether the user agrees or disagrees with a suggested behavior of the machine or AI agent. If there is a conflict, the behavior is adjusted and the new option is tested. This approach is believed to provide a more natural user experience, similar to a cooperative interaction, through the principle of suggestion-confirmation/rejection (Ferrez and Millán, 2008; Omedes et al., 2018; Choi and Kim, 2019).
Building on this, we introduce a paradigm similar to the game “guess what I am thinking.” In this paradigm, the user imagines an object or a scene, and a second player (here represented by a machine algorithm) tries to guess the object by making assumptions. ErrPs are used as signatures of mental cues to confirm or reject the algorithm’s hypotheses. It is known that error-related EEG perturbations demonstrate high variability, which varies not only between individuals but also depending on the experimental paradigm (Iturrate et al., 2014; Abu-Alqumsan et al., 2017), error frequency and BCI outcome significance (Holroyd and Coles, 2002; LoTemplio et al., 2023). The emotionality of the presenting stimuli affects the shape of ErrPs (Larson et al., 2006). Furthermore, it is well studied that cortical responses are influenced by context within which stimuli are presented (Wlotko and Federmeier, 2012), the presence of faces and emotions they express (Bötzel and Grüsser, 1989), and the aesthetic qualities of stimuli (de Tommaso et al., 2008). In light of the above considerations, we propose that the appearance of the stimulus may influence the responses evoked by correct and incorrect feedback in BCI. In this study, we aimed to analyze the spatiotemporal characteristics of ErrPs in this proposed collaborative paradigm.
In this study, we asked participants to play a game with a computer in which they imagined a particular picture from a set, and the machine, as explained to the participants, would guess the imagined picture by presenting it on a screen. Observing the machine’s response was expected to elicit specific EEG signatures of mental agreement or disagreement in the participants. Given the wide range of scenes and objects that can be imagined, we used four different visual stimulus sets with different appearance and semantics (objects, animals, emojis, and numbers). Our primary goal was to investigate the consistency of the shape of the ErrPs across these different visual stimuli when presented as a machine response in a BCI loop. We hypothesized that the shape of ErrPs would be affected by the distinct visual semantic content of the stimuli, rather than remaining invariant, despite evidence suggesting that ErrPs are relatively conservative in shape for different feedback types (Iwane et al., 2022). We expected ErrPs in the studied paradigm would not only be shaped by components related to the processing of observed errors (Ferrez and Millán, 2005; Xavier Fidêncio et al., 2022; Syrov et al., 2023) but also by the processing of the content of the visual stimuli themselves. Also, our paradigm presents not a homogeneous N-back (namely 0-back) type problem, but a constantly updating one, which is more similar to the actual interaction with BCI.
The results of this study may serve as a foundation for further advancement in the development of collaborative BCIs and significantly enhance our understanding of how feedback content affects ErrP shapes. It is crucial for researchers to recognize that, in addition to categorizing feedback as correct or incorrect, the content of stimuli may also influence ErrP shapes. Consequently, ErrP-based feedback validation procedures may be enhanced by considering the content and semantics of the stimuli.
2 Materials and methods
2.1 Participants and ethical considerations
A total of 23 healthy volunteers (11 males, mean age = 23 ± 4.5 years, right-handed) were recruited for this study. Eligibility criteria included the absence of self-reported neurological disorders, injuries, or other impairments of the nervous system. Detailed information about the research protocol was provided to all participants, and informed consent was obtained prior to participation in the study. The experimental design and procedures were reviewed and approved by the Ethics Committee of the Immanuel Kant Baltic Federal University.
2.2 Experimental design
To investigate ErrPs in the collaborative paradigm, the fake BCI paradigm was chosen, as its validity for studying human-machine interaction was detailed by Lynn et al. (2010), Évain et al. (2016), Lopez-Sola et al. (2021). Following the approach of Lynn et al. (2010), participants underwent specific procedures to enhance their belief in working with a genuine BCI. All participants were novices to BCI. Prior to experimental sessions, during EEG cap preparation, participants dedicated 40 min to understanding the principles of BCI technology, capable of decoding mental states through brain signal processing. Participants were also informed they would use similar technology during the experiments, fostering confidence in working with a BCI predicting their imagined images.
There were four consecutive runs for each participant, with 42 trials in each run. Each trial in the experimental session comprised three stages: (1) Memorization period, the first presentation of a particular visual stimulus which participants were instructed to memorize; (2) Imagination period during which participants were encouraged to mentally reconstruct the image from the stimulus; (3) Feedback, the presentation of a visual stimulus that, as it has been instructed to the participants, is the result of a computer prediction of what picture the participants have imagined. The memorization period took 2 s, after which the stimulus image gradually disappeared for 2.2 s until it disappeared completely and the fixation cross appeared in its place. This lasted for 4.5 s (totaling the 6.7 s imagination period) until feedback. The feedback stimulus was presented for 3 s, then participants could advance to the next trial by pressing space on a keyboard. This was a self-paced approach that allowed participants to control the pace of the study by pressing a button to advance to the next trial start. If the button was not pressed, the next trial started automatically 15 s after the end of the previous trial. Feedback stimuli could be identical to the initially presented stimuli or different. During feedback, participants assessed whether it matched the initial stimulus, mentally agreeing or disagreeing with the computer’s “prediction”. The number of correct (mental response Yes) and incorrect (mental response No) feedback events was predetermined and amounted to 24 and 18, respectively, resulting in an accuracy of 57.1% (total number of trials n = 42). The order of trials with correct and incorrect feedback was randomized for each run. The fake-BCI paradigm aimed to boost participant’s motivation. A visual representation of the experimental session design is presented in Figure 1.
Figure 1. Experimental design. The gradient green bar illustrates the stimulus presentation sequence used in the experimental session. Initially, the first stimulus (e.g. a duck) was displayed for 2 s (memorization period). Then, over the next 6.7 s, the first stimuli gradually dissolved into a fixation cross (imagination period). After a 6.7 s interval from the onset of the first stimulus, the feedback was presented for 3 s. If the feedback image matched the initial stimulus (indicating a Yes response), participants were expected to mentally agree with the feedback presented. Conversely, if there was no match (indicating a No response), participants were instructed to mentally disagree with the feedback. After the feedback presentation, participants were given control to continue the experimental procedure by pressing a button on the keyboard (self-paced design).
2.2.1 Stimulus sets
The study used four different stimulus sets, each containing different types of visual stimuli. Each stimulus set was designed to represent different semantic domains: set#1 contained images of various everyday objects and tools (referred to as objects), set #2 consisted of animal emojis (referred to as animals), set#3 contained numbers (referred to as numbers), and set#4 contained emojis depicting different facial expressions (referred to as emojis). Each set contained seven unique images with a black background. Further details on the composition of the stimulus sets can be found in Appendix 1.
2.3 EEG data acquisition
EEG data were recorded using a BrainVision actiCHamp+ amplifier (Brain Products GmbH, Germany) with 64 active channels. The sampling rate was set to 1,000 Hz. The placement of the active Ag/AgCl sensors followed the international 10/10 electrodes placement system. The TP10 channel position was used for the reference. To ensure optimal signal quality, the impedance between the electrodes and the skin was kept below 10 kΩ. During recording, a photo sensor was used to accurately mark the onset of stimulus presentation. A custom written Python script was used for stimulus presentation.
2.4 Data preprocessing and analysis
The raw EEG signal was downsampled to 500 Hz, then band-pass filtered in the 1–15 Hz range using a causal finite impulse response (FIR) filter. Interpolation of noisy channels was performed using the spherical spline method (Perrin et al., 1989). Independent Component Analysis (fastICA method) was applied to remove oculographic artifacts: the components highly correlated with Fp1 and Fp2 signals were excluded from the recording. The signal was then re-referenced using the common average reference (CAR) (McFarland et al., 1997). The preprocessed signal was then segmented into 900 ms epochs based on the photo sensor triggers. Each epoch started 100 ms before and ended 800 ms after the onset of the feedback stimulus. A baseline correction was applied using the time interval (−100; 0 ms).
In order to obtain ERPs related to mental (dis-)agreement, the epochs for each stimulus set were averaged separately according to the correctness of the feedback stimulus. In accordance with previous studies on ErrPs, which have localized the signal sources of these potentials (Falkenstein et al., 2000), we averaged epochs across the fronto-central channel group (F1, Fz, F2, FC1, FCz, FC2, C1, Cz, C2). This also allowed us to improve the signal-to-noise ratio.
Peak amplitudes were calculated using the following criteria: each peak was defined as the extreme value within a given latency range, and the mean peak value was taken within ±20 time samples around this value. Since we detected three pronounced peaks in No responses – an early positive peak and two peaks related to error-related negativity (ERN) and error-related positivity (Pe) – we used the following latency intervals for peak amplitude estimation: the maximal value for the early positive peak was found in the 140–260 ms range, the minimal value for the ERN peak was found in the 200–340 ms range, and the maximal value for the Pe peak was found in the 340–540 ms range, all in accordance with the stimulus onset. To obtain the corresponding values for Yes responses, the same latencies estimated from No responses were taken.
2.5 Statistical analysis
Before assessing the effects of feedback correctness and stimulus set on distinct components, we analyzed the distribution of peak amplitude values for all three components. Since the data did not follow a normal distribution according to the Shapiro–Wilk tests, we used the Friedman test to identify the influence of the stimulus set factor on the amplitude of all identified ERP components in Yes and No responses.
Subsequently, peak amplitudes for all three components were compared between Yes- and No-trials for each stimulus set, separately, utilizing the Wilcoxon signed-rank test. The Bonferroni correction was applied by multiplying the p-values by the number of paired comparisons performed.
To comprehensively explore the spatio-temporal characteristics of components in feedback-related responses and determine whether their appearance was significantly related to Yes/No categorization rather than the processing of stimulus content, we subtracted ERPs elicited by stimuli from the memorization period from the fake-feedback evoked activity. A spatio-temporal cluster-based permutation test with 10,000 permutations (Maris and Oostenveld, 2007) was employed to assess the result of the subtraction with zero. This statistical analysis allowed us to identify clusters related to components specifically associated with participants’ mental agreement with computer guesses.
For the EEG data processing and statistical analysis, we used self-written Python (v3.10) scripts utilizing the libraries MNE v1.5.0 (Gramfort et al., 2013), NumPy v1.26.0 (Harris et al., 2020), and SciPy v1.11.0. (Virtanen et al., 2020).
3 Results
Analysis of ERPs related to Yes and No responses revealed distinct peaks between the two reactions, including the early positive peak, the negative ERN peak, and the late positive Pe peak. Figure 2A shows the curves for both Yes and No for all stimulus sets.
Figure 2. (A) Yes (on the left) and No (on the right) responses for each stimuli type objects, animals, numbers and emojis (orange, blue, purple and green lines, respectively). For No responses, early peak, ERN and Pe peaks are allocated in a group of channels (F1, Fz, F2, FC1, FCz, FC2, C1, Cz, C2) electrodes. (B) Grand average (N = 23) ErrPs for every stimulus set in a group of channels (F1, Fz, F2, FC1, FCz, FC2, C1, Cz, C2). Orange solid line represents Yes responses, blue solid line – No responses, black solid line – difference curve.
The results of the Friedman tests are presented in Table 1. The results demonstrated a significant effect of the stimulus set factor on the amplitude of the ERN component for No responses and on the amplitude of the Pe component for Yes responses.
Table 1. Friedman tests results for the Yes and No responses between different stimuli for each ERP component (early peak, ERN, Pe).
Further, we compared the amplitudes of the revealed components between Yes and No responses separately for each set of stimuli.
Figure 2B shows the ERP curves for Yes and No responses for different stimulus sets, averaged over the group of channels. Detailed results for paired comparisons of ERP peaks for Yes and No responses are shown in Figure 3.
Figure 3. Comparisons between the peak amplitudes of ERP components for No (blue) and Yes (orange) responses across different stimulus sets. Horizontal lines within the boxes correspond to the median values, boxes – to the interquartile range and [Q1-1.5*IQR; Q3 + 1.5*IQR] range is shown by whiskers. Black circles represent mean values for the group. Colored dots represent individual values. Corrected p-values are shown for the Wilcoxon signed-rank test with Bonferroni correction.
For the objects the most prominent ErrP peak was the ERN peak (W = 14; p = 0.00008), which appeared on average at around 266 ms, and the Pe peak (W = 18; p = 0.00018), which appeared at around 386 ms. For the animals, the most prominent ErrP peaks were also the ERN peak (W = 12; p = 0.0002) and the Pe peak (W = 29; p = 0.0048), appearing at approximately 304 ms and 402 ms, respectively. For the numbers, the most prominent peaks were the early peak (W = 10; p = 0.00197) and the Pe peak (W = 4; p = 0.00032), appearing at approximately 192 ms and 406 ms, respectively. For the emojis all peaks were prominent, although the early positive (W = 24; p = 0.00851) and ERN peaks (W = 12; p = 0.0008) were more pronounced compared to the Pe peak (W = 32; p = 0.02836), which appeared at approximately 178 ms (early positive peak), 308 ms (ERN) and 428 ms (Pe).
Thus, the early positive and ERN peaks were not robust to differences in stimulus material in the paradigm used, although the ERN occurred for 3 out of 4 stimulus sets (objects, animals, emojis) and the early positive peak occurred for the numbers and emojis. Although the significance levels for the Pe varied depending on the type of visual stimuli, it was the most stable component of the ErrP (Figure 2B).
Figure 4 depicts the distinctions between No-ERPs and Yes-ERPs for each stimulus set, including the average difference at electrode FCz (Figure 4A). The topographies presented illustrate the signal distribution for each identified peak (Figure 4B). The early positive peak was most prominent in the fronto-central channels (Figure 4B, leftmost topographic projection), with an average latency of 185 ms from stimulus onset. The amplitude and latency of the early positive peak vary slightly depending on the stimulus set, with the largest differences observed for the numbers, as noted during the analysis of ErrPs in individual sets. The ERN peak showed a fronto-central spatial distribution with a slight right lateralization (Figure 4B, topographic projection in the middle). This peak was usually observed approximately 300 ms after stimulus onset. The shape of the ERN peak remained consistent across the three stimulus sets but tended to show a delayed onset and reduced amplitude for the numbers. The spatial distribution of the Pe (Figure 4B, rightmost topographic projection) demonstrates a tendency to appear in fronto-central channels, similar to other components. On average, this peak was observed approximately 400 ms after the feedback presentation.
Figure 4. (A) Grand average (N = 23) of ErrPs (data presented for FCz channel). Color lines represent single stimulus sets, the black solid line corresponds to average value across all stimulus sets. Shaded area corresponds to the peaks’ time ranges used for statistical analysis and topographic mapping. (B) Topographic localization for three peaks observed on the ErrP curve.
In Figure 5A, Yes and No responses are juxtaposed with ErrPs triggered by simple picture presentation during the memorization period. The figure exhibits a notable overlap between No-ERPs and ERPs associated with stimuli from memorization trials. To scrutinize the impact of components related to mere image content processing on ERPs associated with Yes/No categorization during computer response perception, we conducted a subtraction of memorization-related ERPs from those related to Yes and No responses (Figure 5B). This method allowed us to pinpoint peaks specifically linked to error and correct outcomes processing by eliminating potentially contaminating components. For statistical validation of the resulting peaks on the subtracted curves, a non-parametric spatio-temporal permutation test was employed.
Figure 5. Differences between Yes/No-related ERPs and ERPs evoked by stimuli presented in memorization trials. (A) ERP curves from Fz and Cz channels for all stimulus sets. (B) Result of subtracting memorization-related ERPs (Memo ERPs) from Yes/No-related ERPs. ERPs from the Cz channel. Blue lines represent ERPs elicited during memorization trials. (C) Results of the spatio-temporal permutation test for three out of four sets (objects, animals, emojis). On the left side of each panel, the topographic distribution of the F-statistic averaged over the time interval associated with the significant cluster; white circles indicate channels belonging to the cluster. On the right side, curves obtained by subtracting the memorization-related ERP from the Yes responses, averaged over all channels in the cluster. The shaded areas indicate the time interval during which significant differences were found.
The results of the Friedman tests revealed an effect of the stimulus set factor on the ERN peak amplitude in No responses, and late positive component peak amplitude in Yes responses. Its instability was also confirmed by analyzing the results of Wilcoxon signed-rank tests for each individual stimulus set. Our findings indicate that only Yes responses exhibited significant differences from the ERPs of memorization trials. Conversely, No responses showed no significant clusters according to permutation testing. Spatio-temporal clusters derived for the Yes responses reveal two peaks: a slight negative deflection with a latency of approximately 200 ms after the stimulus onset, succeeded by a positive peak within the time interval corresponding to the ERN deflection on the ErrP curve (~250–400 ms after the stimulus onset). These peaks exhibit similar shapes and spatial characteristics across all stimulus sets, underscoring their specificity for processing correct feedback. Interestingly, no clusters were found for the numbers, consistent with the findings that this stimulus set did not show significant differences between Yes and No responses within the time window related to the ERN peak.
4 Discussion
While ongoing endeavors to leverage ErrPs for BCI control have predominantly concentrated on enhancing spelling technologies relying on P300 and other evoked potentials (Dal Seno et al., 2010; Cruz et al., 2017), an equally noteworthy alternative is to consider ErrPs as the primary control EEG signal (Batzianoulis et al., 2021). Through establishing a framework of continuous interaction between the user and the “AI agent” via a suggestion-confirmation/rejection dialogue, our fake-BCI paradigm strived to recreate a real BCI user experience. Participants were tasked with memorizing a visual stimulus and providing a mental Yes or No response based on whether the computer-presented feedback stimulus matched the memorized one. In line with existing literature, our study confirmed that incorrect stimuli induced error-related ERPs characterized by a negative peak around 300 ms, followed by a subsequent positive deflection (Falkenstein et al., 2000; Glazer et al., 2018).
An early positive peak with a latency of approximately 150–250 ms was evident across all stimulus sets in both Yes and No responses. However, the numbers and emojis stimulus sets showed significant differences between Yes and No responses in this component. This may be due to differences in detail and color. We interpret numbers and emojis as simple, mostly monochromatic stimuli with few details, and according to a previous study by Michie et al. (1999), less detailed stimuli are easier to process and integrate. Our results suggest that visual attention systems are more responsive to simple stimuli in the BCI framework. Consequently, the task of comparing two visually simple stimuli can elicit signatures of image comparison even in the early stages of stimuli integration.
The negative peak observed in the No responses around 300 ms after the feedback stimulus, classified as the ERN, mainly manifested in the fronto-central group of channels. The spatial distribution and latency of this peak align with findings from previous neuroimaging studies focused on error processing (Gehring et al., 1993; Gehring and Fencsik, 2001), which associated this activity with detecting discrepancies between expected and actual outcomes (Bellebaum and Daum, 2008; Glazer et al., 2018). In our study, the amplitude of this negative deflection was greater in No-trials, whereas Yes-trials exhibited a positive deflection within the same latency range. As a result, the difference curve obtained by subtracting Yes-trials from No-trials revealed a distinct negative peak in the ErrP curve (Figure 3). This ErrP profile has been observed in paradigms involving both user behavioral errors and BCI errors (Ferrez and Millán, 2005). The ERN is known to be sensitive to various factors, including error probability (Ganushchak and Schiller, 2008) subject motivation, task salience, and the perceived cost of error (Lynn et al., 2010). For instance, Hajcak et al. (2005) and Rawls et al. (2020) demonstrated that high motivation conditions lead to increased amplitude and latency of the ERN compared to low motivation conditions, particularly for high-value trials. Additionally, both the ERN and the feedback-related negativity (FRN) show larger amplitudes with increasing reward size (Hajcak et al., 2005), irrespective of whether the reward is positive or negative (Gehring and Fencsik, 2001).
Several studies have reported larger ERN amplitudes when semantically similar stimuli were presented (Herrmann et al., 2004; Rawls et al., 2020), indicating the influence of stimulus characteristics and semantic salience on ErrP shape. In this study, we delved more deeply into this field and demonstrated that distinct stimuli exert a pronounced influence on the shape of ErrP. The observed significant effects of the stimulus set factor on the ERN in No responses suggest that the semantic characteristics of the visual stimuli may have an influence on this component. This influence may be attributed to several factors. First, Falkenstein et al. (2000) and Coles et al. (2001) proposed that the ERN is related to the processes of comparing two stimuli and their underlying representations. Our findings may indicate distinct processing of semantically diverse stimuli, meaning that for different feedback stimuli, outcome evaluation is associated with retrieving visually and semantically different images from memory. Secondly, prior research has emphasized the role of emotional content in stimulus-induced ErrPs (Wlotko and Federmeier, 2012), suggesting that emotionally salient stimuli could influence error processing and corresponding EEG responses. In contrast, the Friedman test revealed that the amplitude in Yes responses within the ERN latency was not affected by the stimulus set factor, probably because the neural processes of verification in Yes responses are less influenced by the stimulus content or its valence.
Following the ERN, we observed a positive peak with a latency of approximately 400–500 ms. We interpret this peak as the Pe, which is known to be larger for error events and associated with error awareness and subsequent behavioral correction (Endrass et al., 2007; Glazer et al., 2018; Pyasik et al., 2022). Typically, the Pe can be divided into early (frontally localized) and late (posterior) components, with the latter primarily associated with behavioral adjustments (Pyasik et al., 2022). Although we did not specifically focus on the separation of Pe components in our study, the fronto-central spatial distribution of the observed Pe suggests that the early Pe component predominated in our paradigm.
Our analysis demonstrated that the Pe component exhibited a distinct pattern in No responses. However, it did not show changes related to the stimulus set factor. In contrast, the positive components of the same latency in Yes responses demonstrated significant variability influenced by this factor. This suggests that ERN is more reactive to the visual content of erroneous stimuli, while Pe in No responses could reflect higher-level processes of integration, acceptance, and adaptation of error consequences (Ullsperger et al., 2014; Iwane et al., 2022).
Regarding the positive component with a latency range corresponding to Pe in Yes responses, we attribute this potential to activity similar to the feedback-related correct response positivity described by Cockburn and Holroyd (2018). These authors associate this potential with reward prediction error (RPE) systems, which are strongly related to the level of attention and salience of feedback stimuli (Walsh and Anderson, 2012). The observed effects of stimulus set factor on this late positive component in Yes-trials may be explained by differences in attractiveness of the different visual stimuli used. However, since our paradigm did not include explicit reward stimuli, we discuss this finding and its association with RPE with caution and acknowledge its speculative nature.
It is interesting that within the same experimental paradigm, some stimulus sets demonstrated the presence of both ERN and Pe (objects, animals, emojis), whereas in the numbers only Pe component was present. Our findings suggest that the presence of ERN is not a mandatory indicator of error awareness, aligning with a previous review (Xavier Fidêncio et al., 2022) reporting variable differences in the ERN and Pe across individuals. Some participants exhibited a virtually absent Pe while the ERN was prominent. Enriching the evidence for intersubject variability of Pe and ERN (Xavier Fidêncio et al., 2022), we described in this study that the frequency of error-related components in ERP curves can vary depending on the appearance of the stimuli used. It is important to emphasize, especially for future practical applications, that the observed differences in ErrPs across different stimulus sets make it challenging to effectively transfer classification algorithms for ErrP detection into a BCI loop: algorithms trained on a specific stimulus set may not generalize well to other stimulus sets.
Notably, we did not find ERN components within No responses for the numbers, which was unexpected given the clear distinctions between correct and incorrect stimuli within this set. Previous studies (Herrmann et al., 2004; Hajcak et al., 2005) have shown that the amplitude of ErrPs tends to increase as the error approaches correct performance. Therefore, we initially expected to find pronounced ErrPs for the numbers. This lack of pronounced disagreement reactions may be due to the short semantic distance between numbers, causing even erroneous numbers to be perceived as less distinct and thus not eliciting strong disagreement reactions.
Differences in the subjective attractiveness of stimuli across sets, influencing participant motivation, could explain the varied prominence of ErrP components. For example, participants may have found certain sets, such as the objects, more appealing and easier to visualize during the memorization interval than the numbers. However, since we did not conduct any surveys or assessments regarding the subjective attractiveness of the stimulus sets, we cannot provide direct evidence for this hypothesis.
It’s worth considering the impact of semantic connections among pictures within each stimulus set on the observed ErrPs. Specifically, errors in the numbers might be semantically closer to the correct answers compared to the erroneous stimuli in the objects and animals. Previous studies have reported a relationship between the amplitude of the ERN and the subjective distance between the error and the correct output (Hajcak et al., 2005; Hill, 2009; Spinelli et al., 2018; Iwane et al., 2022).
Our hypothesis posited that the shape of ErrPs could be influenced by the content of the stimuli, contributing to differences in ErrPs across the stimulus sets in our study. The objective of this study was to determine the extent to which the components associated with the object of attention modulate neural responses of agreement and disagreement. It was hypothesized that the ErrP waveform represents a mixture of picture-related visual potentials and Yes/No-related components.
To thoroughly investigate this hypothesis, we performed an additional analysis by subtracting the ERPs obtained during the memorization period from the Yes-/No-ERPs (Figure 5). The results show a high degree of similarity between the memorization ERPs and the Yes-/No-ERPs: during the ERN latency, both the memorization ERPs and the No-related ERPs show a negative deflection, although the No-related peak is more pronounced. A similar trend is observed for Pe, with the positive peak in this latency range also present in the ERPs recorded in memorization trials. At the same time, the Yes-ERPs differ significantly from the memorization-related responses. Permutation cluster-based tests indicated significant differences between Yes- and memorization-related ERPs within the time range where the ERN was described in No responses. On the other hand, No-ERPs did not show significant differences from memorization-ERPs. This suggests that the cortical responses in No reactions are highly modulated by responses to stimulus content. These findings align with the results of the Friedman test, which indicate that the amplitude of the ERN varies depending on the specific stimuli presented. The subtraction of memorization-ERPs from Yes-ERPs revealed a consistent pattern of negative and positive peaks in all stimulus sets. These peaks were most pronounced in the fronto-central area. The negative component in correct trials we found may be similar to previously described ERN-like activity, as identified by Coles et al. (2001) in correct trials. Coles et al. raised questions about the validity of theories specifically associating the ERN with error processing. The study proposed that output processing may be more strongly elicited by correct trials, potentially contaminating the ERN deflection in averaged ErrP. Our finding is also in line with the theory that the ERN-like activity primarily reflects a comparison between the representation of the correct answer and the actual output process, rather than solely being an indicator of error detection resulting from this process (Cruz et al., 2017). Given that this comparison process is also necessary for correct responses, it is plausible that the ERN is present in correct responses as well.
5 Conclusion
We want to highlight that previous studies have concentrated on error-related responses, whereas our findings indicate that there is also a need for further investigation into responses to correct inputs. The verification process is also conducted in correct trials. The results revealed that Yes-related potentials had components related to outcome processing that were invariant to stimulus content. This suggests that correct trials may provide insight into error processing cortical networks and could be highly valuable for practical application. Our results may be useful in the development of generalized classifications for ErrP detection in a zero-calibration approach across different paradigms (Cruz et al., 2018, 2022; Iwane et al., 2022), where the effects of stimulus scene on ErrPs should be taken into account.
Data availability statement
The data supporting the conclusions of this article will be made available by the authors, without undue reservation.
Ethics statement
The studies involving humans were approved by Institutional Ethics Committee of Immanuel Kant Baltic Federal University (protocol code № 40, 26.06.2023) for studies involving humans. The studies were conducted in accordance with the local legislation and institutional requirements. The participants provided their written informed consent to participate in this study.
Author contributions
AB-A: Formal analysis, Investigation, Visualization, Writing – original draft. NiS: Data curation, Investigation, Methodology, Software, Visualization, Writing – original draft, Writing – review & editing. LY: Data curation, Investigation, Methodology, Writing – original draft, Writing – review & editing. AM: Formal analysis, Investigation, Writing – original draft. FG: Investigation, Writing – original draft. NaS: Funding acquisition, Project administration, Resources, Writing – original draft. AK: Conceptualization, Funding acquisition, Supervision, Writing – original draft, Writing – review & editing.
Funding
The author(s) declare that financial support was received for the research, authorship, and/or publication of this article. The study was carried out within the framework of State Assignment of the Ministry of Education and Science of the Russian Federation FZWM-2024-0013 and the Priority 2030 Program.
Conflict of interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Publisher’s note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
References
Abiri, R., Borhani, S., Sellers, E. W., Jiang, Y., and Zhao, X. (2019). A comprehensive review of EEG-based brain–computer interface paradigms. J. Neural Eng. 16:011001. doi: 10.1088/1741-2552/aaf12e
Abu-Alqumsan, M., Kapeller, C., Hintermüller, C., Guger, C., and Peer, A. (2017). Invariance and variability in interaction error-related potentials and their consequences for classification. J. Neural Eng. 14:066015. doi: 10.1088/1741-2552/aa8416
Batzianoulis, I., Iwane, F., Wei, S., Correia, C. G. P. R., Chavarriaga, R., Millán, J. D. R., et al. (2021). Customizing skills for assistive robotic manipulators, an inverse reinforcement learning approach with error-related potentials. Commun. Biol. 4:1406. doi: 10.1038/s42003-021-02891-8
Bellebaum, C., and Daum, I. (2008). Learning-related changes in reward expectancy are reflected in the feedback-related negativity. Eur. J. Neurosci. 27, 1823–1835. doi: 10.1111/j.1460-9568.2008.06138.x
Bötzel, K., and Grüsser, O. J. (1989). Electric brain potentials evoked by pictures of faces and non-faces: a search for "face-specific" EEG-potentials. Exp. Brain Res. 77, 349–360. doi: 10.1007/BF00274992
Chavarriaga, R., Sobolewski, A., and Millán, J. D. R. (2014). Errare machinale Est: the use of error-related potentials in brain-machine interfaces. Front. Neurosci. 8:208. doi: 10.3389/fnins.2014.00208
Choi, J. W., and Kim, K. H. (2019). Covert intention to answer "yes" or "no" can be decoded from single-trial electroencephalograms (EEGs). Computational Intelligence and Neuroscience, 2019. Available at: https://hdl.handle.net/10125/103127
Cockburn, J., and Holroyd, C. B. (2018). Feedback information and the reward positivity. Int. J. Psychophysiol. 132, 243–251. doi: 10.1016/j.ijpsycho.2017.11.017
Coles, M. G., Scheffers, M. K., and Holroyd, C. B. (2001). Why is there an ERN/ne on correct trials? Response representations, stimulus-related components, and the theory of error-processing. Biol. Psychol. 56, 173–189. doi: 10.1016/S0301-0511(01)00076-X
Cruz, A., Pires, G., and Nunes, U. J. (2017). Double ErrP detection for automatic error correction in an ERP-based BCI speller. IEEE Trans. Neural Syst. Rehabil. Eng. 26, 26–36. doi: 10.1109/TNSRE.2017.2755018
Cruz, A., Pires, G., and Nunes, U. J. (2018). Generalization of ErrP-calibration for different error-rates in P300-based BCIs. 2018 IEEE international conference on systems, man, and cybernetics (SMC) (pp. 644–649).
Cruz, A., Pires, G., and Nunes, U. J. (2022). Spatial filtering based on Riemannian distance to improve the generalization of ErrP classification. Neurocomputing 470, 236–246. doi: 10.1016/j.neucom.2021.10.078
Dal Seno, B., Matteucci, M., and Mainardi, L. (2010). Online detection of P300 and error potentials in a BCI speller. Comput. Intell. Neurosci. 2010, 1–5. doi: 10.1155/2010/307254
de Tommaso, M., Pecoraro, C., Sardaro, M., Serpino, C., Lancioni, G., and Livrea, P. (2008). Influence of aesthetic perception on visual event-related potentials. Conscious. Cogn. 17, 933–945. doi: 10.1016/j.concog.2007.09.003
Endrass, T., Reuter, B., and Kathmann, N. (2007). ERP correlates of conscious error recognition: aware and unaware errors in an antisaccade task. Eur. J. Neurosci. 26, 1714–1720. doi: 10.1111/j.1460-9568.2007.05785.x
Évain, A., Argelaguet, F., Strock, A., Roussel, N., Casiez, G., and Lécuyer, A. (2016). Influence of error rate on frustration of BCI users. In Proceedings of the international working conference on advanced visual interfaces (pp. 248–251).
Falkenstein, M., Hoormann, J., Christ, S., and Hohnsbein, J. (2000). ERP components on reaction errors and their functional significance: a tutorial. Biol. Psychol. 51, 87–107. doi: 10.1016/S0301-0511(99)00031-9
Ferrez, P. W., and Millán, J. D. R. (2005). You are wrong!---automatic detection of interaction errors from brain waves. In Proceedings of the 19th international joint conference on artificial intelligence (No. Conf).
Ferrez, P. W., and Millán, J. D. R. (2008). Error-related EEG potentials generated during simulated brain–computer interaction. IEEE Trans. Biomed. Eng. 55, 923–929. doi: 10.1109/TBME.2007.908083
Ganushchak, L. Y., and Schiller, N. O. (2008). Motivation and semantic context affect brain error-monitoring activity: an event-related brain potentials study. NeuroImage 39, 395–405. doi: 10.1016/j.neuroimage.2007.09.001
Gehring, W. J., and Fencsik, D. E. (2001). Functions of the medial frontal cortex in the processing of conflict and errors. J. Neurosci. 21, 9430–9437. doi: 10.1523/JNEUROSCI.21-23-09430.2001
Gehring, W. J., Goss, B., Coles, M. G., Meyer, D. E., and Donchin, E. (1993). A neural system for error detection and compensation. Psychol. Sci. 4, 385–390. doi: 10.1111/j.1467-9280.1993.tb00586.x
Gillini, G., Di Lillo, P., Arrichiello, F., Di Vito, D., Marino, A., Antonelli, G., et al. (2022). A dual-arm mobile robot system performing assistive tasks operated via p300-based brain computer interface. Industrial Robot 49, 11–20. doi: 10.1108/IR-07-2020-0137
Glazer, J. E., Kelley, N. J., Pornpattananangkul, N., Mittal, V. A., and Nusslock, R. (2018). Beyond the FRN: broadening the time-course of EEG and ERP components implicated in reward processing. Int. J. Psychophysiol. 132, 184–202. doi: 10.1016/j.ijpsycho.2018.02.002
Gramfort, A., Luessi, M., Larson, E., Engemann, D. A., Strohmeier, D., Brodbeck, C., et al. (2013). MEG and EEG data analysis with MNE-python. Front. Neurosci. 7:70133. doi: 10.3389/fnins.2013.00267
Guo, Y., Wang, M., Zheng, T., Li, Y., Wang, P., and Qin, X. (2020). NAO robot limb control method based on motor imagery EEG. 2020 international symposium on computer, consumer and control (IS3C) (pp. 12123–12524).
Hajcak, G., Moser, J. S., Yeung, N., and Simons, R. F. (2005). On the ERN and the significance of errors. Psychophysiology 42, 151–160. doi: 10.1111/j.1469-8986.2005.00270.x
Harris, C. R., Millman, K. J., Van Der Walt, S. J., Gommers, R., Virtanen, P., Cournapeau, D., et al. (2020). Array programming with NumPy. Nature 585, 357–362. doi: 10.1038/s41586-020-2649-2
Herrmann, M. J., Römmler, J., Ehlis, A. C., Heidrich, A., and Fallgatter, A. J. (2004). Source localization (LORETA) of the error-related-negativity (ERN/ne) and positivity (Pe). Cogn. Brain Res. 20, 294–299. doi: 10.1016/j.cogbrainres.2004.02.013
Hill, H. (2009). An event-related potential evoked by movement planning is modulated by performance and learning in visuomotor control. Exp. Brain Res. 195, 519–529. doi: 10.1007/s00221-009-1821-6
Holroyd, C. B., and Coles, M. G. (2002). The neural basis of human error processing: reinforcement learning, dopamine, and the error-related negativity. Psychol. Rev. 109, 679–709. doi: 10.1037/0033-295X.109.4.679
Iturrate, I., Chavarriaga, R., Montesano, L., Minguez, J., and Millán, J. (2014). Latency correction of event-related potentials between different experimental protocols. J. Neural Eng. 11:036005. doi: 10.1088/1741-2560/11/3/036005
Iwane, F., Iturrate, I., Chavarriaga, R., and Millan, J. D. R. (2022). EEG error-related potentials encode magnitude of errors and individual perceptual thresholds. bioRxiv, 2022–2012. doi: 10.1101/2022.12.06.519418
Ke, Y., Wang, P., Chen, Y., Gu, B., Qi, H., Zhou, P., et al. (2015). Training and testing ERP-BCIs under different mental workload conditions. J. Neural Eng. 13:016007. doi: 10.1088/1741-2560/13/1/016007
Larson, M. J., Perlstein, W. M., Stigge-Kaufman, D., Kelly, K. G., and Dotson, V. M. (2006). Affective context-induced modulation of the error-related negativity. Neuroreport 17, 329–333. doi: 10.1097/01.wnr.0000199461.01542.db
Lopez-Sola, E., Moreno-Bote, R., and Arsiwalla, X. D. (2021). Sense of agency for mental actions: insights from a belief-based action-effect paradigm. Conscious. Cogn. 96:103225. doi: 10.1016/j.concog.2021.103225
LoTemplio, S. B., Lopes, C. L., McDonnell, A. S., Scott, E. E., Payne, B. R., and Strayer, D. L. (2023). Updating the relationship of the ne/ERN to task-related behavior: a brief review and suggestions for future research. Front. Hum. Neurosci. 17:138. doi: 10.3389/fnhum.2023.1150244
Lotte, F. (2012). On the need for alternative feedback training approaches for BCI. Berlin brain-computer Interface workshop.
Lynn, M. T., Berger, C. C., Riddle, T. A., and Morsella, E. (2010). Mind control? Creating illusory intentions through a phony brain–computer interface. Conscious. Cogn. 19, 1007–1012. doi: 10.1016/j.concog.2010.05.007
Maris, E., and Oostenveld, R. (2007). Nonparametric statistical testing of EEG-and MEG-data. J. Neurosci. Methods 164, 177–190. doi: 10.1016/j.jneumeth.2007.03.024
McFarland, D. J., McCane, L. M., David, S. V., and Wolpaw, J. R. (1997). Spatial filter selection for EEG-based communication. Electroencephalogr. Clin. Neurophysiol. 103, 386–394. doi: 10.1016/S0013-4694(97)00022-2
Michie, P. T., Karayanidis, F., Smith, G. L., Barrett, N. A., Large, M. M., O'Sullivan, B. T., et al. (1999). An exploration of varieties of visual attention: ERP findings. Cogn. Brain Res. 7, 419–450. doi: 10.1016/S0926-6410(98)00047-0
Moore, M. M. (2003). Real-world applications for brain-computer interface technology. IEEE Trans. Neural Syst. Rehabil. Eng. 11, 162–165. doi: 10.1109/TNSRE.2003.814433
Omedes, J., Schwarz, A., Müller-Putz, G. R., and Montesano, L. (2018). Factors that affect error potentials during a grasping task: toward a hybrid natural movement decoding BCI. J. Neural Eng. 15:046023. doi: 10.1088/1741-2552/aac1a1
Perrin, F., Pernier, J., Bertrand, O., and Echallier, J. F. (1989). Spherical splines for scalp potential and current density mapping. Electroencephalogr. Clin. Neurophysiol. 72, 184–187. doi: 10.1016/0013-4694(89)90180-6
Plass-Oude Bos, D., Poel, M., and Nijholt, A. (2011). A study in user-centered design and evaluation of mental tasks for BCI. In Advances in Multimedia Modeling: 17th International Multimedia Modeling Conference, MMM 2011, Taipei, Taiwan, January 5–7, 2011, Proceedings, Part II. (pp. 122–134).
Pyasik, M., Scandola, M., and Moro, V. (2022). Electrophysiological correlates of action monitoring in brain-damaged patients: a systematic review. Neuropsychologia 174:108333. doi: 10.1016/j.neuropsychologia.2022.108333
Rawls, E., Miskovic, V., Moody, S. N., Lee, Y., Shirtcliff, E. A., and Lamm, C. (2020). Feedback-related negativity and frontal midline theta reflect dissociable processing of reinforcement. Front. Hum. Neurosci. 13:452. doi: 10.3389/fnhum.2019.00452
Spinelli, G., Tieri, G., Pavone, E. F., and Aglioti, S. M. (2018). Wronger than wrong: graded mapping of the errors of an avatar in the performance monitoring system of the onlooker. NeuroImage 167, 1–10. doi: 10.1016/j.neuroimage.2017.11.019
Syrov, N., Yakovlev, L., Miroshnikov, A., and Kaplan, A. (2023). Beyond passive observation: feedback anticipation and observation activate the mirror system in virtual finger movement control via P300-BCI. Front. Hum. Neurosci. 17:1180056. doi: 10.3389/fnhum.2023.1180056
Ullsperger, M., Fischer, A. G., Nigbur, R., and Endrass, T. (2014). Neural mechanisms and temporal dynamics of performance monitoring. Trends Cogn. Sci. 18, 259–267. doi: 10.1016/j.tics.2014.02.009
Virtanen, P., Gommers, R., Oliphant, T. E., Haberland, M., Reddy, T., Cournapeau, D., et al. (2020). SciPy 1.0: fundamental algorithms for scientific computing in python. Nat. Methods 17, 261–272. doi: 10.1038/s41592-019-0686-2
Walsh, M. M., and Anderson, J. R. (2012). Learning from experience: event-related potential correlates of reward processing, neural adaptation, and behavioral choice. Neurosci. Biobehav. Rev. 36, 1870–1884. doi: 10.1016/j.neubiorev.2012.05.008
Wang, X., Chen, H. T., Wang, Y. K., and Lin, C. T. (2022). Implicit robot control using error-related potential-based brain–computer interface. IEEE Trans. Cogn. Dev. Syst. 15, 198–209. doi: 10.1109/TCDS.2022.3151860
Wlotko, E. W., and Federmeier, K. D. (2012). So that's what you meant! Event-related potentials reveal multiple aspects of context use during construction of message-level meaning. NeuroImage 62, 356–366. doi: 10.1016/j.neuroimage.2012.04.054
Wolpaw, J. R., Birbaumer, N., McFarland, D. J., Pfurtscheller, G., and Vaughan, T. M. (2002). Brain–computer interfaces for communication and control. Clin. Neurophysiol. 113, 767–791. doi: 10.1016/S1388-2457(02)00057-3
Wolpaw, J. R., Millán, J. D. R., and Ramsey, N. F. (2020). Brain-computer interfaces: definitions and principles. Handb. Clin. Neurol. 168, 15–23. doi: 10.1016/B978-0-444-63934-9.00002-0
Xavier Fidêncio, A., Klaes, C., and Iossifidis, I. (2022). Error-related potentials in reinforcement learning-based brain-machine interfaces. Front. Hum. Neurosci. 16:806517. doi: 10.3389/fnhum.2022.806517
Yasemin, M., Cruz, A., Nunes, U. J., and Pires, G. (2023). Single trial detection of error-related potentials in brain–machine interfaces: a survey and comparison of methods. J. Neural Eng. 20:016015. doi: 10.1088/1741-2552/acabe9
Keywords: ERN, error-related potentials, ErrPs, BCI, neurofeedback, brain-AI interaction
Citation: Berkmush-Antipova A, Syrov N, Yakovlev L, Miroshnikov A, Golovanov F, Shusharina N and Kaplan A (2024) Yes or no? A study of ErrPs in the “guess what I am thinking” paradigm with stimuli of different visual content. Front. Psychol. 15:1394496. doi: 10.3389/fpsyg.2024.1394496
Edited by:
Yuxiao Yang, Zhejiang University, ChinaReviewed by:
Hao Fang, University of Central Florida, United StatesLin Yao, Zhejiang University, China
Copyright © 2024 Berkmush-Antipova, Syrov, Yakovlev, Miroshnikov, Golovanov, Shusharina and Kaplan. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Artemiy Berkmush-Antipova, artemgantipov@gmail.com