SSVEP detection assessment by combining visual stimuli paradigms and no-training detection methods

Introduction: Brain-Computer Interfaces (BCI) based on Steady-State Visually Evoked Potentials (SSVEP) have great potential for use in communication applications because of their relatively simple assembly and in some cases the possibility of bypassing the time-consuming training stage. However, among multiple factors, the efficient performance of this technology is highly dependent on the stimulation paradigm applied in combination with the SSVEP detection algorithm employed. This paper proposes the performance assessment of the classification of target events with respect to non-target events by applying four types of visual paradigms, rectangular modulated On-Off (OOR), sinusoidal modulated On-Off (OOS), rectangular modulated Checkerboard (CBR), and sinusoidal modulated Checkerboard (CBS), with three types of SSVEP detection methods, Canonical Correlation Analysis (CCA), Filter-Bank CCA (FBCCA), and Minimum Energy Combination (MEC).

Methods: We set up an experimental protocol in which the four types of visual stimuli were presented randomly to twenty-seven participants and after acquiring their electroencephalographic responses to five stimulation frequencies (8.57, 10.909, 15, 20, and 24 Hz), the three detection methods were applied to the collected data.

Results: The results are conclusive, obtaining the best performance with the combination of either OOR or OOS visual stimulus and the FBCCA as a detection method, however, this finding contrasts with the opinion of almost half of the participants in terms of visual comfort, where the 51.9% of the subjects felt more comfortable and focused with CBR or CBS stimulation.

Discussion: Finally, the EEG recordings correspond to the SSVEP response of 27 subjects to four visual paradigms when selecting five items on a screen, which is useful in BCI navigation applications. The dataset is available to anyone interested in studying and evaluating signal processing and machine-learning algorithms for SSVEP-BCI systems.

1. Introduction

The phenomenon of Steady-State Visual Evoked Potentials (SSVEP) is manifested as electrical brain patterns elicited when a user focuses his/her attention on a repetitive visual stimulus (a light source) flickering at frequencies higher than 6 Hz (Faller et al., 2017). These are periodic oscillations prominently observed in the occipital and occipito-parietal areas of the cerebral cortex. SSVEP responses appear as an increase in the amplitude of the signal at the fundamental frequency and its harmonics for the corresponding stimulus attended by the user (Antelis et al., 2020). In addition to the usual clinical purpose of diagnosing visual pathway and brain mapping impairments, the SSVEP can serve as a basis for Brain-Computer Interfaces (BCI) applications (Amiri et al., 2013; Chen et al., 2021).

BCI can be considered systems within the field of biomedical engineering and neurotechnologies, with the role of restoring or replacing lost neurological functions (Bockbrader et al., 2018), control of devices (Velasco-Álvarez et al., 2021) and establishing communication channels by alternative mechanisms, applying digital signal processing and machine learning techniques to electrical brain waves (McFarland et al., 1997). These technologies use different paradigms that allow them to infer mental states. The main paradigms we can find are: P300 (Farwell and Donchin, 1988), motor imagery (Lotze and Cohen, 2006; Jiang et al., 2020; Pei et al., 2022), and SSVEP (Volosyak et al., 2009). BCIs based on the SSVEP paradigm are the most widespread (Singla, 2018) for the sake of high communication rate, easy system configuration, and less user training (Gao et al., 2003, 2014; Wang et al., 2008; Chen et al., 2015b).

Currently, it is still a pending issue to implement SSVEP-based BCI applications with optimum performance that also meets the user's comfort requirements. The reason for this is due to the multi-factor dependence of SSVEP-based BCI on several elements to achieve the best execution. In response to this, Li et al. (2021) identified four ways to optimize BCI systems based on the SSVEP phenomenon used with spelling function: improving the classification algorithm, adding a spelling prediction function, designing better stimulation paradigms, and adding new triggering methods. The number of target elements, which can be considered as one of the most sensitive (Gembler et al., 2016) parameters, have a direct influence on each of the four optimization methods mentioned. The design of the stimuli, previously noted, is another key aspect since different associated properties are considered, such as the frequency at which the stimuli oscillate, the dimensions of the stimuli, the colors presented, and the type of signal that controls the stimuli (Cysewska-Sobusiak and Jukiewicz, 2016). In the study carried out by Siribunyaphat and Punsawad (2022), an exhaustive analysis of various visual stimulation proposals for BCIs based on the SSVEP phenomenon was performed. This paper reported a summary of the state of the art regarding the design of visual stimuli in SSVEP-based BCI, taking into account parameters such as the Proposed Method, which was included to evaluate performance, the Visual Stimulus, the Electrodes Position, and the Results obtained. All this with the objective of exploring alternative visual paradigms that would achieve better visual comfort and at the same time guarantee a good performance of the BCI. SSVEP detection methods are undoubtedly another aspect to be taken into account for the proper performance of the BCI.

When the visual paradigm is discussed, the type of signal that modulates the flicker is rarely considered. Generally the waveforms used are triangular, sinusoidal, and rectangular, the latter with different levels of duty cycle. According to relevant studies (Teng et al., 2011; Chen et al., 2019), modulation of the stimulus with a rectangular signal leads to better performance results. The waveform that modulates the luminance of the SSVEP stimulation signal has been the subject of relatively few studies compared to other parameters of the visual stimulus, such as the number of targets or the detection methods. Although the answer to the fact that there are so few studies addressing the effect on the performance based on the waveform of the modulating signal may lie in the minor importance of this aspect, it is pertinent to argue that the choice of this parameter of the visual stimulus influences the number of targets to be presented.

Results reported in Li et al. (2021) and Siribunyaphat and Punsawad (2022) compiles several published studies where the number of targets, the design of the stimulus, the SSVEP detection methods, the electrode position, number of subjects, frequencies of stimuli, and the results obtained are taken into account. Although the multi-factorial influence of these parameters is well-known, most studies address their association with performance individually rather than comprehensively. In addition, the combination of parameters that makes optimal the BCI performance is particular to each user, which makes necessary a previous process of calibration.

Newly implemented SSVEP detection algorithms require databases to evaluate their effectiveness. Similarly, certain BCI applications based on the SSVEP paradigm often rely on databases to estimate their viability (Bian et al., 2022). However, with respect to other paradigms, such as motor imagery (Pei et al., 2021) and P300, the SSVEP databases are scarce, as has been reported by Choi et al. (2019a). In the SSVEP database supplied by İşcan and Nikulin (2018) a four-class SSVEP-based BCI was assessed under different perturbations, where the subjects were speaking, thinking, or listening depending on the given task. Liu et al. (2022) implemented an SSVEP database targeting the elderly population, this way providing an opportunity to design BCI systems better suited for eldercare applications. Zhu et al. (2021) makes an interesting contribution by proposing an open-access dataset with a large number of subjects (102) for a wearable SSVEP-based BCI toward practical applications. This proposal comprehensively compares the SSVEP data obtained by wireless wet and dry electrodes. A concise and detailed analysis of the availability of SSVEP databases and the need for public domain access to them was carried out at Liu et al. (2020), concluding that there is a demand for more SSVEP paradigm databases to foster method design and evaluation.

Based on the literature reviewed, we want to answer the question of whether there is a specific combination of SSVEP's visual stimulus scheme with a no-training detection method that arouses a better performance in terms of Accuracy (ACC) and Information Transfer Rate (ITR) when classifying target events with respect to non-target events. According to Li et al. (2021), no study to date has compared the available paradigms to identify the one that delivers the best performance because, when the same methods of SSVEP detection are used, the results obtained are inconsistent owing to the different paradigms used, and thus the influence of the paradigm on the performance of the SSVEP speller cannot be ignored. We addressed this challenge by comparing four types of visual paradigms, i.e., rectangular modulated On-Off (OOR), sinusoidal modulated On-Off (OOS), rectangular modulated Checkerboard (CBR), and sinusoidal modulated Checkerboard (CBS), and studied their interaction with three SSVEP detection methods, i.e., Canonical Correlation Analysis (CCA), Filter-Bank CCA (FBCCA), and Minimum Energy Combination (MEC). We conducted an experiment where we captured non-invasive electroencephalographic (EEG) signals from 27 participants. These subjects were exposed to the four types of visual stimuli, distributed in 4 rounds of 10 min, which at the same time were subdivided into 40 trials of 15 s organized in 5 s of pre-stimulus, 5 s of stimulation, and 5 s of rest. The acquired brain signals were processed and transformed to the frequency domain, where the influence of the type of visual stimulus on the spectral power was evaluated. At the same time, the three SSVEP detection methods were applied to the electroencephalographic dataset, obtaining results in terms of ACC and ITR, where we analyzed which combination (visual stimulus type and detection method) achieved the best performance. At the time this research was conceived, we were unable to find studies that addressed in an integrated manner the influence of parameters such as the visual stimulus paradigm and the SSVEP detection method on the performance when classifying target events with respect to non-target events.

The novelty of our proposal lies in the integrated analysis of two aspects that have a direct impact on the performance of SSVEP-based BCI systems, specifically the visual paradigm applied and the detection method employed. Finally, another contribution of our study lies in the electroencephalographic dataset obtained during the experimental sessions. These training datasets correspond to the SSVEP response to four visual paradigms in which 27 subjects were shown five flashing elements on an LCD screen. This provides approximately 1,080 min of EEG. Our database is not oriented to spelling applications, as is the case with most publicly available SSVEP databases. It is rather oriented as a complement for the assessment of SSVEP detection algorithms and navigation applications although it could have multi-stage communication applications, where the graphical interface has two or more stages and the number of targets in each stage is relatively small (Li et al., 2021).

Our work is structured in a Section 1; a Section 2, where the experimental protocol is described, and the analyses performed on the EEG signals are applied; a Section 3, where we report the results obtained from applying different visual stimuli and SSVEP detection methods; a Section 4, where the results obtained are analyzed and contrasted with the literature consulted, and finally, a Section 5, where the main findings of the research are summarized and future studies derived from this research are suggested.

2. Methods and materials

2.1. Visual stimulus types

In this work, we consider the on-off and checkerboard patterns with luminance modulated by rectangular and sinusoidal functions, resulting in a total of four types of visual stimuli. The description of these visual stimuli is as follows:

• On-Off pattern with rectangular modulated signal (OOR). The luminance in this type of visual stimulus switches suddenly and repeatedly with each rising and falling edge following a rectangular function. High and low states of the function indicate that the visual stimulus is fully on and fully off, respectively, therefore, there are no intermediate intensities for the visual stimulus.

• On-Off pattern with sinusoidal modulated signal (OOS). In this case, the luminance of the visual stimulus gradually turns on and off according to a sine function. Crests and valleys represent the visual stimulus fully on and fully off, respectively, while the rest of the sinusoidal shape represents intermediate intensities of the visual stimulus.

• Checkerboard pattern with rectangular modulated signal (CBR). This visual stimulus consists of a matrix (usually 8 × 8, but can vary) with squares whose luminance alternates following a rectangular function. The transition from high to low (and vice versa) in the rectangular function indicates the sudden luminance change of the boxes in the visual stimulus.

• Checkerboard pattern with sinusoidal modulated signal (CBS). In this case, the luminance of the squares in the matrix gradually varies according to a sinusoidal function. Crests/valleys represent the visual stimulus with squares fully on/off and the rest of the sinusoidal shape represents intermediate color intensities of the visual stimulus.

We choose the on-off and checkerboard patterns because these two types of stimuli have generally been used to evoke SSVEPs for a BCI application on computer screens (Zhu et al., 2010), and the rectangular and sinusoidal modulated signals because these waveforms are the most employed to control the stimulus luminance (Cysewska-Sobusiak and Jukiewicz, 2016; Chen et al., 2019). Note that for the case of visual stimulus based on the on-off pattern the frequency of the modulating signal represents the stimulation frequency expected to be observed in the SSVEP responses. However, for visual stimulus based on the checkerboard pattern, the stimulation frequency expected to be observed in the SSVEP responses is twice the frequency of the modulating signal. This is because the SSVEP is produced at its rate of phase-reversal or alternation rate (Regan, 1977; Burkitt et al., 2000; Lalor et al., 2005). As an illustration, Figure 1 depicts how the intensity of the on-off pattern is modulated by rectangular and sinusoidal functions. In this work, the colors of the four types of visual stimulus were black and white with color transitions from black, in the off-state, to white, in the on-state.

FIGURE 1

Figure 1. Illustration of how the intensity of the On-Off and Checkerboard patterns are modulated by rectangular and sinusoidal functions. (A) On-Off paradigm transition, where the square's luminance alternates with the background controlled by a modulating periodic waveform. (B) Checkerboard paradigm transition, where the luminance of the square grid array alternates controlled by a modulating periodic waveform. (C) Rectangular waveform luminance modulation. (D) Sine waveform luminance modulation.

2.2. SSVEP detection methods

The SSVEP detection methods employed have the particularity that they do not require prior training to be applied to EEG recordings. To present the technical details, we define a multidimensional variable X = [x_e(t)]N_e×N_t that is representative of the brain electrical activity captured on the scalp, organized in segments of N_e electrodes and N_t samples.

It is assumed at all times that the analyzed signals correspond to the instances in which the participants were gazing at each blinking element applied during an experimental session, thus corresponding to EEG recordings in which the participant focused on one of the N_target visual stimuli that were flickering at a frequency f_i ∈ ℝ; f_i:{f₁, f₂, ..., f_Ntarget}. In each of the approaches, the goal is to estimate which of the targets the participant was focusing on considering as cues the EEG activity X, the N_target sources of visual stimulation, and the f_i stimulation frequencies.

2.2.1. Canonical correlation analysis (CCA)

In the SSVEP CCA method, the idea is to find linear combinations that maximize the correlation between two vectors: the EEG signal samples X = [x_e(t)]N_e×N_t corresponding to the moment when the subject is gazing at one of the N_target blinking targets; and a template array of reference sinusoidal signals: Y_fi = [y_r(t)]2N_h×N_t for each of the f_i stimulation frequencies, where N_h represents the number of harmonics for a sinusoidal template signal of frequency f_i. Here we define Y_fi as:

$\begin{array}{ll}{Y}_{f_i}(t)=\left[\begin{array}{c}sin(2\pi f_{i}t)\\ cos(2\pi f_{i}t)\\ ⋮\\ sin(2\pi N_h{f}_{i}t)\\ cos(2\pi N_h{f}_{i}t)\end{array}\right]& (1)\end{array}$

The linear combinations are defined as p = Xw_p and q = Y_fiw_q that maximize the so-called canonical correlation ρ between them. Hence, the weight vectors ${\mathbf{\text{w}}}_p\in {ℝ}^{N_e\times 1}$ and ${\mathbf{\text{w}}}_q\in {ℝ}^{2N_h\times 1}$ are found by solving:

$\begin{array}{ll}\rho ={\displaystyle \underset{{\text{w}}_p,{\text{w}}_q}{max}}corr(p,\text{q})& (2)\end{array}$

which can be rewritten as the following optimization problem (Zhang et al., 2014):

$\begin{array}{ll}\rho ={\displaystyle \underset{{\text{w}}_p,{\text{w}}_q}{max}}\frac{{\text{w}}_p^{\top }{\text{C}}_{pq}{\text{w}}_q}{\sqrt{{\text{w}}_p^{\top }{\text{C}}_{pp}{\text{w}}_p{\text{w}}_q^{\top }{\text{C}}_{qq}{\text{w}}_q}}& (3)\end{array}$

where C_pq is the cross-covariance matrix and C_pp and C_qq are the auto-covariance matrices for X and Y_fi, respectively. The solution is obtained by solving a generalized eigenvalue problem (Hardoon et al., 2004), from which the weight vector w_p is an eigenvector of ${\text{C}}_{pp}^{-1}{\text{C}}_{pq}{\text{C}}_{qq}^{-1}{\text{C}}_{qp}$ and the weight vector w_q is an eigenvector of ${\text{C}}_{qq}^{-1}{\text{C}}_{qp}{\text{C}}_{pp}^{-1}{\text{C}}_{pq}$. The maximum canonical correlation corresponds to the maximum value of ρ with respect to w_q and w_p (Hardoon et al., 2004).

2.2.2. Filter-bank canonical correlation analysis (FBCCA)

The ability to incorporate harmonic components in frequency identification methods is an advantage since they provide useful information for the performance of these procedures. The FBCCA method is able to extract the discriminative information embedded in the harmonic components of the SSVEP responses in a more efficient way. This is achieved by using band-pass filter banks that decompose the SSVEPs into several sub-band components. By splitting the original SSVEP response into several signal versions with different bandwidths, it is ensured that not only the fundamental frequency component contributes to the detection process but also its different harmonics. FBCCA method was proposed to enhance the CCA detection method on SSVEP phenomenon (Chen et al., 2015a). The algorithm involves three stages: filter bank analysis; CCA between the SSVEP components filtered by sub-bands and the reference sinusoidal signals, and finally; the identification of the target element.

First, a filter bank analysis, with SB band-pass filters, was applied on the EEG segment X and from this procedure, SB versions of the original signal were obtained (X_j, j = 1, 2, ..., SB). In our study, five Chebyshev band-pass type I Infinite Impulse Response (IIR) filters were implemented with pass bands ranging from 6 − 90, 9 − 90, 13 − 90, 18 − 90, and 22 − 90 Hz, respectively (Chen et al., 2015a). CCA was then applied between the reference signal (Y_fi, i = 1, 2, ..., N_target) and each X_j sub-band component separately:

$\begin{array}{ll}{\rho }_{f_i}=\left[\begin{array}{c}{\rho }_{f_i}^1\\ \\ {\rho }_{f_i}^2\\ \\ ⋮\\ \\ {\rho }_{f_i}^{N_{SB}}\end{array}\right]& (4)\end{array}$

$\begin{array}{ll}=\left[\begin{array}{c}\rho ({\text{X}}_{j_1}^{\top }{\text{W}}_{\text{X}}({\text{X}}_{j_1}{\text{Y}}_{f_i}),{\text{Y}}^{\top }{\text{W}}_{\text{Y}}({\text{X}}_{j_1}{\text{Y}}_{f_i}))\\ \rho ({\text{X}}_{j_2}^{\top }{\text{W}}_{\text{X}}({\text{X}}_{j_2}{\text{Y}}_{f_i}),{\text{Y}}^{\top }{\text{W}}_{\text{Y}}({\text{X}}_{j_2}{\text{Y}}_{f_i}))\\ ⋮\\ \rho ({\text{X}}_{j_{SB}}^{\top }{\text{W}}_{\text{X}}({\text{X}}_{j_{SB}}{\text{Y}}_{f_i}),{\text{Y}}^{\top }{\text{W}}_{\text{Y}}({\text{X}}_{j_{SB}}{\text{Y}}_{f_i}))\end{array}\right]& (5)\end{array}$

Sub-band EEG components were obtained for each frequency by applying a weighted sum of squares of the previously derived coefficients:

$\begin{array}{ll}{\tilde{\rho }}_{f_i}={\displaystyle \sum _{n=1}^{SB}}w(n)·{({\rho }_{f_i}^n)}^2& (6)\end{array}$

In Equation (6), n is the sub-band index.

The target frequency is then considered to correspond to the reference signal yielding the maximal ${\tilde{\rho }}_{f_i}$.

In our proposal, we used the fundamental frequency and three harmonics of this, for a total of N_h = 4. We chose this number of harmonics since it was the one that gave the best performance when testing different values from 1 to 10 with the standard CCA method, as proposed in Chen et al. (2015a).

2.2.3. Minimum energy combination (MEC)

The Minimum Energy Combination finds spatial filters to improve EEG responses of the oscillatory components modulated in one particular control task (Friman et al., 2007). An EEG signal segment can be modeled as follows:

$\begin{array}{ll}{x}_e(t)={\displaystyle \sum _{h=1}^{N_h}}(a_{e,h}sin(2\pi h{f}_{i}t)+b_{e,h}cos(2\pi h{f}_{i}t))+{\eta }_{e,f_i}(t)& (7)\end{array}$

here N_h (as in CCA) is the number of harmonics of the model; a_{e, h} and b_{e, h} are multiplicative constants for each channel e and harmonic h and η_e,fi(t) represents the measured activity that is not associated with the SSVEP manifestation. This relationship can also be expressed in matrix format as follows (Volosyak, 2011):

$\begin{array}{ll}\text{X}=\text{G}{\text{X}}_{f_i}^{ref}+{\eta }_{f_i}& (8)\end{array}$

In Equation (8), G is defined as

$\begin{array}{ll}G={\left|\begin{array}{ccccc}{a}_{1,1}& b_{1,1}& \mathrm{...}& a_{1,n_h}& b_{1,N_h}\\ a_{2,1}& b_{2,2}& \mathrm{...}& a_{2,n_h}& b_{2,N_h}\\ ⋮& ⋮& \mathrm{...}& ⋮& ⋮\\ a_{N_e,1}& b_{N_e,1}& \mathrm{...}& a_{N_e,N_h}& b_{N_e,N_h}\end{array}\right|}_{N_e\times 2N_h}& (9)\end{array}$

where N_e represents the number of channels and ηf_i = [η_e,fi(t)]_Ne×Nt.

An optimized approximation of the η_fi matrix can be obtained from Equation (8)

$\begin{array}{ll}{\eta }_{f_i}^{\ast }=X-G^{\ast }{X}_{f_i}^{ref}& (10)\end{array}$

As ${\eta }_{f_i}^{*}$ represents the interfering and artifact activity captured at the electrodes, then the objective is to minimize this manifestation. The MEC method precisely obtains N_m spatial filters that minimize the energy of ${\eta }_{f_i}^{*}$ in such a way that the new channels obtained only contain information associated with the brain's electrical activity resulting from the visual stimulation paradigm applied at a given frequency f_i. The matrix of spatial filters W is obtained from

$\begin{array}{ll}\text{W}=\left[\frac{v_1}{\sqrt{{\lambda }_1}}\frac{v_2}{\sqrt{{\lambda }_2}}...\frac{v_{N_m}}{\sqrt{{\lambda }_{N_m}}}\right]=\left[w^1{w}^2...w^{N_m}\right]& (11)\end{array}$

where N_m is the number of spatial filters; v_i = {v₁, v₂, ..., v_Nm} and λ_i = {λ₁, λ₂, ..., λ_Nm} are the eigenvectors and their corresponding eigenvalues for the optimized matrix M defined by $\text{M}={\eta }_f^{*}{({\eta }_f^{*})}^{\top }$.

Finally, the average power for each target frequency f_i is obtained as

$\begin{array}{ll}{p}_{f_i}=\frac{1}{N_m{N}_h}{\displaystyle \sum _{i=1}^{N_m}}{\displaystyle \sum _{j=1}^{2N_h}}{\displaystyle \sum _{t=1}^{N_t}}\left(x_{j,f_i}^{ref}(t){\displaystyle \sum _{e=1}^{N_e}}{w}_e^i{x}_e(t)\right)& (12)\end{array}$

The biggest estimated power for each target frequency is considered the detected frequency (Mendoza-Montoya, 2018).

2.3. Experimental protocol

The experiments were carried out in an acoustically isolated room. Participants were seated in a comfortable chair in front of a 27-inch Samsung computer screen, model C27RG50FQL, with 1920 × 1080 resolution and 240 Hz refresh rate. A Graphical User Interface (GUI) was displayed on this screen with five squares evenly distributed and with an instruction box at the bottom. The squares, with dimensions of 116 × 116 pixels, were used as a visual stimulus and therefore they flickered at a specific frequency. The instruction box, with dimensions of 1920 × 116 pixels, was used to guide the users on the execution of the experiment. Figure 2A shows a sketch of the experimental setup with a participant, the EEG recording system, and the computer screen with the GUI. Figure 2B shows the GUI with the five squares used as a visual stimulus, their flickering frequencies, and the instruction box.

FIGURE 2

Figure 2. Description of the experimental protocol. (A) Sketch of the experimental setup with a participant, the EEG recording system, and the computer screen with the GUI displaying the five visual stimuli and the instruction box. (B) Configuration of the GUI with the five visual stimuli and their stimulation frequencies: 8.5714 Hz in the center, 10.9091 Hz in the bottom, 15 Hz in the right, 20 Hz in the top, and 24 Hz in the left square symbol.

The flickering of each square was performed at a specific frequency depending on the refresh rate of the monitor (240 Hz). The square located at the center, bottom, right, top, and left flickered at a frequency of 8.5714, 10.9091, 15, 20, and 24 Hz, respectively. These frequencies were selected because they are not multiples of each other, and they are in the range that has been commonly used in previous studies to induce SSVEP responses (Ng et al., 2012; Cysewska-Sobusiak and Jukiewicz, 2016; Chen et al., 2019). These studies suggest that the stimulation frequencies that generate the strongest SSVEP responses are in the range of 5 to 25 Hz. As the stimulation frequencies were generated using as reference the monitor refresh rate, they are indeed 28, 22, 16, 12, and 10 times slower than the 240 Hz, respectively. Note in Figure 2B that each square contains inside them an arrow, with dimensions of 16 × 9 pixels, pointing up, down, left, and right respectively if the square is located on the top, bottom, left, and right of the screen, while the square located on the screen center contains a traffic stop symbol, with dimensions of 16 × 16 pixels. This is because the GUI, as one of its applications, has the functionality of a navigation system (Mendoza-Montoya, 2018).

Experiments were executed in trials where the participant focused her/his visual attention on one out of the visual stimuli according to the information shown in the instruction box of the GUI. The basic timing sequence of a trial is depicted in Figure 3 and consisted of the following five phases:

1. Fixation. A cross symbol is shown during 2s in the information box, which indicates to be prepared and relaxed. None of the squares flickers during this phase.

2. Target Presentation. One of the five squares is randomly highlighted with a blue background and the corresponding arrow or stop symbol is shown in the information box. This event last 2s and indicates to the participant the specific visual stimulus that they have to focus their attention on during the subsequent Stimulation phase.

3. Preparation. None of the squares is highlighted or shown in the information box. This last 1s and indicates to be ready for the upcoming Stimulation phase.

4. Stimulation. All five squares flicker each one at its specific stimulation frequency. In this phase, participants are requested to focus their gaze on the square specified in the Target Presentation phase while ignoring the other visual stimuli. This visual stimulation last 5s.

5. Rest. None of the stimuli is highlighted and the text Rest is presented in the information box. This instructed the participants to rest from the experiment during 5s.

FIGURE 3

Figure 3. Temporal sequence of a trial. Each trial consists of five phases: Fixation (2 s), Target Presentation (2 s), Preparation (1 s), Stimulation (~5 s), and Rest (~5 s).

For each participant, experiments were carried out in a single experimental session consisting of 4 recording runs. In each run, only one visual stimulus type (either OOR, OOS, CBR, or CBS) was employed. Moreover, the order of the visual stimulus type was randomized across runs for each of the participants. A total of forty trials were recorded per run (which represents ~10min of data recording per run), yielding eight trials for each of the five stimulation frequencies. Figure 4 shows the temporal sequence of a typical experimental session. To avoid fatigue and reduce tiredness, participants were allowed to rest between runs for about 2 or 3 min, or longer if needed. During the execution of the experiment, the participants were duly instructed to avoid moving the body or head and blinking between the Fixation and the Stimulation phases, while they were advised to move during the Rest phase if needed. At the end of the experimental session of a participant, four recording runs were obtained, one for each type of visual stimulation, and each run contains forty trials, that is, eight trials for each of the five stimulation frequencies.

FIGURE 4

Figure 4. Temporal sequence of an experimental session. A session consists of four recording runs, one for each type of visual stimulus. Each run consists of 40 trials, that is, eight trials for each of the five stimulation frequencies.

It is important to note that during the stimulation phase, the squares are the ones that change in luminance and not the arrows or hexagon symbols located inside them. These symbols have no function other than to indicate the direction in BCI navigation applications.

2.4. Participants and data acquisition

Twenty-seven (27) participants (12 women and 15 men) were recruited for this study. The age range was between 18 and 24 years. All volunteers had normal or corrected vision with glasses. In the recruitment of volunteers, exclusion criteria were implemented such as having had epileptic episodes, having been diagnosed with a psychiatric disorder, or subjects with significant progressive disorders or unstable medical conditions requiring acute intervention. Participation in the study was voluntary and all subjects had the opportunity to quit the experiment at any time they wished. Each volunteer was instructed in detail about the objective of the study and the procedure to be carried out. All participants voluntarily signed an informed consent form, which complied with the standards of the Declaration of Helsinki.

To collect the EEG signals, 8 channels of a g.SCARABEO Ag/AgCl active electrodes system and a g.USBamp biosignal amplifier were used. The acquisition of EEG signals was enhanced through the inoculation of conductive gel on the active electrodes, attached to a g.GAMMAcap. The previously described instrumentation comes from the manufacturer g.tec medical engineering GmbH, Schiedlberg, Austria. According to the international 10-20 system, the channels used were PO7, PO3, POz, PO4, PO8, O1, Oz, and O2, in addition to the ground, placed to the AFz channel, and a reference channel, located in the right ear lobe. The EEG signals were discretized at a sampling frequency of 256 Hz. A band-pass filter in the interval between 0.5 and 60 Hz and a Notch filter configured to suppress the presence of the power line frequency were applied to the sampled recordings.

The interface used in our experiments (Mendoza-Montoya, 2018) allows registering, together with the 8 EEG channels, an additional channel containing the time markers associated with each of the five events generated from the computer during the occurrence of a trial. Thus, for each trial, there are labels indicating the precise moment when the fixation cross is presented to each participant, the target to be focused on during the subsequent stimulation stage, the preparation instant, the stimulation phase, and the resting phase. The encoding of these time marks is presented in detail in the description of the database, located at https://zenodo.org/record/7758425#.ZBvGmnbMLIW.

The user comfort was associated with the performance of the representative visual stimuli. This was subjectively measured by a two-question survey applied to each participant at the end of each experiment. The survey questions were:

• For the four types of stimuli (OOR, OOS, CBR, CBS), was it possible to distinguish the type of modulation (rectangular from sinusoidal) on the luminous intensity of the stimulus?

• Which of the four types of visual stimulus best contributed to keeping vision focused on the screen?

2.5. Frequency analysis

The goal of this analysis was to study the spectral power of the EEG signals during visual stimulation and no stimulation at all. To perform this analysis we extracted two different EEG epochs from each trial: (i) Stimulation epochs comprising EEG signals of the entire 5 s-long Stimulation phase; and (ii) No-stimulation epochs comprising EEG signals of the Fixation, Target Presentation, and Preparation phases, and therefore, having a duration of 5 s. Figure 5A shows how these two epochs are extracted from each trial.

FIGURE 5

Figure 5. Graphical illustration of the EEG data epochs used for the frequency analysis and for the assessment of SSVEP detection. (A) For the frequency analysis two 5 s-long epochs were extracted from each trial, No-stimulation epoch (comprising the Fixation, Target Presentation and Preparation phases) and Stimulation epoch (containing the Stimulation phase). (B) For the assessment of SSVEP detection, one 6 s-long epoch was extracted from each trial initiating 1 s prior to the onset of the Stimulation phase and finishing at the end of this phase.

This frequency analysis is essential because we expect to find larger spectral power values in Stimulation than in No-stimulation EEG response, specifically at the stimulation frequencies and its harmonics, and to determine which one of the four visual stimulation types induces the larger spectral power values during Stimulation.

The Power Spectral Density (PSD) method was used to compute the spectral power as this is one of the most common and robust approaches to performing frequency analysis of EEG signals (Wang et al., 2006; Nakanishi et al., 2018). The PSD was computed using the Fieldtrip toolbox (Oostenveld et al., 2010) for each of the trials corresponding to each of the five stimulation frequencies, both for the Stimulation and No-stimulation conditions. Raw EEG trials were tapered by a 2-s Hanning window (Proakis and Manolakis, 2006) with no overlapping. Then, the Fast-Fourier-Transform (FFT) of the data was taken. This was done for frequencies between 2 and 50 Hz with steps of 0.5 Hz.

The spectral power of the Stimulation and No-stimulation conditions were studied for each of the four visual stimulus types (OOR, OOS, CBR, and CBS) and each of the five stimulation frequencies (8.5714, 10.9091, 15, 20, and 24 Hz). The non-parametric paired data statistical test Wilcoxon rank-sum test was used to determine significant differences between the two conditions. Statistical analyses were carried out separately for each channel (8 in total) and frequency (96 in total), resulting in channel-frequency maps of statistical significance that allow visual inspection of the channels and frequencies where there are and there are no significant differences between the two conditions. These statistical tests were performed at a confidence level of α = 0.01. Bonferroni correction was applied to account for the multiple comparisons errors (Henry, 2015) due to the number of channels and frequencies.

Another frequency analysis implemented was the estimation of the wide-band signal-to-noise ratio (SNR), defined in Equation (13). This parameter allows us to properly characterize the broadband noise as well as the contribution of harmonics to the signals (Liu et al., 2020).

$\begin{array}{ll}SNR=10lo{g}_{10}\frac{{\displaystyle {\sum }_{k=1}^{k=Nh}}P(k.f_i)}{{\displaystyle {\sum }_{f=0}^{f=f_s/2}}P(f)-{\displaystyle {\sum }_{k=1}^{k=Nh}}P(k.f_i)}& (13)\end{array}$

Here Nh represents the number of harmonics (Nh = 4), P(f_i) denotes the power spectrum at each stimulation frequency f_i, and f_s/2 is the Nyquist frequency.

2.6. Assessment of SSVEP detection

To carry out this analysis we extracted epochs comprising EEG signals from 1s before the initiation of the Stimulation phase and up to the end of this phase. Hence, the duration of the epochs was 6s. Epochs were time re-referenced to the initiation of the Stimulation phase, that is, t = 0 and t = 5s represent the initiation and the end of the visual stimulation, respectively, while there is no stimulation during −1 ≤ t < 0s. Figure 5B illustrates how an epoch is extracted from a trial to perform SSVEP detection.

Given an EEG data epoch, the detection of the stimulation frequency was carried out using sliding EEG data windows of length T_win seconds in steps of T_shift seconds. Figure 6 illustrates the process employed to detect the stimulation frequency in an epoch. This procedure was chosen because it allows calculating the stimulation frequency over time as new EEG observations are available resembling the case of a realistic online situation. Following previous studies (Chen et al., 2015b; Nakanishi et al., 2018; Liang et al., 2020), in our analysis we used a time window of length 1s (T_win = 1s) and steps of 0.05s (T_shift = 0.05s). We considered other window lengths and steps and these two values provided better performance in terms of accuracy and detection time.

FIGURE 6

Figure 6. Illustration of the EEG data windows used to carry out SSVEP detection in a 6 s-long data epoch.

The detection of the SSVEP responses was assessed for each combination of detection methods (CCA, FBCCA, and MEC) with visual stimulus type (OOR, OOS, CBR, and CBS), in each one of the stimulation frequencies (8.5714, 10.9091, 15, 20, and 24 Hz). The following metrics were computed to assess performance in the detection of the SSVEP responses:

• Detection Accuracy (DA). This metric measures the percentage of correct detections. For the estimation of DA, the number of correct detections N_detects was computed concerning the total number of trials N_Trials according to DA = N_detects/N_Trials. One DA value was obtained for each stimulation frequency and visual paradigm.

• Detection Time (DT). This metric measures the time elapsed to decide the final stimulation frequency in an epoch as in the case of a realistic online situation of SSVEP detection. Therefore, the detection time in an epoch was computed as the time at which three consecutive windows are associated with the same stimulation frequency. We decided to use three consecutive time windows to choose the stimulation frequency of an epoch because this allows us to select it as in an online setup. Finally, the average Detection Time across trials was calculated, obtaining a value for each stimulation frequency and visual paradigm.

• Information Transfer Rate (ITR). This metric estimates the online BCI performance, however, we propose to use this measure to assess the pseudo-online evaluation of SSVEP target classification, using as parameters the previously defined DA, the speed with which a target is correctly chosen (DT), and the number of targets (Volosyak, 2011). Thus, the ITR was calculated across-all epochs by the following equation:

$\begin{array}{ll}ITR=s\left[log(N)+DAlog(DA)+(1-DA)log(\frac{1-DA}{N-1})\right]& (14)\end{array}$

where s = 60/DT is the number of commands performed per minute, N is the number of targets (in our case N = 5) and DA is the detection accuracy.

Statistical non-parametric Kruskal-Wallis test was applied to assess significant differences between distributions of DA, ITR, and DT respectively, for the four visual paradigms and the three detection methods and this way obtain the winning combination in each of the three performance parameters. All statistical tests were carried out at a confidence level of α = 0.05.

2.7. Dataset description

Each participant performed one session consisting of a 5-target SSVEP selection task, giving four data files per subject at the end of each experiment. The four data files are according to the four stimuli types: OOR, OOS, CBR, and CBS. The raw EEG signals along with a detailed description of the recorded data are freely available and can be accessed to an open-access site with https://zenodo.org/record/7758425#.ZBs5DnbMLIU. The database for this study is also available on request to the corresponding author.

3. Results

The results obtained from the analysis of SSVEP signals are presented below. The main objective was to study the brain response to different combinations of visual stimuli (On-Off and Checkerboard) and light pattern modulating waveforms (sinusoidal and rectangular pulse), and second, to investigate the effect on the classification accuracy of applying three SSVEP detection methods (MEC, CCA, and FBCCA) to the EEG signals obtained after presenting four types of visual paradigm. Finally, we also assessed the visual comfort of users by applying different visual paradigms that elicit an SSVEP response.

3.1. User perception

In the user comfort assessment, we found that out of the 27 participants, only 5 stated that they were able to distinguish the rectangular signal modulation from sinusoidal signal modulation, for both, On-Off and checkerboard patterns, thus compiling an 18.5%. Out of the 27 subjects, 51.9% expressed a predilection for the checkerboard pattern in terms of comfort and focus, regardless of the modulating signal shape.

3.2. Frequency analysis

In the results, we initially compare the power response corresponding to the Stimulation vs. No-Stimulation conditions for each of the eight channels and the four visual paradigms. Subsequently, we compared the power of the brain responses for each of the four visual paradigms applied during the Stimulation stage. This was done for all participants and the five stimulation frequencies generated in the GUI. A frequency-domain analysis of the signals was performed to evaluate which of the four visual paradigms elicited the strongest SSVEP response at each of the 5 stimulation frequencies. For this purpose, the PSD was obtained by calculating the FFT of each trial to finally obtain the average power characteristic for each stimulation frequency and each visual paradigm. We found that the spectral power at the stimulation frequency is higher in the Stimulation phase than in the No-Stimulation phase. We also obtained greater power in the brain response to visual paradigms with On-Off stimulation, regardless of the modulating waveform of the visual stimulus, for each of the five stimulation frequencies.

To illustrate each of these results, Figure 7 shows the EEG average power response for the five stimulation frequencies when Participant 27 was subjected to the OOR paradigm. Each of the five stimulation frequencies is represented by a vertical dashed line of a specific color. Thus, the application of a stimulus at a frequency of 8.57 Hz is represented in blue, the stimulus at a frequency of 10.909 Hz is represented in green, the stimulus at a frequency of 15 Hz is represented in red, the stimulus at a frequency of 20 Hz is represented in yellow and finally, the stimulus at a frequency of 24 Hz is represented in gray. The brain response to the application of the visual OOR paradigm during the Stimulation stage is represented with the color previously described corresponding to each stimulation frequency. The brain response during the No-Stimulation stage is represented in black. Additionally, we incorporated in the Supplementary material document the comparison plots of the EEG average power during the Stimulation vs. No-Stimulation stages for the five stimulation frequencies applied to one participant (subject 27). In each of the eight channels, the highest power peaks match the stimulation frequency represented by a vertical dashed line. Except for the PO_z electrode, a difference in amplitude of the peak values in the occipital channels with respect to the parietal-occipital channels can also be distinguished. The response of the central channels (PO_z and O_z) for this participant, as in most of the subjects, is superior to that of the other channels.

FIGURE 7

Figure 7. EEG average power response per channel of one participant (subject 27). For each stimulation frequency, the SSVEP responses are presented in colors corresponding to the Stimulation (blue, green, red, yellow, and gray) and No-Stimulation (black) stages when the visual OOR paradigm is applied at the following frequencies: 8.57 Hz (dashed blue vertical line), 10.909 Hz (dashed green vertical line), 15 Hz (dashed red vertical line), 20 Hz (dashed yellow vertical line), and 24 Hz (dashed gray vertical line).

Table 1 shows the average power values across all channels and subjects and each visual paradigm. For each stimuli frequency (column) the maximum power values have been shaded for each of the four visual paradigms applied. These results are across all participants during the experimental session. For stimulation frequencies 1 and 4 (8.57 and 20 Hz) a maximum average value was obtained when applying the OOR paradigm, while for stimulation frequencies 2, 3, and 5 (10.909, 15, and 24 Hz) the maximum average power values were obtained when applying the OOS paradigm. In the column representing the average power across all stimulation frequencies, the maximum value corresponding to the OOS visual paradigm is highlighted in gray.

TABLE 1

Table 1. Average power (in μV²/Hz units) across-all participants, trials, and channels for each type of visual stimulus at the five target frequencies (f1 = 8.57 Hz, f2 = 10.909 Hz, f3 = 15 Hz, f4 = 20 Hz, and f5 = 24 Hz).

The results of the wide-band SNR estimation are reported in Table 2. SNR values are presented across participants, trials, and channels for each of the four visual paradigms and the five stimulation frequencies. The OOS visual paradigm exhibits higher values than those of the other paradigms for the five stimulation frequencies. To better distinguish in the table, the results corresponding to this type of visual stimulus have been highlighted with a gray background. On the other hand, the lowest values are concentrated in the CBR paradigm, except for the value corresponding to the stimulation frequency of 10,909 Hz (−13.20 dB), associated with the CBS visual paradigm. Another aspect to note is the decrease in SNR as the stimulation frequency increases for the four visual paradigms.

TABLE 2

Table 2. Across subjects, trials, and channels wide-band SNR (in dB units) for each type of visual stimulus and stimulation frequencies, considering Nh = 4, where Nh is the number of harmonics.

3.3. SSVEP detection methods

Figure 8 illustrates the detection accuracy as a function of time across participants, channels, trials, and stimulation frequencies. This was obtained for each of the four visual paradigms and the three detection methods. For the three detection methods, there is a marked difference between stimulation with OOR and OOS paradigms concerning CBR and CBS, the former being superior. Peak performance in all four visual paradigms is achieved after the first second of visual stimulus application. Additional time-dependent performances for each stimulation frequency for the three detection methods are reported in Supplementary Figure 7.

FIGURE 8

Figure 8. Detection Accuracy (DA) curves vs. time for each visual paradigm across subjects, trials, channels, and stimulation frequency: (A) FBCCA, (B) CCA, (C) MEC.

Figure 9 depicts all participant's representative bar graphs of the average accuracy when classifying target vs. non-target events with its corresponding standard deviation at each stimulus frequency for each visual paradigm and with each SSVEP detection method applied. The FBCCA detection method combined with the OOR or OOS visual paradigms outperforms the alternatives involving the checkerboard paradigm in the five stimulation frequencies presented to the users. Also, the MEC detection method combined with the CBR stimulus type yields the worst results in performance at each of the five frequencies applied. The maximum average accuracy (88.83%) was obtained with the FBCCA method for the frequency of 10.909 Hz with the OOS paradigm. The minimum average accuracy value (9.60%) was obtained with the MEC method for the frequency of 24 Hz applying the CBS visual paradigm. This accuracy rate is below the chance level.

FIGURE 9

Figure 9. Across-all participants average accuracy for each stimulation frequency, stimulation pattern, and SSVEP detection method.

Figure 10 shows the violin plot representation of the distribution of DA values for each visual paradigm (OOR, OOS, CBR, and CBS) across trials, channels, and stimulation frequencies, and for each of the three SSVEP detection methods (FBCCA, CCA, and MEC). Violin plots allow us to appreciate the nature of multi-modal distributions of numerical data, providing information about its dispersion or concentration. Each plot shows the probability density of the DA values, with the thicker parts representing regions with more data points. The violin plots show that the medians corresponding to the OOR and OOS distributions are larger than the CBR and CBS distributions. Particularly, the median corresponding to the distribution of the OOS visual paradigm is the highest in the three SSVEP detection methods. There is a greater dispersion of the data in the CBR and CBS distributions for the FBCCA method and in the OOR and OOS distributions for the MEC method, while for the CCA method, the distributions corresponding to the four visual paradigms are more concentrated around their medians. The results of the Kruskal-Wallis and multiple comparison test indicated that there were significant differences in DA across the 12 experimental conditions (four visual stimuli paradigms by three no-training detection methods) for the FBCCA detection method (Figure 10A). Specifically, there were significant differences in DA between OOR and CBR; OOR and CBS; OOS and CBR, and OOS and CBS (p ≤ 0.001). However, there were no significant differences in DA between OOR and OOS and CBR and CBS. It is important to note that this pattern of results was consistent across all three detection methods (FBCCA, CCA, and MEC). These results suggest that the choice of visual stimuli paradigm can significantly impact the accuracy of SSVEP detection and that OOR and OOS paradigms may be more effective than CBS and CBR independent of the specific detection method being used.

FIGURE 10

Figure 10. Violin-plot representation of Detection Accuracy (DA) curves for each visual paradigm across trials, channels, and stimulation frequency: (A) FBCCA, (B) CCA, (C) MEC. Significant differences between pairs of groups of visual paradigms are represented with asterisks, such that ***p ≤ 0.001; no asterisk above the bracket means “not significant”.

Table 3 shows the average detection time values for each stimulation frequency and visual paradigm. The minimum detection time (0.47 ± 0.40s) was obtained for the frequency of 8.57Hz with the CBR paradigm in the FBCCA method, while the maximum value (1.51 ± 1.45s) was found for the frequency of 20Hz for the CBR paradigm in the MEC method. Considering the five stimulation frequencies, three of them (15, 20, and 24 Hz) show the shortest detection time values when the CCA method was applied with the OOR and CBS paradigms. The detection times obtained with the MEC method yielded the slowest response values for the five stimulation frequencies and the four visual paradigms when compared to the CCA and FBCCA methods. Finally, the column representing the detection time across all stimulation frequencies shows that the lowest value corresponds to the visual OOS paradigm for the CCA detection method with a value of 0.93 ± 1.08 s. This cell is highlighted in the gray background for better distinction.

TABLE 3

Table 3. Across-all participants and trials average detection time (in s units) for each type of visual stimulus at the five target frequencies (f1 = 8.57 Hz, f2 = 10.909 Hz, f3 = 15 Hz, f4 = 20 Hz, and f5 = 24 Hz).

The violin plots in Figure 11 illustrate the distributions corresponding to the detection time (DT) parameter for the four visual paradigms and the three SSVEP detection methods. Each graph shows that the medians of each distribution are similar and that the data distributions are well-concentrated around them. This is true for all four visual paradigms and all three detection methods. The results of the Kruskal-Wallis and multiple comparison tests indicated that there were no significant differences in detection time (DT) across the twelve experimental conditions (four visual stimuli paradigms by three no-training detection methods) for the FBCCA detection method (p>0.05). This pattern of results was consistent also for CCA and MEC detection methods.

FIGURE 11

Figure 11. Violin-plot representation of Detection Time (DT) curves for each visual paradigm across trials, channels, and stimulation frequency: (A) FBCCA, (B) CCA, (C) MEC. Significant differences between pairs of groups of visual paradigms are represented such that no asterisk above the bracket means “not significant.”

Finally, the across subjects, channels, and trial results of the ITR, with its corresponding standard deviation, are presented (Figure 12) considering the three detection methods, the four visual paradigms, and the five stimulation frequencies. This parameter has a non-linear proportional dependence on the previously reported accuracy and is inversely proportional with respect to the detection time. As in the accuracy results, the FBCCA outperforms the other SSVEP detection methods, however, it is the CBR visual paradigm for a frequency of 8.57Hz that provides the maximum overall ITR value (86.95 bpm), matching the shortest average detection time (Table 3). For the CBR and CBS visual paradigms at 20 and 24 Hz frequencies, null ITR values were obtained with the CCA and MEC methods because the accuracy values were below the chance level (20%). In the first two stimulation frequencies (8.57 and 10.909 Hz) the ITR is higher for the CBR and CBS paradigms because the detection time is shorter for these cases while the accuracy values are approximately the same.

FIGURE 12

Figure 12. Across all participants average ITR for each stimulation frequency, visual paradigm, and SSVEP detection method.

Figure 13 shows the violin plot representation of the distribution of ITR values for each visual paradigm (OOR, OOS, CBR, and CBS) across trials, channels, and stimulation frequencies, and for each of the three SSVEP detection methods (FBCCA, CCA, and MEC). Each plot shows the probability density of the ITR values, with the thicker parts representing regions with more data points. The violin plots show that the median corresponding to the distribution of the OOS visual paradigm is the highest in the three SSVEP detection methods, the same as for the DA parameter. The results of the Kruskal-Wallis and multiple comparison tests indicated that there were significant differences in ITR in ten out of the twelve experimental conditions. In the FBCCA and MEC methods, significant differences were identified between the OOS paradigm and the two visual stimulus variants involving the checkerboard pattern (CBR and CBS) respectively. However, unlike the behavior in estimating detection accuracy, this performance did not occur in the case of the OOR paradigm vs. CBR and CBS. In the case of the CCA method (Figure 13B), of the six possible combination options, no significant differences were observed between the medians of distributions belonging to the same visual pattern, i.e., OOR vs. OOS and CBR vs. CBS, but differences were observed between OOR vs. CBS, with a significance level of 0.01, between OOR vs. CBR, OOS vs. CBR and between OOS vs. CBS, the latter three comparisons with a significance level of 0.001.

FIGURE 13

Figure 13. Violin-plot representation of Information Transfer Rate (ITR) curves for each visual paradigm across trials, channels, and stimulation frequency: (A) FBCCA, (B) CCA, (C) MEC. Significant differences between pairs of groups of visual paradigms are represented with asterisks, such that ***p ≤ 0.001; **p ≤ 0.01; no asterisk above the bracket means “not significant”.

4. Discussion

The influence of different aspects of the visual stimulus (e.g., number of targets, stimuli frequency, the dimensions of the stimuli, stimuli colors, type of signal controlling the stimuli, number of targets, among others) on the SSVEP response and thus on BCI performance was previously reported. However, as discussed in Li et al. (2021), it is not only these parameters that influence performance but also the visual paradigm employed. It is for this reason that the objective we pursued with this work was to evaluate the performance in discriminating target from non-target elements taking into account four visual paradigms (CBR, CBS, OOS, and OOR) in conjunction with three SSVEP detection methods (CCA, FBCCA, and MEC) since, as stated in the same study, the results are inconsistent when the same SSVEP classification algorithms are applied but with different visual paradigms, demonstrating the importance of the latter in the satisfactory performance of the interface.

According to our findings, brain responses to visual stimuli showed higher average power peaks in the intervals comprising the first three stimulus frequencies (8.57, 10.909, and 15Hz). A similar range of stimulation frequencies has already been successfully used in other studies such as those conducted by Chen et al. (2015a) and Liu et al. (2020). To ensure a high number of targets in such a narrow range of stimulation frequencies, oscillatory stimuli consisting of combinations of frequencies and phase shifts could be incorporated, as proposed in Chen et al. (2015a) and Liu et al. (2020). The strongest brain response to visual stimuli is achieved when applying On-Off patterns and modulation of the visual stimulus is performed with a sinusoidal signal or with a rectangular pulse. The OOR and OOS visual paradigms were superior to CBR and CBS in power response in each of the channels where the signals were recorded, which is consistent with the results reported in Zerafa et al. (2013), although in this study, the type of stimulus modulation was not specified and only a flashing element was presented on the screen. It should be emphasized that the maximum average power peaks in the CBR and CBS paradigms were obtained for frequency values corresponding to twice the stimulation frequency f, where f ∈ ℝ; f:{8.57, 10.909, 15, 20, 24}Hz, which is consistent with the effect produced by the application of the checkerboard pattern. Our results partially disagree with those obtained in Teng et al. (2011) and Chen et al. (2019), in which the modulation of a pulsed signal with 50% duty cycle is superior to the modulation performed by a sinusoidal signal, although it should be highlighted that in the former publication the accuracy values associated with the application of a modulating signal type and not the power values, were reported. According to our findings, the dominance of one paradigm over another depends on the stimulation frequency and the stimulus type. The results also showed us that an association between the electrical brain response and the type of stimulus can be established, with a high level of confidence, but not with respect to the luminance modulating signal, since a pairwise comparison showed that there are differences between the OOR vs. CBR and CBS paradigms, that there are also differences between OOS vs. CBR and CBS, but no differences were found between the OOR paradigm with respect to OOS and CBS with respect to CBR. At the same time, this result partially matches with Cysewska-Sobusiak and Jukiewicz (2016), in which the application of the sinusoidal signal elicits the most intense brain reaction. However, three aspects should be noted here: first, the population sample reported in that study was 8 participants; second, only 2 electrodes of the international 10-20 system (O₁ and O₂) were used; and third, the quantitative parameter used to characterize the brain response according to the waveform of the stimulus modulating signal was the Signal-to-Noise Ratio (SNR).

In summary, our results suggest that, in the aspect related to the visual paradigms, brain responses are more prominent when subjects are exposed to an On-Off stimulation type combined with luminance modulation either by sinusoidal or rectangular signals. The results of the statistical test suggest that, in the visual paradigm, the visual pattern (pattern reversal checkerboard, on-off) is the determinant and not the luminance-modulating waveform (sinusoidal or rectangular) of the visual elements displayed on the screen.

Considering the SSVEP detection methods, the results, obtained by applying each of the three proposed methods to the four types of visual paradigms, showed a superiority of FBCCA over its other two contenders. This is an expected result, consistent with that reported by Ruiz-Olaya et al. (2019) and Liu et al. (2020) when comparing these same three detection methods.

It is noteworthy that as the stimulation frequency increases, the accuracy and ITR parameters decrease. In particular, the frequency of 10,909 Hz indicates maximum values of accuracy. This would be in contradiction with the previous statement since lower values should then be reported with respect to the frequency of 8.57 Hz. However, the explanation may be given by the fact that the stimulation frequency is coincidentally in the center of the alpha band spectrum. In terms of detection time, the CBR paradigm shows the lowest overall values for the stimulus frequencies 8.57 and 10.909 Hz in the three detection methods, and then, for the remaining stimulation frequencies (15, 20, and 24 Hz), the global maximum values of detection times are reached. We found no reasonable explanation for this behavior of detection times other than the influence of artifacts on the EEG data, the presence of alpha and beta activity, and a greater incidence of harmonics as the frequency increases. The consequence of this is that the ITR is maximum for the 8.57 Hz stimulation frequency in each of the three detection methods for the CBR paradigm because the detection time is minimal compared to the other visual paradigms and the accuracy rate is approximately similar. However, for the 10, 909 Hz frequency, the ITR experiences a decrease for the specific cases in which the MEC and CCA methods are applied in the CBR paradigm since, although the detection times remain at the minimum values compared to the ratings associated with the other visual paradigms, the accuracy decreases and consequently the ITR decreases. For the frequencies of 15, 20, and 24 Hz the detection times reach the global maximum values with the CBR paradigm, and at the same time, with the increase of the stimulation frequencies, the accuracy decreases (Regan, 1989; Pathiranage et al., 2018), even below the chance level at 20 and 24 Hz, which is why the ITR becomes null for the MEC detection method, which also applies to the CBS paradigm. Based on the results obtained, we can summarize that the FBCCA method outperforms the other two detection methods and that the best overall performances are obtained by combining this SSVEP detection method with the OOR or OOS paradigms.

In summary, the results suggest that the choice of visual stimuli paradigm can significantly impact the accuracy and ITR parameters of SSVEP detection and that some paradigms may be more effective than others depending on the specific detection method being used. Also, the detection time results suggest that the choice of visual stimuli paradigm may not significantly impact the speed of SSVEP detection. However, it is important to note that this analysis only assessed differences in average DT across the different paradigms and methods. Further analyses or post-hoc tests may be needed to investigate whether there are more subtle differences in DT that were not captured by this initial analysis.

Finally, the survey applied to evaluate the participants' comfort is not conclusive since the paradigms involving the checkerboard pattern (CBR and CBS) narrowly outperform those involving the On-Off type (OOR and OOS). However, the fact that the checkerboard pattern is preferred by users, to avoid visual fatigue and improve their focus, is fully in line with previous research reports mentioned next. The observation that 14 of 27 subjects selected checkerboard pattern stimulation as promoting greater visual comfort and focus is consistent with the contrasting explanations provided in several studies. On one hand, this result is aligned with previous publications such as in Duszyk et al. (2014), where it is postulated that visual stimuli that induce stronger SSVEP responses tend to generate greater visual fatigue (OOR and OOS). Additionally, Zheng et al. (2020) discusses how contrast changes in these visual paradigms can be somewhat intense compared to other visual stimuli such as the checkerboard pattern, leading to an increased demand for attention, which in turn derives in the occurrence of visual fatigue. However, as reported in Zerafa et al. (2013) and Choi et al. (2019b), the application of the checkerboard pattern elicits greater discomfort when compared to the On-Off pattern. As a result of these opposing viewpoints in the literature and the minimal prevalence of one visual paradigm over the other in our survey results, we reaffirm that our findings are consistent with this divided opinion on the subject matter and we suggest expanding the population sample in future experiments.

To achieve a balance between performance and comfort we propose the use of the OOS visual paradigm combined with the FBCCA in future instances, but with the modification that the applied stimulation frequencies be in the range of 15–24 Hz. This can be achieved by implementing the Joint Frequency-Phase Modulation (JFPM) method described in Chen et al. (2015b), thus allowing improved discriminability between SSVEPs responses over a narrow range of stimulation frequencies. We consider that the fact that the checkerboard paradigm is more propitious to achieve better visual focus and comfort, is not a strong enough argument to propose this visual pattern in future experiments considering that it outperforms the On-Off paradigm by only one participant. Another element that supports our proposal is the weak performance, manifested in accuracy and ITR values, obtained with this paradigm, mainly for frequencies higher than 15 Hz.

5. Conclusions

This study proposes a comprehensive analysis of the effect generated on the performance of an SSVEP-based BCI by simultaneously evaluating different visual stimulation paradigms and detection methods. In summary, and based on the results obtained, we can formulate the following statements: (i) On average, the signal-to-noise ratio is higher when an On-Off standard visual stimuli is modulated by sinusoidal (OOS) or rectangular (OOR) signals. This is particularly evident in the occipital channels; (ii) evidence indicates that stimulation with oscillating patterns modulated by sinusoidal or rectangular signals and standard scheme (OOS or OOR), when combined with the FBCCA method, leads to better performance; (iii) results also suggest that to achieve better performance, the frequencies of the visual stimuli should be between 8.5 and 15 Hz; (iv) however, this result is in contrast with the user's perception of comfort, since, according to the survey applied, the checkerboard stimulus pattern, whether modulated by a pulsed or sinusoidal signal (CBR or CBS) and at high frequencies (>20 Hz), favors a more pleasant visual experience and a lower incidence of ocular fatigue.

Our study suggests that there are combinations of visual paradigms with SSVEP detection methods that yield better performance in discriminating targets from non-target items with a pertinent level of confidence.

These results, including the ITR reports, inspire further research exploring stimulation schemes with mixed visual paradigms for specific frequencies implemented in the front-end application. Also, in our study, the superiority of the FBCCA method over MEC and CCA is evident, and therefore its comparison with other successful SSVEP detection methods such as Task-Related Component Analysis (TRCA), under the same visual stimulation variants, would be appropriate.

We provided a free and open-access database of electroencephalographic recordings of the response to four visual paradigms inducing the SSVEP phenomenon. The usefulness of this dataset is oriented to the implementation of new detection algorithms based on the SSVEP paradigm, mainly for a reduced number of elements to be detected, which is common in BCI applications, for example for navigation or remote control of mobility systems, such as robots or wheelchairs.

Data availability statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics statement

The studies involving human participants were reviewed and approved by School of Medicine's Ethics Committee in Investigation at Tecnologico de Monterrey (ITESM) and the School of Medicine's Committee of Investigation at Tecnologico de Monterrey (ITESM). The patients/participants provided their written informed consent to participate in this study.

Author contributions

JC, JA, RC, and OM-M: conceptualization. LH-R, JA, RC, and OM-M: methodology. OM-M: software. JC: conducted the experiments. JC, LH-R, JA, and OM-M: validation and data curation. LH-R, JA, and OM-M: resources. JC, LH-R, JA, RC, and OM-M: writing—original draft preparation. All authors contributed to the article and approved the submitted version.

Funding

This research has been funded by the National Council of Science and Technology of Mexico (CONACyT) through grant PN2015-873.

Acknowledgments

JC acknowledges the National Council of Science and Technology of Mexico (CONACyT) for the scholarship with CVU number 1011762.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fnins.2023.1142892/full#supplementary-material

References

Amiri, S., Rabbi, A., Azinfar, L., and Fazel-Rezai, R. (2013). “A review of P300, SSVEP, and hybrid P300/SSVEP brain-computer interface systems,” in Brain-Computer Interface Systems, ed R. Fazel-Rezai (Rijeka: IntechOpen), 1–20.

Google Scholar

Antelis, J. M., Rivera, C. A., Galvis, E., and Ruiz-Olaya, A. F. (2020). Detection of SSVEP based on empirical mode decomposition and power spectrum peaks analysis. Biocybern. Biomed. Eng. 40, 1010–1021. doi: 10.1016/j.bbe.2020.05.007

CrossRef Full Text | Google Scholar

Bian, R., Meng, L., and Wu, D. (2022). SSVEP-based brain-computer interfaces are vulnerable to square wave attacks. Sci. China Inform. Sci. 65, 140406. doi: 10.1007/s11432-022-3440-5

CrossRef Full Text | Google Scholar

Bockbrader, M. A., Francisco, G., Lee, R., Olson, J., Solinsky, R., and Boninger, M. L. (2018). Brain computer interfaces in rehabilitation medicine. PM R 10, S233–S243. doi: 10.1016/j.pmrj.2018.05.028

PubMed Abstract | CrossRef Full Text | Google Scholar

Burkitt, G. R., Silberstein, R. B., Cadusch, P. J., and Wood, A. W. (2000). Steady-state visual evoked potentials and travelling waves. Clin. Neurophysiol. 111, 246–258. doi: 10.1016/S1388-2457(99)00194-7

PubMed Abstract | CrossRef Full Text | Google Scholar

Chen, L., Chen, P., Zhao, S., Luo, Z., Chen, W., Pei, Y., et al. (2021). Adaptive asynchronous control system of robotic arm based on augmented reality-assisted brain–computer interface. J. Neural Eng. 18, 066005. doi: 10.1088/1741-2552/ac3044

PubMed Abstract | CrossRef Full Text | Google Scholar

Chen, X., Wang, Y., Gao, S., Jung, T.-P., and Gao, X. (2015a). Filter bank canonical correlation analysis for implementing a high-speed SSVEP-based brain–computer interface. J. Neural Eng. 12, 046008. doi: 10.1088/1741-2560/12/4/046008

PubMed Abstract | CrossRef Full Text | Google Scholar

Chen, X., Wang, Y., Nakanishi, M., Gao, X., Jung, T.-P., and Gao, S. (2015b). High-speed spelling with a noninvasive brain-computer interface. Proc. Natl. Acad. Sci. U.S.A. 112, E6058-67. doi: 10.1073/pnas.1508080112

PubMed Abstract | CrossRef Full Text | Google Scholar

Chen, X., Wang, Y., Zhang, S., Xu, S., and Gao, X. (2019). Effects of stimulation frequency and stimulation waveform on steady-state visual evoked potentials using a computer monitor. J. Neural Eng. 16, 066007. doi: 10.1088/1741-2552/ab2b7d

PubMed Abstract | CrossRef Full Text | Google Scholar

Choi, G.-Y., Han, C.-H., Jung, Y.-J., and Hwang, H.-J. (2019a). A multi-day and multi-band dataset for a steady-state visual-evoked potential-based brain-computer interface. GigaScience 8, giz133. doi: 10.1093/gigascience/giz133

PubMed Abstract | CrossRef Full Text | Google Scholar

Choi, K.-m., Park, S., and Im, C.-H. (2019b). Comparison of visual stimuli for steady-state visual evoked potential-based brain-computer interfaces in virtual reality environment in terms of classification accuracy and visual comfort. Comput. Intell. Neurosci. 2019, 9680697. doi: 10.1155/2019/9680697

PubMed Abstract | CrossRef Full Text | Google Scholar

Cysewska-Sobusiak, A., and Jukiewicz, M. (2016). Stimuli design for SSVEP-based brain-computer interface. Int. J. Electron. Telecommun. 62, 109–113. doi: 10.1515/eletel-2016-0014

CrossRef Full Text | Google Scholar

Duszyk, A., Bierzyńska, M., Radzikowska, Z., Milanowski, P., Kuś, R., Suffczyński, P., et al. (2014). Towards an optimization of stimulus parameters for brain-computer interfaces based on steady state visual evoked potentials. PLoS ONE 9, e112099. doi: 10.1371/journal.pone.0112099

PubMed Abstract | CrossRef Full Text | Google Scholar

Faller, J., Allison, B. Z., Brunner, C., Scherer, R., Schmalstieg, D., Pfurtscheller, G., et al. (2017). A feasibility study on SSVEP-based interaction with motivating and immersive virtual and augmented reality. arXiv preprint arXiv:1701.03981. doi: 10.48550/arXiv.1701.03981

CrossRef Full Text | Google Scholar

Farwell, L., and Donchin, E. (1988). Talking off the top of your head: toward a mental prosthesis utilizing event-related brain potentials. Electroencephalogr. Clin. Neurophysiol. 70, 510–523.

PubMed Abstract | Google Scholar

Friman, O., Volosyak, I., and Graser, A. (2007). Multiple channel detection of steady-state visual evoked potentials for brain-computer interfaces. IEEE Trans. Biomed. Eng. 54, 742–750. doi: 10.1109/TBME.2006.889160

PubMed Abstract | CrossRef Full Text | Google Scholar

Gao, S., Wang, Y., Gao, X., and Hong, B. (2014). Visual and auditory brain–computer interfaces. IEEE Trans. Biomed. Eng. 61, 1436–1447. doi: 10.1109/TBME.2014.2300164

PubMed Abstract | CrossRef Full Text | Google Scholar

Gao, X., Xu, D., Cheng, M., and Gao, S. (2003). A BCI-based environmental controller for the motion-disabled. IEEE Trans. Neural Syst. Rehabil. Eng. 11, 137–140. doi: 10.1109/TNSRE.2003.814449

PubMed Abstract | CrossRef Full Text | Google Scholar

Gembler, F., Stawicki, P., and Volosyak, I. (2016). “Exploring the possibilities and limitations of multitarget SSVEP-based BCI applications,” in 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) (Orlando, FL), 1488–1491.

PubMed Abstract | Google Scholar

Hardoon, D. R., Szedmak, S., and Shawe-Taylor, J. (2004). Canonical correlation analysis: an overview with application to learning methods. Neural Comput. 16, 2639–2664. doi: 10.1162/0899766042321814

PubMed Abstract | CrossRef Full Text | Google Scholar

Henry, R. (2015). Etymologia: Bonferroni correction. Emerg. Infect. Dis. J. 21, 289. doi: 10.3201/eid2102.ET2102

PubMed Abstract | CrossRef Full Text | Google Scholar

İşcan Z. and Nikulin, V. V.. (2018). Steady state visual evoked potential (SSVEP) based brain-computer interface (BCI) performance under different perturbations. PLoS ONE 13, e0191673. doi: 10.1371/journal.pone.0191673

PubMed Abstract | CrossRef Full Text | Google Scholar

Jiang, J., Wang, C., Wu, J., Qin, W., Xu, M., and Yin, E. (2020). Temporal combination pattern optimization based on feature selection method for motor imagery BCIs. Front. Hum. Neurosci. 14, 231. doi: 10.3389/fnhum.2020.00231

PubMed Abstract | CrossRef Full Text | Google Scholar

Lalor, E., Kelly, S., Finucane, C., Burke, R., Smith, J., Reilly, R., et al. (2005). Steady-state VEP-based brain-computer interface control in an immersive 3d gaming environment. EURASIP J. Adv. Sig. Proc. 2005, 3156–3164. doi: 10.1155/ASP.2005.3156

CrossRef Full Text | Google Scholar

Li, M., He, D., Li, C., and Qi, S. (2021). Brain–computer interface speller based on steady-state visual evoked potential: a review focusing on the stimulus paradigm and performance. Brain Sci. 11, 1–25. doi: 10.3390/brainsci11040450

PubMed Abstract | CrossRef Full Text | Google Scholar

Liang, L., Lin, J., Yang, C., Wang, Y., Chen, X., Gao, S., et al. (2020). Optimizing a dual-frequency and phase modulation method for SSVEP-based BCIs. J. Neural Eng. 17, 046026. doi: 10.1088/1741-2552/abaa9b

PubMed Abstract | CrossRef Full Text | Google Scholar

Liu, B., Huang, X., Wang, Y., Chen, X., and Gao, X. (2020). Beta: a large benchmark database toward SSVEP-BCI application. Front. Neurosci. 14, 627. doi: 10.3389/fnins.2020.00627

PubMed Abstract | CrossRef Full Text | Google Scholar

Liu, B., Wang, Y., Gao, X., and Chen, X. (2022). ELDbeta: a large eldercare-oriented benchmark database of SSVEP-BCI for the aging population. Sci. Data 9, 252. doi: 10.1038/s41597-022-01372-9

PubMed Abstract | CrossRef Full Text | Google Scholar

Lotze, M., and Cohen, L. G. (2006). Volition and imagery in neurorehabilitation. Cogn. Behav. Neurol. 19, 135–140. doi: 10.1097/01.wnn.0000209875.56060.06

PubMed Abstract | CrossRef Full Text | Google Scholar

McFarland, D. J., Lefkowicz, A. T., and Wolpaw, J. R. (1997). Design and operation of an EEG-based brain-computer interface with digital signal processing technology. Behav. Res. Methods Instrum. Comput. 29, 337–345. doi: 10.3758/BF03200585

CrossRef Full Text | Google Scholar

Mendoza-Montoya, O. (2018). Development of a hybrid brain-computer interface for autonomous systems (Ph.D. thesis). Free University of Berlin, Dahlem, Germany.

Google Scholar

Nakanishi, M., Wang, Y., and Jung, T.-P. (2018). Spatial Filtering Techniques for Improving Individual Template-Based SSVEP Detection. London: The Institution of Engineering and Technology.

PubMed Abstract | Google Scholar

Ng, K. B., Bradley, A. P., and Cunnington, R. (2012). Stimulus specificity of a steady-state visual-evoked potential-based brain–computer interface. J. Neural Eng. 9, 036008. doi: 10.1088/1741-2560/9/3/036008

PubMed Abstract | CrossRef Full Text | Google Scholar

Oostenveld, R., Fries, P., Maris, E., and Schoffelen, J.-M. (2010). Fieldtrip: open source software for advanced analysis of MEG, EEG, and invasive electrophysiological data. Comput. Intell. Neurosci. 2011, 156869. doi: 10.1155/2011/156869

PubMed Abstract | CrossRef Full Text | Google Scholar

Pathiranage, S., Paranawithana, I., Perera, M., and De Silva, A. C. (2018). “An in-depth study of SSVEP signals against stimulus frequency and distance to the stimulus,” in 2018 Moratuwa Engineering Research Conference (MERCon) (Moratuwa), 60–65.

Google Scholar

Pei, Y., Luo, Z., Yan, Y., Yan, H., Jiang, J., Li, W., et al. (2021). Data augmentation: using channel-level recombination to improve classification performance for motor imagery EEG. Front. Hum. Neurosci. 15, 645952. doi: 10.3389/fnhum.2021.645952

PubMed Abstract | CrossRef Full Text | Google Scholar

Pei, Y., Luo, Z., Zhao, H., Xu, D., Li, W., Yan, Y., et al. (2022). A tensor-based frequency features combination method for brain–computer interfaces. IEEE Trans. Neural Syst. Rehabil. Eng. 30, 465–475. doi: 10.1109/TNSRE.2021.3125386

PubMed Abstract | CrossRef Full Text | Google Scholar

Proakis, J. G., and Manolakis, D. K. (2006). Digital Signal Processing, 4th Edn. Prentice Hall.

Google Scholar

Regan, D. (1977). Steady-state evoked potentials. J. Opt. Soc. Am. 67, 1475–1489.

Google Scholar

Regan, D. (1989). Human Brain Electrophysiology : Evoked Potentials and Evoked Magnetic Fields in Science and Medicine. New York, NY: Elsevier Science.

Google Scholar

Ruiz-Olaya, A. F., Antelis, J. M., and Cerquera, A. (2019). “Decoding steady-state visual evoked potentials from EEG signals: towards an EEG-triggered fes system to restore hand grasp function,” in 2019 IEEE 10th Annual Ubiquitous Computing, Electronics & Mobile Communication Conference (UEMCON) (New York, NY), 1152–1157.

Google Scholar

Singla, R. (2018). “SSVEP-based BCIs,” in Evolving BCI Therapy, ed D. Larrivee (Rijeka: IntechOpen), 142–160.

Google Scholar

Siribunyaphat, N., and Punsawad, Y. (2022). Steady-state visual evoked potential-based brain-computer interface using a novel visual stimulus with quick response (QR) code pattern. Sensors 22, 1–18. doi: 10.3390/s22041439

PubMed Abstract | CrossRef Full Text | Google Scholar

Teng, F., Chen, Y., Choong, A. M., Gustafson, S., Reichley, C., Lawhead, P., et al. (2011). Square or sine: finding a waveform with high success rate of eliciting SSVEP. Comput. Intell. Neurosci. 2011, 364385. doi: 10.1155/2011/364385

PubMed Abstract | CrossRef Full Text | Google Scholar

Velasco-Álvarez, F., Fernandez-Rodrioguez, A., Vizcaino-Martin, F.-J., Diaz-Estrella, A., and Ron-Angevin, R. (2021). Brain–computer interface (BCI) control of a virtual assistant in a smartphone to manage messaging applications. Sensors 21, 3716. doi: 10.3390/s21113716

PubMed Abstract | CrossRef Full Text | Google Scholar

Volosyak, I. (2011). SSVEP-based bremen–BCI interface—boosting information transfer rates. J. Neural Eng. 8, 036020. doi: 10.1088/1741-2560/8/3/036020

PubMed Abstract | CrossRef Full Text | Google Scholar

Volosyak, I., Cecotti, H., Valbuena, D., and Graser, A. (2009). “Evaluation of the bremen SSVEP based BCI in real world conditions,” in 2009 IEEE International Conference on Rehabilitation Robotics (Kyoto), 322–331.

Google Scholar

Wang, Y., Gao, X., Hong, B., Jia, C., and Gao, S. (2008). Brain-computer interfaces based on visual evoked potentials. IEEE Eng. Med. Biol. Mag. 27, 64–71. doi: 10.1109/MEMB.2008.923958

PubMed Abstract | CrossRef Full Text | Google Scholar

Wang, Y., Wang, R., Gao, X., Hong, B., and Gao, S. (2006). A practical VEP-based brain-computer interface. IEEE Trans. Neural Syst. Rehabil. Eng. 14, 234–240. doi: 10.1109/TNSRE.2006.875576

PubMed Abstract | CrossRef Full Text | Google Scholar

Zerafa, R., Camilleri, T., Falzon, O., and Camilleri, K. P. (2013). “Comparison of plain and checkerboard stimuli for brain computer interfaces based on steady state visual evoked potentials,” in 2013 6th International IEEE/EMBS Conference on Neural Engineering (NER) (San Diego, CA), 33–36.

Google Scholar

Zhang, Y., Zhou, G., Jin, J., Wang, X., and Cichocki, A. (2014). Frequency recognition in SSVEP-based BCI using multiset canonical correlation analysis. Int. J. Neural Syst. 24, 1450013. doi: 10.1142/S0129065714500130

PubMed Abstract | CrossRef Full Text | Google Scholar

Zheng, X., Xu, G., Zhang, Y., Liang, R., Zhang, K., Du, Y., et al. (2020). Anti-fatigue performance in SSVEP-based visual acuity assessment: a comparison of six stimulus paradigms. Front. Hum. Neurosci. 14, 301. doi: 10.3389/fnhum.2020.00301

PubMed Abstract | CrossRef Full Text | Google Scholar

Zhu, D., Bieger, J., Garcia Molina, G., Aarts, R. M., and Vialatte, F. (2010). A survey of stimulation methods used in SSVEP-based BCIs. Comput. Intell. Neurosci. 2010, 702357. doi: 10.1155/2010/702357

PubMed Abstract | CrossRef Full Text | Google Scholar

Zhu, F., Jiang, L., Dong, G., Gao, X., and Wang, Y. (2021). An open dataset for wearable SSVEP-based brain-computer interfaces. Sensors 21, 1–17. doi: 10.3390/s21041256

PubMed Abstract | CrossRef Full Text | Google Scholar