A flexible speller based on time-space frequency conversion SSVEP stimulation paradigm under dry electrode

Introduction: Speller is the best way to express the performance of the brain-computer interface (BCI) paradigm. Due to its advantages of short analysis time and high accuracy, the SSVEP paradigm has been widely used in the BCI speller system based on the wet electrode. It is widely known that the wet electrode operation is cumbersome and that the subjects have a poor experience. In addition, in the asynchronous SSVEP system based on threshold analysis, the system flickers continuously from the beginning to the end of the experiment, which leads to visual fatigue. The dry electrode has a simple operation and provides a comfortable experience for subjects. The EOG signal can avoid the stimulation of SSVEP for a long time, thus reducing fatigue.

Methods: This study first designed the brain-controlled switch based on continuous blinking EOG signal and SSVEP signal to improve the flexibility of the BCI speller. Second, in order to increase the number of speller instructions, we designed the time-space frequency conversion (TSFC) SSVEP stimulus paradigm by constantly changing the time and space frequency of SSVEP sub-stimulus blocks, and designed a speller in a dry electrode environment.

Results: Seven subjects participated and completed the experiments. The results showed that the accuracy of the brain-controlled switch designed in this study was up to 94.64%, and all the subjects could use the speller flexibly. The designed 60-character speller based on the TSFC-SSVEP stimulus paradigm has an accuracy rate of 90.18% and an information transmission rate (ITR) of 117.05 bits/min. All subjects can output the specified characters in a short time.

Discussion: This study designed and implemented a multi-instruction SSVEP speller based on dry electrode. Through the combination of EOG and SSVEP signals, the speller can be flexibly controlled. The frequency of SSVEP stimulation sub-block is recoded in time and space by TSFC-SSVEP stimulation paradigm, which greatly improves the number of output instructions of BCI system in dry electrode environment. This work only uses FBCCA algorithm to test the stimulus paradigm, which requires a long stimulus time. In the future, we will use trained algorithms to study stimulus paradigm to improve its overall performance.

1. Introduction

Brain-computer interface (BCI) is currently a research hotspot in the multidisciplinary cross-field (Abiri et al., 2019; Mcfarland, 2020; Pan et al., 2020), which aims to establish a direct communication channel between the brain and external devices without relying on peripheral nerve and muscle tissue (Zhou et al., 2014; Edelman et al., 2019; Li et al., 2019; Jin et al., 2020; Lu et al., 2020; Ge et al., 2021). At present, the BCI system is widely used in medical rehabilitation, mechanical control, speller, etc. BCI speller is the best way to express the performance of the BCI paradigm (Kapgate et al., 2020; Li et al., 2021a). Many researchers have attempted to implement multi-instruction BCI spellers to improve the information transmission rate (ITR) of the BCI system (Rezeika et al., 2018; Xu et al., 2020; Li et al., 2021a; Kundu and Ari, 2022). Therefore, it is of great significance for the development of BCI to study the speller that outputs multiple instructions in a short time. However, considering the actual use of patients with motion disorders, the system design should improve the convenience of patients while ensuring the accuracy of instructions.

The BCI speller is a typical visual application of the BCI and was among the earliest implementations of the concept. Steady-state visual-evoked potential (SSVEP) (Zhang and Chen, 2022) is often used to design spellers because of its stable evoked characteristics and high signal-to-noise ratio. Nakanishi et al. proposed an SSVEP speller based on an FPHC paradigm to design the speller (Nakanishi et al., 2014). Xu et al. designed a 108 characters speller based on the hybrid paradigm of P300 and SSVEP in 2020 (Xu et al., 2020). In 2021, Ge et al. designed a 48-instruction speller based on the dual frequency SSVEP-biased coding paradigm (Ge et al., 2021). Although there are many speller studies on SSVEP, those speller systems can only be completed in the laboratory environment using wet electrodes, which are cumbersome and inflexible, and the subject experience is poor.

The dry electrode (Li et al., 2021b) based on high comfort and good portability has become the research focus of various BCI systems. In 2016, Chi Chun Lo et al. designed a new non-contact 12-instruction SSVEP control system, allowing disabled patients to activate the nurse emergency call system and adjust other equipment (Lo et al., 2016). In 2018, Xiao et al. used claw-shaped flexible dry electrodes to design and implement a speller with 12 targets (Xing et al., 2018), with an average accuracy of 93.2%, which is convenient for the subjects to complete character output in any environment. At present, although there are studies based on dry electrode SSVEP, the research is not mature enough and there are few output instructions.

In recent years, many researchers have begun to design and implement BCI systems based on asynchronous SSVEP (Pfurtscheller et al., 2010; Diez et al., 2011) in order to improve the flexibility of the system. Some studies use the threshold value of the stimulus flicker as the standard (Pan et al., 2013; Zhou et al., 2020). When the stimulus flickers continuously since the beginning of the experiment (including the control and idle state), the long-time flicker will make the subjects feel strong fatigue. Because the EOG signal is easy to be detected, it is widely concerned. Some studies combine EOG with MI to reflect the intention of the subjects and send commands to external devices. In 2017, He et al. proposed a hybrid BCI based on MI and EOG signals to operate web browsers (He et al., 2017). In 2019, Huang et al. used EOG for button selection and MI for direction control to integrate the control of the wheelchair robot arm system (Huang et al., 2019). Although EOG performs well in the MI paradigm, there are few studies that combine EOG with SSVEP and even fewer studies that use EOG to improve the flexibility of the SSVEP system.

This study designed and implemented a speller that can output multiple instructions flexibly. We designed a brain-controlled switch based on EOG and SSVEP. By using the brain-controlled switch to wake up and turn off the flashing stimulus, no external stimulus is needed in the idle state, which reduces the visual fatigue caused by flashing and is more consistent with the perception of the idle state. In addition, this study designed a time-space frequency conversion (TSFC) SSVEP stimulation paradigm based on the Neuracle 24-channel dry electrode and increased the number of output instructions within the limited frequency range by constantly changing the frequency of SSVEP sub-stimulation block space in two stages. First, we divided the stimulus interface into eight stimulus regions, which can accurately identify the target stimulus region in a short time. Second, the spatial local flicker designed to reduce the influence of irrelevant stimulus blocks as much as possible and effectively improve the accuracy of stimulus block recognition. The speller system based on our proposed TSFC-SSVEP paradigm can output 60 characters accurately. The main contribution of this study is to design a flexible speller combined with a brain-controlled switch, which provides flexibility and comfort. This study proposed the TSFC-SSVEP stimulation paradigm that can make speller output multiple instructions under the dry electrode. The study provides a new idea for the speller based on the brain-controlled system.

2. Methods

2.1. Participants

In this study, seven healthy subjects (aged 25.14 ± 0.98 years) volunteered to participate in offline and online experiments, and all subjects had normal vision or corrected vision. The experiment was conducted in a quiet laboratory. In addition, before each experiment, the experimenter informed the subjects of SSVEP-related theoretical knowledge and precautions for looking at the screen. All subjects had informed consent to the experimental study, and the study was approved by the Ethics Committee of the Taiyuan University of Technology.

2.2. Data acquisition

The dry electrode experiment part of this study uses the Neuracle 24-channel dry electrode, which is designed as a claw-like structure. The electrode tip is coated with a silver/silver chloride mixture of conductive ink to improve the electrochemical performance. The structure and material characteristics make it lightweight and wear-resistant. The wet electrode experiment part uses the Brain Products GmbH 32 moisture-conducting electrode made in Germany. The dry electrode can pass through the hair and contact the scalp well, and the correlation with the wet electrode signal is >90%.

The EEG signal sampling rate is 300 Hz. Six electrodes in the parietal occipital region (P3, P4, T5, T6, O1, and O2) and four electrodes in the frontal lobe (Fp1, Fp2, F7, and F8) were recorded. The reference electrode is located at the top of the head (PFz), and the grounding electrodes are the A1 and A2 of the earlobe. The impedance of all electrodes is < 50 kΩ.

2.3. Paradigms design

As shown in Figure 1, this paradigm is divided into two stages. First, the brain-controlled switch is designed to control the system flexibly. Second, the design is based on the TSFC-SSVEP stimulation paradigm to increase the number of BCI instruction codes in the dry electrode environment. Finally, the two are integrated to design a flexible and comfortable speller for the subjects.

FIGURE 1

Figure 1. Experimental technology roadmap.

2.3.1. Design of brain-controlled switch

The brain-controlled switch is composed of EOG and SSVEP. The EOG in the brain-controlled switch is used to wake up stimulus because the EOG is easy to detect. EOG is the depolarization and hyperpolarization between the retina and cornea caused by different eye movements, which forms a potential difference between the retina and cornea, and its amplitude is greater than that of EEG and background physiological signals (Crea et al., 2018; Huang et al., 2019; Zhu et al., 2020). Therefore, EOG can be easily and accurately detected by using several electrodes around the eyes. When the subject blinks continuously, we consider that the subject intends to wake up the stimulus. The SSVEP in the brain-controlled switch is used to turn off system stimulation. When the subjects are looking at the “Stop” SSVEP stimulus block in the stimulation keyboard, the stimulation can be turned off.

As shown in Figure 2, the system will first enter the switch mark background, which lasts for 1 s. This background is a sign to prompt the subject to open the stimulus. Then, the system will switch to a static background, with a duration of 3 s. Within 3 s, the subject can complete the operation of whether to wake up the stimulus according to their own intentions. If the subject blinks three times or more, the system switches to the stimulation keyboard interface, and vice versa. The three blinks here are the thresholds we set, mainly because the normal blinking of the subjects is excluded. Under normal conditions, the subjects also have a high probability of having 1–3 physiological blinks in a 3-s period (Zhu et al., 2020). Finally, after the subject wakes up the stimulation keyboard, he/she can output the required characters according to his/her intention. When all characters are output, the subject looks at the “Stop” SSVEP stimulus block on the stimulation keyboard to close the stimulation keyboard, and the system will switch to the background of the switch mark.

FIGURE 2

Figure 2. Schematic diagram of the brain-controlled switch.

2.3.2. Design of TSFC-SSVEP stimulation paradigm

As shown in Figure 3, the stimulation time of the TSFC-SSVEP paradigm in this study is 2 s. In the offline experiment, the character cue time is 0.5 s (the red square is marked with prompted characters), the stimulus shift time is 0.5 s, and the character feedback time is 0.5 s, with a total of 3.5 s. In the online experiment, the time of stimulus shift was 0.5 s and the time of character feedback was 0.5 s, with a total of 3 s. The specific stimulation is divided into two stages. The duration of the first stage stimulation is 1 s. As shown in Figure 4, the stimulation interface is divided into eight large modules corresponding to the frequency ranging from 10 to 17 Hz of the sine wave. The reasons for the selection of the 10–17 Hz range are specified in the results section. The spatial range of the subjects' gaze can be determined by analyzing the frequency of the modules within 1 s. The 1–1.5 s is the stimulus conversion process, which feeds back the spatial position analyzed in the first stage to the interface and transfers to the second stage of the stimulus interface. As shown in Figure 5, the 1.5–2.5 s is the second-stage stimulus. The stimulus is frequency coded again in each module of the first stage. The stimulus process is only carried out within the scope of the first-stage analysis, and other modules remain unchanged. As shown in Figure 3, the gray area is that selected by the subject in the first stage. Among them, compared with medium frequency (15–30 Hz) and high frequency (>30 Hz) stimuli, medium and low frequency (about 15 Hz) stimuli can induce stronger SSVEP responses. Therefore, in this paradigm, the stimulation frequency range of the stimulation flicker block is selected as the medium- and low-frequency band.

FIGURE 3

Figure 3. TSFC-SSVEP stimulation paradigm timeline.

FIGURE 4

Figure 4. Schematic diagram of the first stage of TSFC-SSVEP stimulation paradigm.

FIGURE 5

Figure 5. Schematic diagram of the second stage of the TSFC-SSVEP stimulation paradigm.

The frequency conversion combination of SSVEP stimuli in time and space is carried out according to the above two stages, which can encode different instructions, greatly improving the performance of the SSVEP paradigm. Moreover, the signal-to-noise ratio of feature signals is high, and the requirement for display is low, which is convenient for target recognition.

2.4. Signal processing

In this article, the features extracted by the speller are divided into two categories, one is EOG features, and the other is the SSVEP feature of EEG. Among them, the EOG feature extraction is divided into three steps. First, we use 0.5–10 Hz band-pass filtering for preprocessing. Then, the data of the four channels (Fp1, Fp2, F7, and F8) are averaged. Finally, the peak value above the 0.07 mV threshold is extracted by using the signal-peak search algorithm. In the signal-peak search algorithm, a peak or local maximum is defined as any sample whose two direct neighbors have a smaller amplitude. When the number of peaks is >3, it can be judged as conscious blinking, otherwise, it is normal. In addition, the feature extraction of SSVEP in EEG is divided into two steps. The first is preprocessing, i.e., 10–90 Hz band-pass filtering, 50 Hz notch processing, and ICA artifact removal. The second is to classify the filter bank canonical correlation analysis (FBCCA) (Chen et al., 2015a). First, the FBCCA is used to analyze the stimulus block region that the subject is currently looking at, then the FBCCA is used to analyze the specific stimulus block of the subject's gaze region, and finally, the feature fusion analysis is performed to obtain the final character output result.

Filter bank analysis uses multiple filters with different passbands for sub-band decomposition. A zero-phase Chebyshev I infinite pulse filter is used to extract the sub-band componentX_SBn(n = 1, 2, …, N) from the original EEG signal X. After the filter bank analysis, the standard Canonical Correlation Analysis (CCA) (Lin et al., 2007; Bin et al., 2009; Chen et al., 2014) process is applied to each sub-band component, respectively, and the correlation values between the sub-band component and the predefined reference signals corresponding to all stimulus frequencies Y_fk are obtained. For the kth reference signal, a correlation vector ρ_k composed of N correlation values is defined as follows:

$\begin{array}{ll}\left[\begin{array}{c}{\rho }_k^1\\ {\rho }_k^2\\ ⋮\\ {\rho }_k^N\end{array}\right]=\left[\begin{array}{c}\rho \left[X_{S{B}_1}^T{W}_X(X_{S{B}_1}{Y}_{fk}),Y^T{W}_Y(X_{S{B}_1}{Y}_{fk})\right]\\ \rho \left[X_{S{B}_2}^T{W}_X(X_{S{B}_2}{Y}_{fk}),Y^T{W}_Y(X_{S{B}_2}{Y}_{fk})\right]\\ ⋮\\ \rho \left[X_{S{B}_N}^T{W}_X(X_{S{B}_N}{Y}_{fk}),Y^T{W}_Y(X_{S{B}_N}{Y}_{fk})\right]\end{array}\right]& (1)\end{array}$

In formula (1), ρ(x, y) represents the correlation coefficient between x and y. The weighted sum of squares of the correlation values of all corresponding sub-band components is calculated as follows:

$\begin{array}{ll}{\tilde{\rho }}_k={\displaystyle \sum _{n=1}^N}\omega (n){({\rho }_k^n)}^2& (2)\end{array}$

In formula (2), n represents the index of the sub-band; ω(n) is the weight of the sub-band component, and the calculation formula is given as follows:

$\begin{array}{ll}\omega (n)=n^{-a}+b,n\in \left[1,N\right]& (3)\end{array}$

In formula (3), a and b are constants that maximize the classification performance, which can be determined by using the grid search method in offline analysis. For all stimulus frequencies$({\tilde{\rho }}_1,{\tilde{\rho }}_2,\dots ,{\tilde{\rho }}_N)$, ${\tilde{\rho }}_k$ is used to determine the frequency of SSVEP. When ${\tilde{\rho }}_k$ is the maximum, the frequency of the reference signal is the frequency of SSVEP.

Compared with standard CCA, the filter group analysis in FBCCA can decompose SSVEP into multiple sub-band components, so as to extract the discriminant information in the harmonic components of SSVEP. Therefore, FBCCA provides richer and more robust harmonic information for SSVEP target recognition and has a better recognition effect.

Based on previous research (Chen et al., 2015a), this article selects the best M3 sub-band division FBCCA method. The specific process is as follows:

Step 1: Divide the effective frequency band (10–90 Hz) of the signal into 10 segments (10–90, 18–90, 26–90, 34–90, 42–90, 50–90, 58–90, 66–90, 74–90, and 82–90 Hz);

Step 2: Pass the collected EEG signals through the above 10 band-pass filters, respectively;

Step 3: Substitute the 10 groups of EEG signals into the 10 standard CCAs, and then calculate the maximum correlation coefficient sum through the weight adjustment formula. The frequency corresponding to the maximum correlation coefficient sum is recorded as the prediction frequency.

3. Results

3.1. Brain-controlled switch experiment classification results

Brain-controlled switch experiment is divided into the offline experiment and the online experiment. In the experiment on EOG signal wake-up system stimulation, the online experiment and offline experiment were divided into four rounds, each round had 10 consecutive blinks and 10 normal states. Through the analysis and processing of the data obtained from the offline experiment, we set the corresponding signal peak threshold to 0.07 mV and the peak frequency threshold to 3 times. As shown in Table 1, seven subjects conducted offline online experiments. Among them, the accuracy rate of three subjects in the offline experiment was as high as 100.00%, and the accuracy rate of only one subject was 85.00%, with an average accuracy rate of 95.89%. Under the threshold determined by the offline experiment, the two subjects performed well in the online experiment, with an accuracy rate of 100.00% and an average accuracy rate of 94.64%, which fully conforms to our expected experimental hypothesis. In the experiment for the SSVEP signal to turn off the system stimulus, we selected the accuracy of seven subjects in 60 stimulus blocks of the entire stimulate keyboard (refer to the “TSFC-SSVEP Speller Classification Results” section for the specific experiment process). The seven subjects can complete the switching operation of the system stimulus according to their own intentions, which also conforms to our expected experimental hypothesis. The main purpose of the brain-controlled switch is to make the subject more autonomous in the process of SSVEP stimulation. In addition, the brain-controlled switch can avoid the visual fatigue of the subject caused by the long-time flicker after the SSVEP stimulation is turned on.

TABLE 1

Table 1. Classification accuracy of the brain-controlled switch experiment.

3.2. Determination of SSVEP characteristic frequency and time window

In this experiment, dry and wet electrodes were compared in the time domain and frequency domain, respectively. As shown in Figure 6A, in the time domain, after the dry electrode and wet electrode occipital leads are superposed and averaged, respectively, the signals are processed by 10–50 Hz band-pass filtering, and the two signals can be basically matched. As shown in Figure 6B, on the power spectrum, after the dry electrode and wet electrode pillow leads are superposed and averaged, respectively, the signals are Fourier transformed. The characteristics of both signals at the fundamental frequency signal and the harmonic signal are obvious. In conclusion, the dry electrode conditions used in this experiment basically meet the experimental requirements, and subsequent experiments are carried out based on dry electrodes.

FIGURE 6

Figure 6. Dry and wet electrodes: (A) time domain comparison diagram and (B) frequency domain comparison diagram.

Table 2 shows the recognition accuracy of seven subjects using dry electrodes for eight categories of SSVEP (8–15 Hz) at different time windows. Among them, the accuracy rate of only two subjects in 0.4 s was more than 90.00%, while in 1 s, all subjects reached 90.00%, and the average accuracy rate was 95.61%. The results in Table 2 show that the recognition accuracy increases with the increase of the time window, that is, the larger the time window is. It can achieve good results for the eight classification results in ~1 s, which provides a reliable experimental basis for the TSFC-SSVEP experiment.

TABLE 2

Table 2. Recognition accuracy of dry electrodes in different time windows experiment.

In the process of frequency selection, we designed the online experiment of TSFC-SSVEP in the 8–15 Hz frequency band, but the experimental results were not ideal. According to the feedback results of the subjects after the experiment, we made further analysis. The pixel size of 60 stimulus blocks is much smaller than that of 8 stimulus blocks, which results in reducing the induced SSVEP characteristics. In addition, the subjects found that higher frequency (>10 Hz) stimuli could attract the subjects' attention. This led to the involuntary transfer of subjects' attention from a low-frequency stimulus to a high-frequency stimulus when they looked at the stimulus block in the second stage, which affected the experimental results. Hence, we designed online experiments to screen the frequencies from 10 to 20 Hz, respectively, and 7 subjects focused on 60 stimulus blocks, and finally, 10–17 Hz is determined as a better frequency range. Table 3 shows the classification results of the TSFC-SSVEP online experiment of 7 subjects in 8–15 and 10–17 Hz bands, respectively. In this experiment, in order to avoid the influence of the time window on the experimental results, we used a 2-s stimulation time window in both stimulation stages. The results showed that the performance of the seven subjects under the stimulation paradigm of TSFC-SSVEP realized in the higher frequency band was higher than that in the lower frequency band. Therefore, we determined that the frequency range of TSFC-SSVEP was 10-17 Hz.

TABLE 3

Table 3. Online results of TSFC-SSVEP stimulation paradigm at two different frequencies.

3.3. TSFC-SSVEP speller classification results

The data from 7 healthy subjects were collected in this experiment. In the offline experiment, each subject selects 60 characters in each round, that is, 60 trials and the duration of a trial is 3.5 s. During the online experiment, 10 random characters specified by the experimenter shall prevail, that is, 10 trials, with a trial duration of 3 s. Each subject conducted 4 rounds of experiments offline and online, namely, 4 blocks, and the final result was the average of offline and online results. Table 4 shows the accuracy and ITR of 7 subjects using the FBCCA classification algorithm under the TSFC-SSVEP stimulation paradigm. It can be seen from Table 4 that the accuracy of most subjects in the offline experiment is more than 90.00%, the average accuracy of the offline experiment is 90.18%, while the average accuracy of the online experiment is 92.14%, with a difference of 2.00%. We analyzed the reasons and combined them with the feedback information of the subjects, mainly because the number of 60 stimulus targets in a round was large, and the long-term stimulus flickering led to visual fatigue of the subjects, which ultimately affected the experimental results.

TABLE 4

Table 4. Classification results based on TSFC-SSVEP in FBCCA algorithm.

3.4. Different paradigms comparison results

In this experiment, the TSFC-SSVEP paradigm and the current popular Joint Frequency Phase Modulation (JFPM)-SSVEP (Chen et al., 2015b) paradigm were analyzed and compared offline under the same time window for the same subjects. Table 5 shows the offline accuracy of character recognition of 7 subjects using the FBCCA algorithm under 2 different paradigms. JFPM-SSVEP is not as effective as it is in a wet electrode environment. The accuracy rate of the 2 subjects was lower than 80.00%, and there was a large difference between different subjects. The TSFC-SSVEP proposed in this article has a stable effect in the dry electrode environment. Only two subjects have a result lower than 90.00%, and there is less difference between different subjects. The results showed that compared with the JFPM-SSVEP paradigm, the TSFC-SSVEP stimulus paradigm had more advantages in the case of more instructions in the dry electrode environment, and its stability, accuracy, and ITR were higher than those of JFPM-SSVEP.

TABLE 5

Table 5. Classification results of FBCCA algorithm under different stimulus paradigms.

4. Discussion

This study designed and implemented a multi-instruction SSVEP speller based on dry electrodes. At the same time, through the combination of EOG and SSVEP signals in BCI, the speller can be flexibly controlled. The TSFC-SSVEP stimulation paradigm proposed in this study recodes the frequency of SSVEP stimulus sub-blocks in time and space, reduces the influence of surrounding stimulus blocks by using local flicker, performs well under dry electrode testing, and greatly improves the output command of the BCI system. Compared with synchronous BCI, the system uses a brain-controlled switch based on EOG and SSVEP to wake up or turn off stimulus flicker to complete character output, which makes the system more flexible and convenient.

As for asynchronous BCI systems, the previous research on asynchronous SSVEP-based BCI mainly uses traditional threshold methods to distinguish between the control state and the idle state (Pfurtscheller et al., 2010). In these studies, even if the subject is in an idle state, the stimulus in the stimulus interface will flicker continuously from the beginning of the experiment, which is easy to cause visual fatigue in the subject. Several researchers designed novel methods to improve the performance of the asynchronous SSVEP-based BCI. Pfurtscheller et al. used an MI-based brain switch to achieve the self-paced operation of an SSVEP-based orthosis control system. Tomita et al. proposed a bimodal BCI using simultaneously NIRS and EEG signals to estimate whether the subject is in idle or active mode (Tomita et al., 2014). In this study, we used EOG in combination with SSVEP to awaken or turn off systemic stimuli. The subject wakes up or closes the asynchronous operation based on the TSFC-SVEP speller according to their intention. Compared with asynchronous systems using threshold criteria, BCI systems based on EOG and SSVEP do not need to continuous flashing in idle state. In the idle state, no stimulation can alleviate the visual fatigue of the subjects. In addition, compared to MI-based switches, brain-controlled switches based on EOG and SSVEP have the advantage of short response time, which can accurately distinguish between the control state and idle state in a short time, and effectively improve system performance. But the brain-controlled switch based on EOG and SSVEP also has its limitations. For different subjects, the main challenge is that the system needs to collect a round of subjects' blink data in advance because different subjects have different blink ranges. Therefore, future work needs to design a more robust algorithm to classify the blinks of different subjects.

For the BCI classification based on TSFC-SSVEP, we have only tested the FBCCA algorithm. However, relying on FBCCA no-training algorithm analysis requires a time window of at least 1 s, which limits the performance of the stimulus paradigm. According to recent research, TRCA (Jin et al., 2021; Bian and Wu, 2022) and FBDNN (Bassi and Attux, 2022) can shorten the time window of stimulus data to 0.5 s, which will better improve the performance of the stimulus paradigm. We will also study the corresponding algorithm in the future.

5. Conclusion

In this article, a speller based on the brain-controlled switch and TSFC-SVEP stimulation paradigm is designed using a dry electrode. The brain-controlled switch controls the wake-up or turn off of the system stimulus by capturing the blink state of the subject and whether the subject looks at the “Stop” SSVEP stimulus block in the stimulus keyboard, which greatly improves the flexibility of the system. At the same time, the TSFC-SSVEP experimental stimulation paradigm proposed in this article greatly increases the number of instructions in the stimulation paradigm by combining SSVEP sub-stimulation modules in time and space and uses the idea of local stimulus block flashing to avoid irrelevant influence, which improves the recognition accuracy in a short time. The results show that the EOG recognition accuracy of the brain-controlled switch designed in this study is as high as 94.64%, and all subjects can flexibly use the speller. The speller based on TSFC-SSVEP can output 60 characters with an accuracy rate of 90.18%, It opens the way for the portable and comfortable dry electrode BCI system in the future.

Data availability statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics statement

The studies involving human participants were reviewed and approved by Ethics Committee of Taiyuan University of Technology. The patients/participants provided their written informed consent to participate in this study. Written informed consent was obtained from the individual(s) for the publication of any potentially identifiable images or data.

Author contributions

DL: conceived the experiments. ZZ: designed the experiments and wrote the manuscript. JX and XC: provided guidance to the counterpart manuscript. All authors contributed to the article and approved the submitted version.

Funding

This research was supported by the National Natural Science Foundation of China (Grant Nos: 61906130, 62176177, and 61873178).

Acknowledgments

The authors would like to thank the volunteers who participated in the BCI experiments.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fncom.2023.1101726/full#supplementary-material

References

Abiri, R., Borhani, S., Sellers, E. W., Jiang, Y., and Zhao, X. (2019). A comprehensive review of EEG-based brain-computer interface paradigms. J. Neural Eng. 16, 1–21. doi: 10.1088/1741-2552/aaf12e

PubMed Abstract | CrossRef Full Text | Google Scholar

Bassi, P., and Attux, R. (2022). FBDNN: filter banks and deep neural networks for portable and fast brain-computer interfaces. Biomed. Phys. Eng. Express 8, 035018. doi: 10.1088/2057-1976/ac6300

PubMed Abstract | CrossRef Full Text | Google Scholar

Bian, R., and Wu, D. (2022). Overview of the winning approaches in BCI controlled robot contest in world robot contest 2021: calibration-free SSVEP. Brain Sci. Adv. 8, 99–110. doi: 10.26599/BSA.2022.9050002

CrossRef Full Text | Google Scholar

Bin, G., Gao, X., Yan, Z., Hong, B., and Gao, S. (2009). An online multi-channel SSVEP-based brain–computer interface using a canonical correlation analysis method. J. Neural Eng. 6, 046002. doi: 10.1088/1741-2560/6/4/046002

PubMed Abstract | CrossRef Full Text | Google Scholar

Chen, X., Chen, Z., Gao, S., and Gao, X. (2014). A high-itr ssvep-based bci speller. Brain-Comput. Inter. 1, 181–191. doi: 10.1080/2326263X.2014.944469

CrossRef Full Text | Google Scholar

Chen, X., Wang, Y., Gao, S., Jung, T.-P., and Gao, X. (2015a). Filter bank canonical correlation analysis for implementing a high-speed SSVEP-based brain–computer interface. J. Neural. Eng. 12, 046008. doi: 10.1088/1741-2560/12/4/046008

PubMed Abstract | CrossRef Full Text | Google Scholar

Chen, X., Wang, Y., Nakanishi, M., Gao, X., Jung, T.-P., and Gao, S. (2015b). High-speed spelling with a noninvasive brain–computer interface. Proc. Natl. Acad. Sci. USA. 112, E6058–E6067. doi: 10.1073/pnas.1508080112

PubMed Abstract | CrossRef Full Text | Google Scholar

Crea, S., Nann, M., Trigili, E., Cordella, F., Baldoni, A., Badesa, F. J., et al. (2018). Feasibility and safety of shared EEG/EOG and vision-guided autonomous whole-arm exoskeleton control to perform activities of daily living. Sci. Rep. 8, 10823. doi: 10.1038/s41598-018-29091-5

PubMed Abstract | CrossRef Full Text | Google Scholar

Diez, P. F., Mut, V. A., Avila Perona, E. M., and Laciar Leber, E. (2011). Asynchronous BCI control using high-frequency SSVEP. J. Neuroeng. Rehabil. 8, 39. doi: 10.1186/1743-0003-8-39

PubMed Abstract | CrossRef Full Text | Google Scholar

Edelman, B., Meng, J., Suma, D., and Zurn, C. (2019). Noninvasive neuroimaging enhances continuous neural tracking for robotic device control. Sci. Robot. 4, eaaw6844. doi: 10.1126/scirobotics.aaw6844

PubMed Abstract | CrossRef Full Text | Google Scholar

Ge, S., Jiang, Y., Zhang, M., Wang, R., Iramina, K., Lin, P., et al. (2021). SSVEP-based brain-computer interface with a limited number of frequencies based on dual-frequency biased coding. IEEE Trans. Neural Syst. Rehab. Eng. 29, 760–769. doi: 10.1109/TNSRE.2021.3073134

PubMed Abstract | CrossRef Full Text | Google Scholar

He, S., Yu, T., Gu, Z., and Li, Y. (2017). “A hybrid BCI web browser based on EEG and EOG signals,” in 2017 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC)), 1006–1009.

PubMed Abstract | Google Scholar

Huang, Q., Zhang, Z., Yu, T., He, S., and Li, Y. (2019). An EEG-/EOG-based hybrid brain-computer interface: application on controlling an integrated wheelchair robotic arm system. Front. Neurosci. 13, 1243. doi: 10.3389/fnins.2019.01243

PubMed Abstract | CrossRef Full Text | Google Scholar

Jin, J., Li, S., Daly, I., Miao, Y., Liu, C., Wang, X., et al. (2020). The study of generic model set for reducing calibration time in P300-based brain–computer interface. IEEE Trans. Neural Syst. Rehab. Eng. 28, 3–12. doi: 10.1109/TNSRE.2019.2956488

PubMed Abstract | CrossRef Full Text | Google Scholar

Jin, J., Wang, Z., Xu, R., Liu, C., Wang, X., and Cichocki, A. (2021). Robust similarity measurement based on a novel time filter for SSVEPs detection. IEEE Trans. Neural. Netw. Learn. Syst. 10, 1–10. doi: 10.1109/TNNLS.2021.3118468

PubMed Abstract | CrossRef Full Text | Google Scholar

Kapgate, D., Kalbande, D., and Shrawankar, U. (2020). An optimized facial stimuli paradigm for hybrid SSVEP+P300 brain computer interface. Cogn. Syst. Res. 59, 114–122. doi: 10.1016/j.cogsys.2019.09.014

PubMed Abstract | CrossRef Full Text | Google Scholar

Kundu, S., and Ari, S. (2022). Brain-computer interface speller system for alternative communication: a review. IRBM 43, 317–324. doi: 10.1016/j.irbm.2021.07.001

CrossRef Full Text | Google Scholar

Li, M., He, D., Li, C., and Qi, S. (2021a). Brain–computer interface speller based on steady-state visual evoked potential: a review focusing on the stimulus paradigm and performance. Brain Sci. 11, 450. doi: 10.3390/brainsci11040450

PubMed Abstract | CrossRef Full Text | Google Scholar

Li, P., Yin, C., Li, M., Li, H., and Yang, B. (2021b). A dry electroencephalogram electrode for applications in steady-state visual evoked potential-based brain–computer interface systems. Biosens. Bioelectron. 187, 113326. doi: 10.1016/j.bios.2021.113326

PubMed Abstract | CrossRef Full Text | Google Scholar

Li, Q., Lu, Z., Gao, N., and Yang, J. (2019). Optimizing the performance of the visual P300-speller through active mental tasks based on color distinction and modulation of task Difficulty 13, 130. doi: 10.3389/fnhum.2019.00130

PubMed Abstract | CrossRef Full Text | Google Scholar

Lin, Z., Zhang, C., Wu, W., and Gao, X. (2007). Frequency recognition based on canonical correlation analysis for SSVEP-based BCIs. IEEE Trans. Biomed. Eng. 54, 1172–1176. doi: 10.1109/TBME.2006.889197

PubMed Abstract | CrossRef Full Text | Google Scholar

Lo, C.-C., Chien, T.-Y., Pan, J.-S., and Lin, B.-S. (2016). Novel non-contact control system for medical healthcare of disabled patients. IEEE Access 4, 5687–5694. doi: 10.1109/ACCESS.2016.2566668

PubMed Abstract | CrossRef Full Text | Google Scholar

Lu, Z., Li, Q., Gao, N., and Yang, J. (2020). Time-varying networks of ERPs in P300-speller paradigms based on spatially and semantically congruent audiovisual bimodality. J. Neural Eng. 17, 046015. doi: 10.1088/1741-2552/aba07f

PubMed Abstract | CrossRef Full Text | Google Scholar

Mcfarland, D. J. (2020). Brain-computer interfaces for amyotrophic lateral sclerosis. Muscle Nerve. 61, 702–707. doi: 10.1002/mus.26828

PubMed Abstract | CrossRef Full Text | Google Scholar

Nakanishi, M., Wang, Y., Wang, Y. T., Mitsukura, Y., and Jung, T. P. (2014). A high-speed brain speller using steady-state visual evoked potentials. Int. J. Neural Syst. 24, 1450019. doi: 10.1142/S0129065714500191

PubMed Abstract | CrossRef Full Text | Google Scholar

Pan, J., Li, Y., Zhang, R., Gu, Z., and Li, F. (2013). Discrimination between control and idle states in asynchronous SSVEP-based brain switches: a pseudo-key-based approach. IEEE Trans. Neural Syst. Rehab. Eng. 21, 435–443. doi: 10.1109/TNSRE.2013.2253801

PubMed Abstract | CrossRef Full Text | Google Scholar

Pan, J., Xie, Q., Qin, P., Chen, Y., He, Y., Huang, H., et al. (2020). Prognosis for patients with cognitive motor dissociation identified by brain-computer interface. Brain 143, 1050–1053. doi: 10.1093/brain/awaa026

PubMed Abstract | CrossRef Full Text | Google Scholar

Pfurtscheller, G., Solis-Escalante, T., Ortner, R., Linortner, P., and Muller-Putz, G. R. (2010). Self-paced operation of an SSVEP-based orthosis with and without an imagery-based “brain switch:” a feasibility study towards a hybrid BCI. IEEE Trans. Neural Syst. Rehab. Eng. 18, 409–414. doi: 10.1109/TNSRE.2010.2040837

PubMed Abstract | CrossRef Full Text | Google Scholar

Rezeika, A., Benda, M., Stawicki, P., Gembler, F., Saboor, A., and Volosyak, I. (2018). Brain–computer interface spellers: a review. Brain Sci. 8, 57. doi: 10.3390/brainsci8040057

PubMed Abstract | CrossRef Full Text | Google Scholar

Tomita, Y., Vialatte, F. B., Dreyfus, G., Mitsukura, Y., Bakardjian, H., and Cichocki, A. (2014). Bimodal BCI using simultaneously NIRS and EEG. IEEE Trans. Biomed. Eng. 61, 1274–1284. doi: 10.1109/TBME.2014.2300492

PubMed Abstract | CrossRef Full Text | Google Scholar

Xing, X., Wang, Y., Pei, W., Guo, X., Liu, Z., Wang, F., et al. (2018). A high-speed SSVEP-based BCI using dry EEG electrodes. Sci. Rep. 8, 1–10. doi: 10.1038/s41598-018-32283-8

PubMed Abstract | CrossRef Full Text | Google Scholar

Xu, M., Han, J., Wang, Y., Jung, T.-P., and Ming, D. (2020). Implementing over 100 command codes for a high-speed hybrid brain-computer interface using concurrent P300 and SSVEP features. IEEE Trans. Biomed. Eng. 67, 3073–3082. doi: 10.1109/TBME.2020.2975614

PubMed Abstract | CrossRef Full Text | Google Scholar

Zhang, S., and Chen, X. (2022). Effect of background luminance of visual stimulus on elicited steady-state visual evoked potentials. Brain Sci. Adv. 8, 50–56. doi: 10.26599/BSA.2022.9050006

CrossRef Full Text | Google Scholar

Zhou, Y., He, S., Huang, Q., and Li, Y. (2020). A hybrid asynchronous brain-computer interface combining SSVEP and EOG signals. IEEE Trans. Biomed. Eng. 67, 2881–2892. doi: 10.1109/TBME.2020.2972747

PubMed Abstract | CrossRef Full Text | Google Scholar

Zhou, Z., Yin, E., Liu, Y., Jiang, J., and Hu, D. (2014). A novel task-oriented optimal design for P300-based brain–computer interfaces. J. Neural Eng. 11, 056003. doi: 10.1088/1741-2560/11/5/056003

PubMed Abstract | CrossRef Full Text | Google Scholar

Zhu, Y., Li, Y., Lu, J., and Li, P. (2020). A hybrid BCI based on SSVEP and EOG for robotic arm control. Front. Neurorobot. 14, 583641. doi: 10.3389/fnbot.2020.583641

PubMed Abstract | CrossRef Full Text | Google Scholar