EEG-fNIRS-based hybrid image construction and classification using CNN-LSTM

The constantly evolving human–machine interaction and advancement in sociotechnical systems have made it essential to analyze vital human factors such as mental workload, vigilance, fatigue, and stress by monitoring brain states for optimum performance and human safety. Similarly, brain signals have become paramount for rehabilitation and assistive purposes in fields such as brain–computer interface (BCI) and closed-loop neuromodulation for neurological disorders and motor disabilities. The complexity, non-stationary nature, and low signal-to-noise ratio of brain signals pose significant challenges for researchers to design robust and reliable BCI systems to accurately detect meaningful changes in brain states outside the laboratory environment. Different neuroimaging modalities are used in hybrid settings to enhance accuracy, increase control commands, and decrease the time required for brain activity detection. Functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG) measure the hemodynamic and electrical activity of the brain with a good spatial and temporal resolution, respectively. However, in hybrid settings, where both modalities enhance the output performance of BCI, their data compatibility due to the huge discrepancy between their sampling rate and the number of channels remains a challenge for real-time BCI applications. Traditional methods, such as downsampling and channel selection, result in important information loss while making both modalities compatible. In this study, we present a novel recurrence plot (RP)-based time-distributed convolutional neural network and long short-term memory (CNN-LSTM) algorithm for the integrated classification of fNIRS EEG for hybrid BCI applications. The acquired brain signals are first projected into a non-linear dimension with RPs and fed into the CNN to extract essential features without performing any downsampling. Then, LSTM is used to learn the chronological features and time-dependence relation to detect brain activity. The average accuracies achieved with the proposed model were 78.44% for fNIRS, 86.24% for EEG, and 88.41% for hybrid EEG-fNIRS BCI. Moreover, the maximum accuracies achieved were 85.9, 88.1, and 92.4%, respectively. The results confirm the viability of the RP-based deep-learning algorithm for successful BCI systems.

Introduction

Brain–computer interfaces (BCIs) have become an indispensable element for individuals with disabilities. They have become integral components of new medical applications and have been increasingly applied in communication systems, human–machine interfaces (Bai et al., 2020), and neurofeedback applications (Mercado et al., 2021). BCI enables communication between the human brain and the external computer/device through generated brain commands, thereby avoiding the peripheral nervous system (Antonietti et al., 2021). Moreover, BCI is a neurofeedback method that can enhance the quality of life of patients suffering from serious motor debilities due to tetraplegia (Benaroch et al., 2021), stroke (Mane et al., 2020), and other spinal cord injuries (Al-Taleb et al., 2019). BCI also has applications in neurorehabilitation, communication and control, motor therapy and recovery, brain monitoring, and neuro-ergonomics (Asgher et al., 2020a,b; Mughal et al., 2021). The BCI analyzes a biosignal measured from a healthy subject to predict some intangible aspects of their cognitive state. This process usually consists of three main steps: data acquisition from the brain depending on the application and modality chosen, interpretation or pre-processing data into commands, and output to the computer to generate a command. Among the three types of BCI, namely, reactive, active, and passive BCI (pBCI), pBCI is an important research area that estimates human emotions, cognition, intentions, and behavior based on generated brain responses to different situations.

The demand for improved traditional BCI practices has increased with advances in neuroimaging modalities. Primary non-invasive neuroimaging modalities for BCI include functional magnetic resonance imaging (fMRI), electroencephalography (EEG), magnetoencephalography, and functional near-infrared spectroscopy (fNIRS). Among them, EEG and fNIRS are the foremost modalities in terms of cost and manageability (Rahman et al., 2020; Rashid et al., 2020). EEG measures brain activity by calculating the voltage fluctuations from the action potentials of neurons, whereas fNIRS detects brain activity related to hemodynamic response changes (Hong and Zafar, 2018; Liu et al., 2021). Although invasive techniques provide more accurate data than non-invasive techniques, non-invasive modalities are more frequent and appreciated in the research domain. Non-invasive recording techniques for brain activity improve safety and reduce ethical concerns (Burwell et al., 2017; Pham et al., 2018). Over time, various non-invasive techniques have been used in studies. The most commonly used are EEG, fNIRS, electrooculography, and fMRI (Choi et al., 2017). The selection of a non-invasive modality depends on many factors. Usually, the following parameters are considered: cost, ease of use, and temporal and spatial resolution, as needed by the application. Each modality offers some advantages over the others, and there are always some associated trade-offs; the pros of one modality compensate for the cons of the other modality. Thus, hybrid approaches have proven to be more efficient. Hybrid neuroimaging modalities increase accuracy and offer a greater degree of reasonable control (Hong and Khan, 2017; Khan and Hong, 2017; Hong et al., 2018).

Researchers appreciate the use of low-cost neuroimaging modalities (Hong et al., 2020). Modalities that offer convenience for non-laboratory setups are also choices of interest. In this regard, EEG and fNIRS are the most commonly used. Both are portable and inexpensive compared to the alternatives. Electrodes capture EEG signals due to variations in the current generated by neurons due to postsynaptic activities (Sazgar and Young, 2019). Several electrodes are placed on the subject's scalp for EEG data acquisition. Although EEG provides better temporal resolution ranging up to ~0.05 s, it provides a spatial resolution of only ~10 mm (Puce and Hämäläinen, 2017; Fu et al., 2020). The contrasting comparison of the temporal and spatial resolutions manifests trade-offs when using the EEG modality. In contrast to EEG, fNIRS is an optical imaging technique that measures light absorbance to calculate concentration changes in oxy-hemoglobin and deoxy-hemoglobin within the brain. Similar to EEG, fNIRS is cost-effective and portable. However, unlike EEG, fNIRS provides better spatial resolution. Moreover, fNIRS is less influenced by electrical noise (Hasan et al., 2020; Ghafoor et al., 2022). As evidenced by the comparison, fNIRS can compensate for the trade-offs of EEG. Thus, the EEG and fNIRS hybrid method serve as a breakthrough in neuroimaging (Ahn and Jun, 2017) on theoretical grounds.

As fNIRS measures hemodynamic responses, there is an innate delay in the measurement (Saeed et al., 2020). Various methods have been proposed to compensate for this slow command generation. In this regard, a hybrid method comprising EEG and fNIRS techniques can be used, which proceeds by measuring the initial dip [i.e., at the onset of neural firing, the oxygenated hemoglobin (HBO) level first decreases] instead of the actual hemodynamic response (Hong and Khan, 2017; Kamran et al., 2018). The other contrasting difference between the two modalities is the rate at which data are sampled. The EEG data acquisition rate is ~10–100 times faster than that of fNIRS. When the EEG and fNIRS hybrid is used, it is common to downsample the EEG data to make its processing compatible with that of the fNIRS data (Khan and Hasan, 2020; Ortega et al., 2020). Downsampling might discard some segments of valuable data. As EEG signals are prone to electrical noise, fNIRS suffer from physiological noise, instrumentation, and experimental errors. The experimental errors may be spontaneous, unintentional diversions from the intended protocol, such as motion artifacts or changes in the light intensity in the ambiance. The motion artifacts present in the data can be significantly reduced via Wiener filtering-based methods (Jiang et al., 2019) or wavelet analysis-based methods (Islam et al., 2021). Instrumentation can also induce noise in the data, such as noise from the hardware. However, these noise signals are high-frequency components; thus, they can be eliminated using a low-pass filter. Physiological noises can arise due to breathing activity or heartbeats. Although these noises are unavoidable, many methods have been reported to counter these noises; commonly used techniques apply bandpass filters, parameter mapping, and independent component analysis (Rejer and Cieszyński, 2019; Vourvopoulos et al., 2019; Wankhade and Chorage, 2021). Denoising the data further removes data regions; thus, the processed data are even smaller in magnitude than the raw data. Therefore, the downsampling of the EEG data after pre-processing to match the fNIRS data removes a considerable amount of valuable information regarding brain activity.

Recurrence quantification analysis (RQA) of RP has become popular in recent years for analyzing brain activity because brain signals are both recurrent and dynamic. RP, in general terms, is a non-linear evaluation method for recurrent and dynamic signals. It is a visualization displaying the recurrent occurrences of states x(n) of a time signal in phase space. RQA is an analysis technique used to quantify the features of the constructed RP. In the literature, RQA feature analysis has been used in EEG signal detection of epilepsy and Alzheimer's disease, coupling and synchronization in EEG of epileptic discharge, and so on. Cortical function during different sleep stages was also analyzed using RP features. The RQA analysis showed that unique RPs were extracted for different sleep stages (Parro and Valdo, 2018). Several studies have also used artificial neural networks (ANNs) (Torse et al., 2019) and support vector machines (SVMs) (Houshyarifar and Amirani, 2017; Zhao et al., 2021) to classify extracted RQA features. One study used a four-layer ANN for different EEG channels to predict the onset of seizures using RQA measures (Torse et al., 2019).

As machine learning (ML) has rapidly become a state-of-the-art analysis tool, researchers have considered searching for classification features (Park and Jung, 2021). The qualitative aspects of these RPs can be used for classification. Moreover, DNNs are highly efficient training classifiers, resulting in better classification accuracy than ML classifiers (Sattar et al., 2021). However, only a few studies that applied these algorithms in BCI are available (Dehghani et al., 2021; Singh et al., 2021). Only one study used a CNN for the binary classification of epileptic seizures from EEG using RP as images (Gao et al., 2020). The practical application of biological feedback in BCI requires efficient and precise motor activity detection and classification methods. These conventional quantification and feature selection methods and simple ML classifiers face several challenges when implementing real-time BCI. Traditional feature engineering methods involve multiple steps, such as feature extraction, feature selection, finding suitable combinations for various features, and sometimes dimensionality reduction from a comparatively small quantity of data, thus leading to other problems such as overfitting and bias (Asgher et al., 2020b). These inherent constraints hinder adjustments by researchers. Therefore, the initial analysis steps, namely, data mining and pre-processing, are time-consuming.

On the other hand, deep learning (DL) algorithms such as CNNs can be employed in two ways for BCI applications: altering or modifying the CNN algorithm architecture to accommodate the one-dimensional time-series data obtained by the modalities or transforming one-dimensional data into two-dimensional (2D) data to be conveniently input to the CNN. Deep neural networks (DNNs) and other traditional classifiers have also been employed based on fNIRS and EEG signals to recognize three different cognitive states (Huve et al., 2019; Takahashi et al., 2021), electromyography signals classification (Oh and Jo, 2021), control of wearable exoskeleton (Sun et al., 2021), and other control applications (Kim et al., 2021; Li et al., 2021; Yaqub et al., 2021). A similar approach has been used for various other applications, such as controlling robots (Huve et al., 2018), differentiating workloads by analyzing the fNIRS signals, and using deep learning techniques. Shoeibi et al. (2022) used an adaptive neuro-fuzzy interface to detect epileptic seizures from EEG signals. The literature also demonstrates the time-delay neural network for classification purposes. Thyagachandran et al. (2020) used this approach to classify EEG signals; however, the presented model was not sufficiently deep to learn the hierarchical features of the EEG signal. The research that resonated most with our present study is that of Tanveer et al. (2019). The authors investigated deep learning-based BCI to detect driver drowsiness. The output strength of the selected channels was translated into color maps and fed into the CNN classifier as an input. The output color maps were obtained by linear mapping the values from the channel to the color intensity. Recent biosignal analysis techniques have a higher inclination toward non-linear dynamics. One of the most widely used methods is the recurrence plot (RP). The analysis focuses on the repeatability of the time-series states and presents the output in geometric structures, whose topology is analyzed to estimate the characteristics of the dynamics (Nayak et al., 2018; Acharya et al., 2019). In the literature, researchers have also experimented with hybrid CNN-LSTM models for time-series biological signal analysis to detect mental disorders. A study to detect schizophrenia via EEG was carried out by Shoeibi et al. (2021) using different ML and DL models, and a comparison was made between applied algorithms based on their accuracy percentage. Among all ML and DL algorithms, CNN-LSTM proved the best architecture for diagnosing schizophrenia.

This study investigated the RP performance for EEG, fNIRS, and hybrid EEG-fNIRS within a deep convolutional neural network and a long short-term memory (CNN-LSTM) model for neuroimaging brain data for BCI. The obtained RPs of EEG and fNIRS were fed as images into the hybrid CNN-LSTM network for classification. The initial hypothesis of this study was that “The classification accuracy of Hybrid EEG-fNIRS BCI will improve by incorporating all signal information from both modalities using recurrence plots instead of using traditional methods of downsampling EEG signals to make them compatible with fNIRS for hybrid BCI.” The main contributions and novelty of our work are as follows:

(i) Implementing whole EEG and fNIRS signals, without any information loss or downsampling, for Hybrid EEG-fNIRS BCI using recurrence plots.

(ii) Implement the time-distributed CNN-LSTM model for activity detection using EEG and fNIRS recurrence plots for hybrid BCI.

Furthermore, to the best of the authors' knowledge, time-distributional (TD) layers were implemented in a network that was not previously used in the BCI field.

The detailed methodology of this research, the dataset used, RP formation from EEG and fNIRS datasets, and the classification approach used for the four-class classification of constructed RP are detailed in the following sections. The related performance of RP in EEG-BCI, fNIRS-BCI, and hybrid EEG-fNIRS-BCI is discussed, and the study's conclusions are provided.

Methodology

In this study, RP performance for EEG, fNIRS, and hybrid EEG-fNIRS with the deep CNN-LSTM model was investigated for neuroimaging brain data for BCI. RP transformed the time series data into the phase space and provided an alternate method to envisage the periodic nature of a time series trajectory, that is, brain signal data in phase space. RPs of EEG and fNIRS were constructed and used as images to feed the hybrid time-distributed CNN-LSTM network for classification. The detailed methodology is described in this section and illustrated in Figure 1.

FIGURE 1

Figure 1. Methodology of the study shows the construction of the hybrid EEG-fNIRS dataset using RPs and classification using time-distributed CNN-LSTM.

Dataset and experimental protocol

The research used an open-source meta-dataset. The data were recorded at the Technische Universität Berlin (Shin et al., 2018) and collected through three different paradigms from 26 healthy participants while focusing on cognitive tasks. Datasets A, B, and C were chosen for the three different cognitive tasks: n-back, discrimination response, and word generation, respectively. On these grounds, the selected dataset was an appropriate choice for research in the domain of hybrid BCI. First, task A was performed, followed by tasks C and B. In this study, only dataset A (n-back) was used. The entire n-back dataset consisted of three sessions, and each session consisted of three series: 0-back, 2-back, and 3-back tasks. The total recording time for each series was 62 s. The initial 2 s were dedicated to task illustration. The following 40 s were reserved for the task performance (20 numbers displayed for 2 s each on the screen), and the last 20 s were reserved for the rest period. Thus, for every n-task, there were 180 trials. The experimental protocol for the n-back dataset is shown in Figure 2.

FIGURE 2

Figure 2. Experiment paradigm of n-back tasks.

Data acquisition

The EEG and fNIRS data were recorded simultaneously to ensure that the data were synchronized, and a parallel port was used to send the triggers. A BrainAmp EEG amplifier was used to record the EEG data, and the sampling frequency was 1,000 Hz. A stretchable fabric cap was used to place the 30 active electrodes to acquire data in frontal, motor cortex, parietal, and occipital regions according to the internationally recognized 10–5 system (Shin et al., 2018).

The fNIRS data were recorded at a sampling frequency of 10.4 Hz via NIRScout (NIRx Medizintechnik GmbH, Berlin, Germany). Sixteen electrodes, representing a combination of sources with detectors, were positioned at the frontal lobe, motor cortex, parietal lobe, and occipital lobe. The optodes of the NIRS were fixed with EEG electrodes on the same cap. The positioning of the electrodes and optodes is illustrated in Figure 3 green circles: EEG electrodes, red circles: NIRS optodes.

FIGURE 3

Figure 3. EEG and NIRS electrode positions according to the 10–5 system. Green dots represent EEG electrodes and red dots denote NIRS Optodes.

Data pre-processing and labeling

The EEG data were downsampled to 200 Hz. A 6th-order Butterworth bandpass filter with a passband frequency range of 1–40 Hz was used for filtering purposes. The acquired data were first translated into oxy- and deoxy-hemoglobin intensity variations to pre-process the fNIRS data. The conversions were conducted using the modified Beer–Lambert law. The fNIRS raw data were downsampled at 10 Hz. As the fundamental frequency of this dataset was very low, the downsampled data were not fed into the Butterworth bandpass filter. Instead, the data were low-pass filtered to avoid losing the fundamental frequency component. The cutoff frequency of the filter was chosen to be 0.2 Hz. The data were acquired using MATLAB R2013b software. Further processing was performed using Python on Spyder in the Anaconda development environment. After filtration, the dataset was labeled using the activity time markers from the acquired continuous EEG and fNIRS signals. Four classes, namely, 0-, 2-, and 3-back classes, and one of the remaining states, were labeled concerning the experimental protocol. After that, the labeled data were used for RP construction.

Recurrence plots

A recurrence plot (RP) is a contemporary technique for analyzing non-linear data. This technique employs the visualization of a square matrix whose elements link to the dynamic state repetition. The ordered pair of matrices corresponds to the specific timing of the repetition. Recurrence analysis is a graphical technique that aims to identify hidden recurring patterns (Ledesma-Ramirez et al., 2020). To illustrate this idea, our desired information is univariate time series data and that the data under analysis are a subpart of the large n-dimensional dataset. The topological rendering of the original n-dimensional dataset can be obtained using a single observable variable.

Thus, the embedded matrix, namely, x^m, can be constructed as follows:

$\begin{array}{ll}\left[x_i^m=(x_i,x_{i+d},x_{i+2d},.......,x_{i+(m-1)d})\right]& (1)\end{array}$

where x_i is a scalar series, the dimension is represented by m, and d is the delay. In case the condition

$\begin{array}{ll}m\ge 2n+1& (2)\end{array}$

is satisfied, the single output variable exhibits the potential to recreate the entire system. Recreation heavily depends on the sequence of the embedded matrix. The sequence can be controlled by adequately choosing parameters m and d. The asymmetric matrix of the Euclidean distances can also be constructed by measuring the distance between pairs of embedded vectors. These distances are translated into an equivalent color, and each distance has a distinctive color. Thus, an RP is a square assortment of pixels whose color depends on the corresponding magnitude of values. The pixel coordinates also carry useful information that is linearly related to the location of the data in the original data matrix.

The use of ε is commonly employed in RPs. This ε is referred to as the critical radius. Each value is compared with the critical radius to check whether the pixel value is ≤ε; then, the pixel is displayed as a darkened pixel. In other words, RP is a visualization of a square recurrence matrix showing all the instances of times at which a state of a non-linear system repeats; the columns and axes of the recurrence matrix correspond to specific time intervals. In technical terms, an RP shows every time of a non-linear time signal from a dynamical system at which its phase space trajectory spans approximately the same area in the phase space. In graphical terms, this is a graph of

$\begin{array}{ll}\vec{x}(i)\approx \vec{x}(j)& (3)\end{array}$

where i is on the horizontal axis, j is on the vertical axis, and $\vec{x}$ is the phase space trajectory of the dynamical system. Thus, a binary recurrence matrix is constructed using a specific time window w = 5 s, where any two-time steps are separated by the time interval ε = 0.1 and a step size of 10 in the following manner:

$\begin{array}{ll}R(i,j)=\{\begin{array}{l}1if \left|\right|\overrightarrow{x}(i)- \overrightarrow{x}(j)\left|\right|\le \epsilon \\ 0otherwise\end{array}& (4)\end{array}$

where i and j are the horizontal and vertical time axes, i, j ϵ {t₀, t₁….t, …. t_T}. The RP is a visualization of the recurrence matrix with a black square of the lattice at coordinates (i, j) if R(i, j) = 1 and a white square if R(i, j) = 0.

Figure 4 shows the corresponding RPs constructed using the fNIRS dataset for the 0-, 2-, and 3-back classes, and the rest state for Subject 1.

FIGURE 4

Figure 4. RPs of fNIRS dataset. (A) 0-back, (B) 2-back, (C) 3-back, (D) rest.

After experimenting with different values for the parameters of the RPs, ε = 0.1 and step size = 10 were adopted. Figure 4 depicts the non-linear mapping of the acquired brain signals to the new dimension through the RPs with the selected parameters. Each subject's RP data were split into training and test datasets using 10-fold cross-validation before performing classification to avoid overfitting and provide better generalization. Moreover, the model performance was evaluated based on the following performance metrics.

Accuracy

The accuracy of a classifier is the proportion of the total number of correct predictions made by the classifier. If the confusion matrix is given, accuracy is defined as:

$\begin{array}{l}Accuracy=\frac{TruePositives+TrueNegatives}{TotalNumberofSamples}\end{array}$

Precision

Precision or positive predictive value is the proportion of the correctly predicted positive cases of all cases predicted positively by a classifier. Given a confusion matrix, precision is defined as:

$\begin{array}{l}Precision=\frac{TruePositives}{TotalNumberofSamplesPredictedasPositive}\end{array}$

Recall

Recall or sensitivity is the proportion of all actual positive cases that were correctly predicted to be positive by a classifier. Given a confusion matrix, recall is defined as:

$\begin{array}{l}Recall=\frac{TruePositives}{TotalNumberofActualPositiveSamples}\end{array}$

F1 score

The F1 score is the harmonic mean of precision and recall values for a given classification model. Given a confusion matrix, the F1 score is defined as:

$\begin{array}{l}Precision={\left(\frac{Precisio{n}^{-1}+Recal{l}^{-1}}{2}\right)}^{-1}\end{array}$

Time distributed CNN-LSTM

A CNN is a multilayered neural network with architecture to detect complex features in the data. Unlike traditional multilayer perceptron architectures, CNN uses two operations called “convolution” and “pooling” to reduce the image into its essential features, which are used to understand and classify it. CNNs are composed of basic building blocks, which include the following: a convolutional layer with a filter or kernel passed over an image; an activation layer that usually has an activation function; a rectified linear unit (ReLU) to introduce non-linearity that allows the network to train itself through backpropagation; a pooling layer that downsamples and reduces the size of the matrix and is focused on the most prominent information in each feature of the image; and a fully connected layer that outputs the different probabilities associated with every label attached to the image. The label with the highest probability is the classification decision. CNNs are widely used in agriculture, self-driving vehicles, healthcare, and surveillance. LSTM networks are recurrent neural networks (RNNs) that use special and standard units. The special units include the “memory cell,” which maintains information in its memory for longer. LSTM has feedback connections, unlike standard feed-forward neural networks; it can process entire data sequences, including speech and video. LSTM is widely used in speech recognition, handwriting recognition, handwriting generation, music generation, language translation, image captioning, and anomaly detection in intrusion detection systems. A simple LSTM unit comprises a cell, input, output, and forget gate. The cell remembers the information, whereas the gates regulate the flow of information. LSTM networks are modified forms of RNNs; they remember past data in memory.

Over time, researchers have applied different architectures and types of deep learning networks. Unlike images, text, voice, and other widely used datasets, neuroimaging signals are intrinsically different and have an important chronological order. This chronological order dictates the flow of information necessary to detect activities or actions. Examples of such chronological order are the initial dip at the start of activity in fNIRS signals and positive deflection in event-related potential P300 signals in EEG. A novel CNN-LSTM network was designed for this study. The network consists of one CNN and one LSTM module combined with a dense layer. After pre-processing, the data are fed into the CNN module; this module consists of two convolutional layers, each with 16 filters and ReLU as the activation function, and one max-pooling layer. CNNs are best known for their abilities of feature extraction from 2D and 3D images. Considering the data sequence used in the form of chronologically ordered time windows, the relationship between two windows in a given input should be detected. An LSTM layer enables the network to use its memory and enhance its prediction. The convoluted output from the CNN block is reshaped and flattened before being fed into the LSTM layer. The layers preceding the LSTM layers are wrapped inside a time-distributed layer that allows their application to every temporal slice of the input data. This time-distributed wrapper applies the same instance of convolutional layers to each timestamp, such that the same set of weights is used. After passing through another dense layer, the LSTM layer terminates into the output layer.

No researcher has exploited this chronological order using time-distributed layers in deep learning models to the authors' knowledge. The constructed RPs with a fixed window length and an overlapping portion are fed into the network as images. The different configurations of this proposed network for fNIRS, EEG, and hybrid modalities are discussed in detail in the Discussion section. The network architecture for the EEG and fNIRS BCI and the details of the hyperparameters of the DL model used, that is, several layers, dimensions, the number of filters used in each layer, and the number of neurons, among other details, are shown in Figure 7. Researchers have invested tremendous efforts to determine the single best architecture for deep learning neural networks, giving rise to the sub-research field known as neural architecture search (NAS). However, there is no definite answer regarding the optimal neural architecture a priori. The number of neurons, number of filters, number of layers, their combinations, dropout, and max-pooling percentage remain the best hyperparameters. The most viable approach seems to be using intuition and domain knowledge to determine an initial guess for these parameters and then iteratively shortlist them to obtain good values. In this study, the NAS design process was as follows: a network with a minimum number of parameters, a single convolutional layer, a single LSTM layer, and one dense layer was created; other hyperparameters were tuned; more layers were added, and the network hyperparameters were tuned with a grid search using the sklearn wrapper. We performed the above grid search with sample data and chose the best-performing network for EEG, fNIRS, and EEG+fNIRS datasets. However, this approach resulted in input dimension mismatch because of the extra number of features in the hybrid dataset compared to the single modality datasets. We solved this problem by adding another sequence module on top of the EEG network architecture and wrapping it inside the TD layer, similar to the EEG network. The later stages of a network-like dense layer, LSTM layer, and the following layers remained the same; however, this strategy solved the input dimensionality mismatch problem. Figures 5–7 show the network architecture of the study.

FIGURE 5

Figure 5. Inside a TD layer. RP input to two Conv2D layers, each with 16 filters, and ReLu as activation function, followed by a max pool and flatten layer.

FIGURE 6

Figure 6. Time-distributed CNN-LSTM network for classification of four-class mental workload using RPs of EEG and fNIRS dataset.

FIGURE 7

Figure 7. Time-distributed CNN-LSTM network for classification of four-class mental workload using RPs of hybrid EEG-fNIRS dataset.

Results

The time-distributed CNN-LSTM was used in this research to classify four classes, namely, the three n-back activities and the rest state from the fNIRS dataset acquired from 26 subjects. The data acquisition and initial pre-processing included filtering EEG data using a zero-phase, low-pass, 6th-order Butterworth filter. In the case of fNIRS, conversion of light densities into changes in the concentrations of HbO and HbR (hemodynamic response) was performed using the modified Beer–Lambert law followed by a zero-phase, low-pass, 6th-order Butterworth filter. After that, the data were labeled along with outlier rejection and data normalization. The window size selection in the hybrid EEG/fNIRS-based BCI system is essential because hemodynamics response has an inherent delay, which requires 0–10 s to complete after the stimulus. In the literature, researchers have experimented with different windows of varying lengths, such as 2–9, 2–7, and 3–7 s (Khan and Hong, 2017; Gaur et al., 2021). Generally, the smaller the window size, the better the BCI performance will be. After an initial investigation, a window size of 5 s with 20% overlap was used for all BCIs. After that, the RPs for segmented signals were constructed into a sequence of 5 s windows.

The deep learning algorithms were trained on a GTX 1060 graphic card with 3 GB VRAM and an Intel 6th Gen Core i7-6700HQ processor with a 3.2-GHz frequency. The Keras API was used with the TensorFlow backend on Spyder in the Anaconda integrated development environment. The average accuracy achieved for the four-class classification was 78.4% for fNIRS, 86.44% for EEG, and 88.41% for hybrid EEG-fNIRS BCI. The maximum accuracies achieved were 82.4, 89.75, and 93.58%, respectively. Table 1 summarizes the results of the 26 participants in terms of their classification accuracies, precision, recall, and F1 score for EEG-BCI, fNIRS-BCI, and Hybrid EEG-fNIRS BCI. Figure 8 shows the average accuracies achieved by these approaches.

TABLE 1

Table 1. Performance evaluation metrices of fNIRS-BCI, EEG-BCI, and hybrid EEG-fNIRS-BCI.

FIGURE 8

Figure 8. Comparison of accuracies of fNIRS-based BCI, EEG-based BCI, and hybrid EEG-fNIRS-based BCI.

Discussion

Researchers appreciate the use of low-cost neuroimaging modalities. Modalities that offer convenience to non-laboratory setups are also choices of interest. In this regard, EEG and fNIRS are the most commonly used neuroimaging modalities. Both are portable and inexpensive compared to fMRI. However, EEG offers a spatial resolution of only ~10 mm (Puce and Hämäläinen, 2017; Fu et al., 2020). The contrasting comparison of the temporal and spatial resolutions manifests trade-offs when using the EEG modality.

In contrast to EEG, fNIRS constructs functional neuroimages of the brain by employing NIR light. As fNIRS measures hemodynamic responses, there is an innate delay in the measurement (Saeed et al., 2020). Various methods have been proposed to compensate for this slow command generation. In this regard, hybrid EEG-fNIRS can be an option. However, the sampling frequencies of both modalities are different, thus resulting in information loss. Moreover, the most important objective of all studies conducted on BCI is to enhance real-time classification accuracy and reduce computational costs with multiple commands, thus emphasizing the need to develop appropriate identification and classification methods for real-time BCI (Phanikrishna et al., 2021). Usually, multi-channel brain signal acquisition modalities (i.e., EEG) analyze brain motor activity using different methods, such as time and frequency feature analysis, event-related synchronization-desynchronization analysis, common spatial or temporal patterns, and spatial-spectral decomposition. Most of these methods require high computational costs and are less feasible to use for real-time BCI (Janapati et al., 2020). With advances in brain signal acquisition modalities, the demands for better signal processing and feature extraction have also increased. Traditional methods of extracting useful information from multi-channel brain signal acquisition modalities, such as time and frequency analysis, event-related synchronization and desynchronization analysis, and finding common spatial and temporal patterns are computationally expensive and not very feasible for real-time BCI applications. RQA of RP has become popular in recent years for analyzing brain activity because brain signals are both recurrent and dynamic. RQA is an analysis technique used to quantify features of the constructed RP. In the literature, RQA features have been used in EEG signal detection of epilepsy and Alzheimer's disease, coupling, and synchronization in EEG of epileptic discharge. Cortical function during different sleep stages was analyzed using RP features. The RQA analysis showed that unique RPs were extracted for different sleep stages (Parro and Valdo, 2018). Several studies have also used SVM and ANNs to classify extracted RQA features. One study used a four-layer ANN for different EEG channels to predict the onset of seizures using RQA measures (Torse et al., 2019).

Considering the complexity and computational cost of DNNs, researchers have invested tremendous efforts to determine the best architecture for deep learning neural networks, giving rise to the sub-research field known as NAS. However, there is no definite conclusion regarding the optimal neural architecture a priori. The number of neurons, number of filters, number of layers, their combinations, dropout, and max-pooling percentage remain the best hyperparameters. The most viable approach seems to be using intuition and domain knowledge to determine an initial guess for these parameters and then iteratively shortlist to obtain good values. In this study, the NAS design process was as follows: a network with a minimum number of parameters, a single convolutional layer, a single LSTM layer, and one dense layer was created; other hyperparameters were tuned; more layers were added, and the network hyperparameters were tuned with a grid search using the sklearn wrapper. We performed the above grid search with sample data and chose the best-performing network for the fNIRS dataset. Another advantage of using an RP with a DNN is that it incorporates the entire signal and does not require any extra steps, such as feature extraction and feature selection. Moreover, it also minimizes extra pre-processing steps, such as finding temporal or spatial features. The constructed RPs of EEG and fNIRS were fed to the classification network to detect the class of activity (0, 2-back, 3-back, or rest). The classification network used was the time-distributed CNN-LSTM. CNN is best known for feature extraction from multidimensional images. In contrast, the RNN has an excellent pattern recognition ability for input sequences. However, CNN and RNN have stability issues due to either exploding or vanishing gradients. An RNN variant was used to solve this issue by using memory cells and LSTM. The highest classification accuracy for four-class mental workload data for the BCI was achieved using this network.

The study results indicate that using the hybrid modalities for the classification of BCI results in higher accuracy than that of the single modality, along with an increase in the number of commands and a reduction in detection time. Figure 8 shows the comparison of the average accuracies achieved for all BCIs. The results of our study have proven the initial hypothesis that incorporating the entire EEG signal for hybrid EEG-fNIRS BCI instead of downsampling will significantly increase classification accuracy. The classification accuracies achieved with our proposed methodology were the highest compared to other classification methods for four-class EEG-based BCI and other studies on the same dataset for four-class classification for hybrid EEG-fNIRS. A comparison with relevant studies is presented in Tables 2, 3, respectively. Moreover, implementing the TD layers resulted in faster and easier computation. This may prove to be a state-of-the-art algorithm in the present BCI realm. The results show a promising future for the use of RPs in real-time BCI. The proposed classification method can help improve the accuracy of real-time BCI.

TABLE 2

Table 2. Comparison with other 4-class classification studies for BCI.

TABLE 3

Table 3. Comparison with other 4-class hybrid classification studies for BCI with same dataset.

The limitations of our work are as follows: first, the proposed methodology is computationally costly, and substantial computational resources are required to train and test deep learning models with large datasets because the size of RP increases exponentially with the data size fed at a time, as does the model complexity. Second, the proposed algorithm has not yet been implemented for real-time BCI, which leaves room for network improvement and optimization. There are many potential applications for RPs in BCIs other than EEG and fNIRS signals, as all biological signals constitute dynamic time series data, and our study has validated the successful implementation of RP for time-series data analysis. In the future, further work can be conducted to explore and experiment with new deep learning methods in computer vision along with RPs, such as transformers and attention learning for active channel selection for real-time BCI. Working in this direction will help researchers mitigate nuances related to deep learning algorithms in BCI.

Conclusion

This paper provides an inimitable time-distributed convolutional neural network and long short-term memory method for the integrated categorization of fNIRS-EEG for hybrid BCI applications. The recorded brain signals are first projected onto a non-linear dimension via RPs and supplied into the CNN to extract critical characteristics without downsampling. Then, LSTM is utilized to learn the chronological properties and time-dependence relation to identify brain activity. The average accuracy levels were 78.44% for fNIRS, 86.24% for EEG, and 88.41% for hybrid EEG-fNIRS BCI when using the suggested model. The findings support the RP-based deep-learning algorithm's suitability for effective BCI applications.

Data availability statement

The original contributions presented in the study are included in the article, further inquiries can be directed to the corresponding author.

Ethics statement

The original study by Shin et al. (2018) was reviewed and approved. The data set is an open-source meta dataset of Shin et al. (2018). The patients/participants provided their written informed consent to participate in the original study. For the current study, ethical review and approval, and written informed consent, were not required, in accordance with the local legislation and institutional requirements.

Author contributions

NM wrote the first draft manuscript. MK conceived the idea of the manuscript and supervised the process. KK, KJ, and HS collected and analyzed the methodologies reported in the paper. NN, UG, and K-SH participated in the revision of the manuscript. NM, MK, UG, and K-SH revised and edited the manuscript. All authors contributed to the article and approved the submitted version.

Funding

This work was supported by the Technology Innovation Program (Grant No. 20016846, Development of fNIRS-based multimonitoring healthcare device for sleep state diagnosis) funded by the Ministry of Trade, Industry, and Energy (MOTIE, Korea).

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

Acharya, U. R., Hagiwara, Y., Deshpande, S. N., Suren, S., Koh, J. E., Oh, S. L., et al. (2019). Characterization of focal EEG signals: a review. Fut. Gen. Comput. Syst. 91, 290–299. doi: 10.1016/j.future.2018.08.044

CrossRef Full Text | Google Scholar

Ahn, S., and Jun, S. C. (2017). Multi-modal integration of EEG-fNIRS for brain-computer interfaces—current limitations and future directions. Front. Hum. Neurosci. 11, 503. doi: 10.3389/fnhum.2017.00503

PubMed Abstract | CrossRef Full Text | Google Scholar

Al-Taleb, M. K. H., Purcell, M., Fraser, M., Petric-Gray, N., and Vuckovic, A. (2019). Home used, patient self-managed, brain-computer interface for the management of central neuropathic pain post spinal cord injury: usability study. J. Neuroeng. Rehabil. 16, 128. doi: 10.1186/s12984-019-0588-7

PubMed Abstract | CrossRef Full Text | Google Scholar

Antonietti, A., Balachandran, P., Hossaini, A., Hu, Y., and Valeriani, D. (2021). The BCI Glossary: a first proposal for a community review. Taylor Francis 8, 42–53. doi: 10.1080/2326263X.2021.1969789

CrossRef Full Text | Google Scholar

Asgher, U., Khalil, K., Ayaz, Y., Ahmad, R., and Khan, M. J. (2020a). “Classification of mental workload (MWL) using Support Vector Machines (SVM) and Convolutional Neural Networks (CNN),” in 2020 3rd International Conference on Computing, Mathematics and Engineering Technologies: Idea to Innovation for Building the Knowledge Economy, ICoMET. pp. 1–6. doi: 10.1109/iCoMET48670.2020.9073799

PubMed Abstract | CrossRef Full Text | Google Scholar

Asgher, U., Khalil, K., Khan, M. J., Ahmad, R., Butt, S. I., Ayaz, Y., et al. (2020b). Enhanced accuracy for multiclass mental workload detection using long short-term memory for brain–computer interface. Front. Neurosci. 14, 584. doi: 10.3389/fnins.2020.00584

PubMed Abstract | CrossRef Full Text | Google Scholar

Bai, Z., Fong, K. N. K., Zhang, J. J., Chan, J., and Ting, K. H. (2020). Immediate and long-term effects of BCI-based rehabilitation of the upper extremity after stroke: a systematic review and meta-analysis. J. Neuroeng. Rehabil. 17, 57. doi: 10.1186/s12984-020-00686-2

PubMed Abstract | CrossRef Full Text | Google Scholar

Benaroch, C., Sadatnejad, K., Roc, A., Appriou, A., Monseigne, T., Pramij, S., et al. (2021). Long-term BCI training of a tetraplegic user: adaptive riemannian classifiers and user training. Front. Hum. Neurosci. 15, 635653. doi: 10.3389/fnhum.2021.635653

PubMed Abstract | CrossRef Full Text | Google Scholar

Burwell, S., Sample, M., and Racine, E. (2017). Ethical aspects of brain computer interfaces: a scoping review. BMC Med. Ethics 18, 60. doi: 10.1186/s12910-017-0220-y

PubMed Abstract | CrossRef Full Text | Google Scholar

Choi, I., Rhiu, I., Lee, Y., Yun, M. H., and Nam, C. S. (2017). A systematic review of hybrid brain-computer interfaces: taxonomy and usability perspectives. PLoS ONE 12, e0176674. doi: 10.1371/journal.pone.0176674

PubMed Abstract | CrossRef Full Text | Google Scholar

Dehghani, M., Mobaien, A., and Boostani, R. (2021). A deep neural network-based transfer learning to enhance the performance and learning speed of BCI systems. Brain Comput. Interfaces 8, 14–25. doi: 10.1080/2326263X.2021.1943955

CrossRef Full Text | Google Scholar

Fu, Y., Zhao, J., Dong, Y., and Wang, X. (2020). Dry electrodes for human bioelectrical signal monitoring. Sensors 20, 3651. doi: 10.3390/s20133651

PubMed Abstract | CrossRef Full Text | Google Scholar

Gao, X., Yan, X., Gao, P., Gao, X., and Zhang, S. (2020). Automatic detection of epileptic seizure based on approximate entropy, recurrence quantification analysis and convolutional neural networks. Artif. Intell. Med. 102, 101711. doi: 10.1016/j.artmed.2019.101711

PubMed Abstract | CrossRef Full Text | Google Scholar

Gaur, P., Gupta, H., Chowdhury, A., McCreadie, K., Pachori, R. B., and Wang, H. (2021). A sliding window common spatial pattern for enhancing motor imagery classification in EEG-BCI. IEEE Trans. Instrum. Meas. 70, 1–9. doi: 10.1109/TIM.2021.3051996

CrossRef Full Text | Google Scholar

Ge, S., Wang, R., and Yu, D. (2014). Classification of four-class motor imagery employing single-channel electroencephalography. PLoS ONE 9, e98019. doi: 10.1371/journal.pone.0098019

PubMed Abstract | CrossRef Full Text | Google Scholar

Ghafoor, U., Yang, D., and Hong, K.-S. (2022). Neuromodulatory effects of HD-tACS/tDCS on the prefrontal cortex: a resting-state fNIRS-EEG study. IEEE J. Biomed. Health Inform. 26, 2192–2203. doi: 10.1109/JBHI.2021.3127080

PubMed Abstract | CrossRef Full Text | Google Scholar

Hasan, M. A., Khan, M. U., and Mishra, D. (2020). A computationally efficient method for hybrid EEG-fNIRS BCI based on the pearson correlation. Biomed Res. Int. 2020, 1838140. doi: 10.1155/2020/1838140

PubMed Abstract | CrossRef Full Text | Google Scholar

Hong, K.-S., Ghafoor, U., and Khan, M. J. (2020). Brain–machine interfaces using functional near-infrared spectroscopy: a review. Artif. Life Rob. 25, 204–218. doi: 10.1007/s10015-020-00592-9

CrossRef Full Text | Google Scholar

Hong, K.-S., and Khan, M. J. (2017). Hybrid brain-computer interface techniques for improved classification accuracy and increased number of commands: a review. Front. Neurorobot. 11, 35. doi: 10.3389/fnbot.2017.00035

PubMed Abstract | CrossRef Full Text | Google Scholar

Hong, K.-S., Khan, M. J., and Hong, M. J. (2018). Feature extraction and classification methods for hybrid fNIRS-EEG brain-computer interfaces. Front. Hum. Neurosci. 12, 246. doi: 10.3389/fnhum.2018.00246

PubMed Abstract | CrossRef Full Text | Google Scholar

Hong, K.-S., and Zafar, A. (2018). Existence of initial dip for BCI: an illusion or reality. Front. Neurorobot. 12, 69. doi: 10.3389/fnbot.2018.00069

PubMed Abstract | CrossRef Full Text | Google Scholar

Houshyarifar, V., and Amirani, M. C. (2017). Early detection of sudden cardiac death using Poincaré plots and recurrence plot-based features from HRV Signals. Turk. J. Electr. Eng. Comput. Sci. 25, 1541–1553. doi: 10.3906/elk-1509-149

CrossRef Full Text | Google Scholar

Huve, G., Takahashi, K., and Hashimoto, M. (2018). “Brain-computer interface using deep neural network and its application to mobile robot control,” in 2018 IEEE 15th International Workshop on Advanced Motion Control (AMC), 169–174. doi: 10.1109/AMC.2019.8371082

CrossRef Full Text | Google Scholar

Huve, G., Takahashi, K., and Hashimoto, M. (2019). “Online recognition of the mental states of drivers with an fNIRS-based brain-computer interface using deep neural network,” in 2019 IEEE International Conference on Mechatronics (ICM), 238–242. doi: 10.1109/ICMECH.2019.8722936

CrossRef Full Text | Google Scholar

Islam, M. K., Ghorbanzadeh, P., and Rastegarnia, A. (2021). Probability mapping based artifact detection and removal from single-channel EEG signals for brain–computer interface applications. J. Neurosci. Methods 360, 109249. doi: 10.1016/j.jneumeth.2021.109249

PubMed Abstract | CrossRef Full Text | Google Scholar

Janapati, R., Dalal, V., Govardhan, N., and Gupta, R. S. (2020). Review on EEG-BCI classification techniques advancements. IOP Conf. Ser. Mater. Sci. Eng. 981, 032019. doi: 10.1088/1757-899X/981/3/032019

PubMed Abstract | CrossRef Full Text | Google Scholar

Jiang, X., Bian, G., and Tian, Z. A. (2019). Removal of artifacts from EEG signals: a review. Sensors. 19, 987. doi: 10.3390/s19050987

PubMed Abstract | CrossRef Full Text | Google Scholar

Kamran, M. A., Naeem Mannan, M. M., and Jeong, M. Y. (2018). Initial-dip existence and estimation in relation to DPF and data drift. Front. Neuroinform. 12, 96. doi: 10.3389/fninf.2018.00096

PubMed Abstract | CrossRef Full Text | Google Scholar

Khan, M. J., and Hong, K.-S. (2017). Hybrid EEG-FNIRS-based eight-command decoding for BCI: application to quadcopter control. Front. Neurorobot. 11, 6. doi: 10.3389/fnbot.2017.00006

PubMed Abstract | CrossRef Full Text | Google Scholar

Khan, M. U., and Hasan, M. A. (2020). Hybrid EEG-fNIRS BCI fusion using multi-resolution singular value decomposition (MSVD). Front. Hum. Neurosci. 14, 599802. doi: 10.3389/fnhum.2020.599802

PubMed Abstract | CrossRef Full Text | Google Scholar

Kim, G. H., Pham, P. T., Ngo, Q. H., and Nguyen, Q. C. (2021). Neural network-based robust anti-sway control of an industrial crane subjected to hoisting dynamics and uncertain hydrodynamic forces. Int. J. Control Autom. Syst. 19, 1953–1961. doi: 10.1007/s12555-020-0333-9

CrossRef Full Text | Google Scholar

Kwon, J., Shin, J., and Im, C. H. (2020). Toward a compact hybrid brain-computer interface (BCI): performance evaluation of multi-class hybrid EEG-fNIRS BCIs with limited number of channels. PLoS ONE 15, e0230491. doi: 10.1371/journal.pone.0230491

PubMed Abstract | CrossRef Full Text | Google Scholar

Ledesma-Ramirez, C. I., Bojorges-Valdez, E., Yanez-Suarez, O., and Pina-Ramirez, O. (2020). “Recurrence analysis in the detection of continuous task episodes for asynchronous BCI,” in 2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC), 1511–1517. doi: 10.1109/SMC42975.2020.9282907

CrossRef Full Text | Google Scholar

Li, J., Wang, J., Wang, S., Qi, W., Zhang, L., Hu, Y., et al. (2021). Neural approximation-based model predictive tracking control of non-holonomic wheel-legged robots. Int. J. Control Autom. Syst. 19, 372–381. doi: 10.1007/s12555-019-0927-2

CrossRef Full Text | Google Scholar

Liu, Z., Shore, J., Wang, M., Yuan, F., Buss, A., and Zhao, X. (2021). A systematic review on hybrid EEG/fNIRS in brain-computer interface. Biomed. Signal Process. Control 68, 102595. doi: 10.1016/j.bspc.2021.102595

CrossRef Full Text | Google Scholar

Mane, R., Chouhan, T., and Guan, C. (2020). BCI for stroke rehabilitation: motor and beyond. J. Neural Eng. 17, 041001. doi: 10.1088/1741-2552/aba162

PubMed Abstract | CrossRef Full Text | Google Scholar

Mercado, J., Escobedo, L., and Tentori, M. (2021). A BCI video game using neurofeedback improves the attention of children with autism. J. Multimodal User Interfaces 15, 273–281. doi: 10.1007/s12193-020-00339-7

CrossRef Full Text | Google Scholar

Mughal, N. E., Khalil, K., and Khan, M. J. (2021). “fNIRS based multi-class mental workload classification using recurrence plots and CNN-LSTM,” in 2021 International Conference on Artificial Intelligence and Mechatronics Systems (AIMS), 1–6. doi: 10.1109/AIMS52415.2021.9466084

PubMed Abstract | CrossRef Full Text | Google Scholar

Naeem, M., Brunner, C., Leeb, R., Graimann, B., and Pfurtscheller, G. (2006). Seperability of four-class motor imagery data using independent components analysis. J. Neural Eng. 3, 208–216. doi: 10.1088/1741-2560/3/3/003

PubMed Abstract | CrossRef Full Text | Google Scholar

Nayak, S. K., Bit, A., Dey, A., Mohapatra, B., and Pal, K. (2018). A review on the nonlinear dynamical system analysis of Electrocardiogram signal. J. Healthc. Eng. 2018, 6920420. doi: 10.1155/2018/6920420

PubMed Abstract | CrossRef Full Text | Google Scholar

Oh, D. C., and Jo, Y. U. (2021). Classification of hand gestures based on multi-channel EMG by scale Average wavelet transform and convolutional neural network. Int. J. Control Autom. Syst. 19, 1443–1450. doi: 10.1007/s12555-019-0802-1

CrossRef Full Text | Google Scholar

Ortega, P., Zhao, T., and Faisal, A. A. (2020). HYGRIP: full-stack characterization of neurobehavioral signals (fNIRS, EEG, EMG, force, and breathing) during a bimanual grip force control task. Front. Neurosci. 14, 919. doi: 10.3389/fnins.2020.00919

PubMed Abstract | CrossRef Full Text | Google Scholar

Park, J., and Jung, D. J. (2021). Deep convolutional neural network architectures for tonal frequency identification in a lofargram. Int. J. Control Autom. Syst. 19, 1103–1112. doi: 10.1007/s12555-019-1014-4

CrossRef Full Text | Google Scholar

Parro, V. C., and Valdo, L. (2018). Sleep-wake detection using recurrence quantification analysis. Chaos 28, 085706. doi: 10.1063/1.5024692

PubMed Abstract | CrossRef Full Text | Google Scholar

Pham, M., Goering, S., Sample, M., Huggins, J. E., and Klein, E. (2018). Asilomar survey: researcher perspectives on ethical principles and guidelines for BCI research. Brain Comput. Interfaces 5, 97–111. doi: 10.1080/2326263X.2018.1530010

CrossRef Full Text | Google Scholar

Phanikrishna, B. V., Pławiak, P., and Prakash, A. J. (2021). A Brief Review on EEG Signal Pre-Processing Techniques for Real-Time Brain-Computer Interface Applications.

Google Scholar

Puce, A., and Hämäläinen, M. (2017). A review of issues related to data acquisition and analysis in EEG/meg studies. Brain Sci. 7, 58. doi: 10.3390/brainsci7060058

PubMed Abstract | CrossRef Full Text | Google Scholar

Rahman, M. A., Siddik, A. B., Ghosh, T. K., Khanam, F., and Ahmad, M. (2020). A narrative review on clinical applications of fNIRS. J. Digit. Imaging 33, 1167–1184. doi: 10.1007/s10278-020-00387-1

PubMed Abstract | CrossRef Full Text | Google Scholar

Rashid, M., Sulaiman, N., P., P, Abdul Majeed, A., Musa, R. M., Ahmad, A. F., et al. (2020). Current status, challenges, and possible solutions of EEG-based brain-computer interface: a comprehensive review. Front. Neurorobot. 14, 25. doi: 10.3389/fnbot.2020.00025

PubMed Abstract | CrossRef Full Text | Google Scholar

Rejer, I., and Cieszyński, Ł. (2019). Independent component analysis for a low-channel SSVEP-BCI. Pattern Anal. Appl. 22, 47–62. doi: 10.1007/s10044-018-0758-4

CrossRef Full Text | Google Scholar

Saadati, M., Nelson, J., and Ayaz, H. (2020). Convolutional neural network for hybrid fNIRS-EEG mental workload classification. Adv. Intell. Syst. Comput. 953, 221–232. doi: 10.1007/978-3-030-20473-0_22

CrossRef Full Text | Google Scholar

Saeed, A., Naseer, N., and Jabbar, H. (2020). “Improving classification performance of hybrid EEG-fNIRS BCI system by channel optimization,” in PETRA '20: Proceedings of the 13th ACM International Conference on PErvasive Technologies Related to Assistive Environments, 4, 1–4. doi: 10.1145/3389189.3393747

CrossRef Full Text | Google Scholar

Sattar, N. Y., Kausar, Z., Usama, S. A., Farooq, U., and Khan, U. S. (2021). EMG based control of transhumeral prosthesis using machine learning algorithms. Int. J. Control Autom. Syst. 19, 3522–3532. doi: 10.1007/s12555-019-1058-5

CrossRef Full Text | Google Scholar

Sazgar, M., and Young, M. G. (2019). Overview of EEG, Electrode Placement, and Montages. Cham : Springer 117–125. doi: 10.1007/978-3-030-03511-2_5

PubMed Abstract | CrossRef Full Text | Google Scholar

Shin, J., von Lühmann, A., Kim, D.-W., Mehnert, J., Hwang, H.-J., and Müller, K.-R. (2018). Simultaneous acquisition of EEG and NIRS during cognitive tasks for an open access dataset. Sci. Data 5, 180003. doi: 10.1038/sdata.2018.3

PubMed Abstract | CrossRef Full Text | Google Scholar

Shoeibi, A., Ghassemi, N., Khodatars, M., Moridian, P., Alizadehsani, R., Zare, A., et al. (2022). Detection of epileptic seizures on EEG signals using ANFIS classifier, autoencoders and fuzzy entropies. Biomed. Signal Process. Control 73, 103417. doi: 10.1016/j.bspc.2021.103417

CrossRef Full Text | Google Scholar

Shoeibi, A., Sadeghi, D., Moridian, P., Ghassemi, N., Heras, J., Alizadehsani, R., et al. (2021). Automatic diagnosis of schizophrenia in EEG signals using CNN-LSTM models. Front. Neuroinform. 15, 777977. doi: 10.3389/fninf.2021.777977

PubMed Abstract | CrossRef Full Text | Google Scholar

Singh, S. A., Meitei, T. G., Devi, N. D., and Majumder, S. (2021). A deep neural network approach for P300 detection-based BCI using single-channel EEG scalogram images. Phys. Eng. Sci. Med. 44, 1221–1230. doi: 10.1007/s13246-021-01057-4

PubMed Abstract | CrossRef Full Text | Google Scholar

Sun, J., Wang, J., Yang, P., Zhang, Y., and Chen, L. (2021). Adaptive finite time control for wearable exoskeletons based on ultra-local model and radial basis function neural network. Int. J. Control Autom. Syst. 19, 889–899. doi: 10.1007/s12555-019-0975-7

CrossRef Full Text | Google Scholar

Takahashi, K., Yokono, R., Chu, C., Huve, G., and Hashimoto, M. (2021). “fNIRS–Based BCI using deep neural network with an application to deduce the driving mode based on the driver's mental state,” in Proceedings of the International Neural Networks Society (Cham: Springer), 213–219. doi: 10.1007/978-3-030-80568-5_18

CrossRef Full Text | Google Scholar

Tanveer, M. A., Khan, M. J., Qureshi, M. J., Naseer, N., and Hong, K.-S. (2019). Enhanced drowsiness detection using Deep learning: an fNIRS study. IEEE Access 7, 137920–137929. doi: 10.1109/ACCESS.2019.2942838

CrossRef Full Text | Google Scholar

Thyagachandran, A., Kumar, M., Sur, M., Aghoram, R., and Murthy, H. (2020). “Seizure detection using time delay neural networks and lstms,” in 2020 IEEE Signal Processing in Medicine and Biology Symposium (SPMB), 1–5. doi: 10.1109/SPMB50085.2020.9353636

PubMed Abstract | CrossRef Full Text | Google Scholar

Torse, D. A., Khanai, R., and Desai, V. V. (2019). “Classification of epileptic seizures using recurrence plots and machine learning techniques,” in 2019 International Conference on Communication and Signal Processing (ICCSP) 0611–0615. doi: 10.1109/ICCSP.2019.8697989

CrossRef Full Text | Google Scholar

Vourvopoulos, A., Jorge, C., Abreu, R., Figueiredo, P., Fernandes, J. C., and Bermúdez i Badia, S. (2019). Efficacy and brain imaging correlates of an immersive motor imagery BCI-driven VR system for upper limb motor rehabilitation: a clinical case report. Front. Hum. Neurosci. 13, 244. doi: 10.3389/fnhum.2019.00244

PubMed Abstract | CrossRef Full Text | Google Scholar

Wang, X.-W., Nie, D., and Lu, B.-L. (2014). Emotional state classification from EEG data using machine learning approach. Neurocomputing 129, 94–106. doi: 10.1016/j.neucom.2013.06.046

CrossRef Full Text | Google Scholar

Wankhade, M. M., and Chorage, S. S. (2021). “Eye-blink artifact detection and removal approaches for BCI using EEG,” in 2021 International Conference on Recent Trends on Electronics, Information, Communication and Technology (RTEICT) 718–721. doi: 10.1109/RTEICT52294.2021.9574024

PubMed Abstract | CrossRef Full Text | Google Scholar

Yaqub, M. A., Hong, K.-S., Zafar, A., and Kim, C.-S. (2021). Control of transcranial direct current stimulation duration by assessing functional connectivity of near-infrared spectroscopy signals. Int. J. Neural Syst. 32, 2150050. doi: 10.1142/S0129065721500507

PubMed Abstract | CrossRef Full Text | Google Scholar

Zhao, X., Ge, C., Ji, F., and Liu, Y. (2021). Monte Carlo method and quantile regression for uncertainty analysis of wind power forecasting based on Chaos-LS-SVM. Int. J. Control Autom. Syst. 19, 3731–3740. doi: 10.1007/s12555-020-0529-z

CrossRef Full Text | Google Scholar