A novel multi-feature fusion attention neural network for the recognition of epileptic EEG signals

Epilepsy is a common chronic brain disorder. Detecting epilepsy by observing electroencephalography (EEG) is the main method neurologists use, but this method is time-consuming. EEG signals are non-stationary, nonlinear, and often highly noisy, so it remains challenging to recognize epileptic EEG signals more accurately and automatically. This paper proposes a novel classification system of epileptic EEG signals for single-channel EEG based on the attention network that integrates time-frequency and nonlinear dynamic features. The proposed system has three novel modules. The first module constructs the Hilbert spectrum (HS) with high time-frequency resolution into a two-channel parallel convolutional network. The time-frequency features are fully extracted by complementing the high-dimensional features of the two branches. The second module constructs a grayscale recurrence plot (GRP) that contains more nonlinear dynamic features than traditional RP, fed into the residual-connected convolution module for effective learning of nonlinear dynamic features. The third module is the feature fusion module based on a self-attention mechanism to assign optimal weights to different types of features and further enhance the information extraction capability of the system. Therefore, the system is named HG-SANet. The results of several classification tasks on the Bonn EEG database and the Bern-Barcelona EEG database show that the HG-SANet can effectively capture the contribution degree of the extracted features from different domains, significantly enhance the expression ability of the model, and improve the accuracy of the recognition of epileptic EEG signals. The HG-SANet can improve the diagnosis and treatment efficiency of epilepsy and has broad application prospects in the fields of brain disease diagnosis.

1 Introduction

Epilepsy is a kind of brain disease caused by the abnormal hypersynchronous firing of neurons in the brain, which poses a great threat to the life and health of patients (Acharya et al., 2013). Therefore, an accurate epilepsy diagnosis is of great clinical significance in reducing the harm caused by epileptic seizures to patients. Electroencephalography (EEG) is the most commonly used and effective procedure for diagnosing epilepsy (Noachtar and Rémi, 2009). The diagnosis of epilepsy is a continuous and long-term process (Sazgar and Young, 2019; Jang and Lee, 2020). Moreover, the characteristic pattern of epileptic seizures varies greatly among different patients and even within the same patient (Ren et al., 2023). Therefore, the diagnosis of epilepsy and the pattern analysis of epileptic seizures are usually carried out by neurologists through the detailed analysis of a large number of EEG data by visual detection and manual annotation (Peng et al., 2022). Since EEG signals are nonlinear, non-stationary, highly noisy, and tend to be of long duration, manual judgment to analyze EEG signals is very time-consuming and subject to the subjective judgment of the clinician (Andrzejak et al., 2001; San-Segundo et al., 2019; Hamavar and Asl, 2021). Therefore, more efficient automated detection and analysis methods have received much attention recently. This work will explore automatic and accurate recognition techniques of epileptic EEG signals to assist neurologists in analyzing EEG signals, reduce the burden of neurologists, and improve the efficiency of epilepsy diagnosis and treatment.

For the classification methods of epileptic EEG signals, scholars mainly use statistical analysis-based methods, traditional machine learning and deep learning methods. Gao et al. (2018) propose a statistical analysis-based method to detect seizures. First, they compute joint time-domain features and use the auto-regressive (AR) linear model to model the data. Then, based on the non-parametric statistical test of random power martingale (RPM), the decision is made. Das et al. (2018) extracted time-domain and frequency-domain features of EEG signals based on variational mode decomposition (VMD) and then detected epileptic seizure events by thresholding. Chen et al. (2019) used various distance measurement methods, such as Bhattacharyya distance, to solve the feature similarity of the power spectrum features based on short-time Fourier transform (STFT) of EEG signals at different moments and then detected the EEG signals by null hypothesis test. The above method has the advantages of easy implementation and fast detection speed. Since EEG signals are non-stationary signals, they are easily disturbed by noise generated by brain activity, and the extracted features are easily statistically unstable, leading to inaccurate detection results. In addition, scholars have conducted a lot of research on the classification of epileptic EEG signals based on machine learning and deep learning. Wang et al. (2017) extracted time-domain, frequency-domain, and time-frequency-domain features of EEG signals based on wavelet transform (WT), extracted nonlinear features based on information theory, and then combined the two types of features for epileptic seizure detection by machine learning methods such as k-nearest neighbor classification (KNN) and support vector machine (SVM). Lu et al. (2021) extracted several nonlinear features, such as sample entropy and Higuchi’s fractal dimension, and combined them with SVM for epileptic EEG classification. Then, they found that phase space reconstruction and Poincaré section can improve the recognition accuracy of epileptic EEG signals. Jang and Lee (2020) use the wavelet transform (WT) and phase space reconstruction (PSR) to extract features and then input features to the neural network with weighted fuzzy membership (NEWFM) to detect seizure. Sui et al. (2021) proposed a time-frequency hybrid network (TFHybridNet) based on STFT and a convolutional neural network (CNN) for epileptic focus localization. Varlı and Yılmaz (2023) propose a combined deep learning model based on CNN and long short-term memory (LSTM) to detect seizures. This model uses continuous wavelet transform (CWT) and STFT methods to input the signal conversion time-frequency image to the CNN module and the raw EEG signal to the LSTM module. Compared with traditional machine learning models and statistical analysis-based methods, deep learning models have stronger learning ability and better performance. Current deep learning methods mainly focus on the construction of deep network structures. Combining the non-stationary and nonlinear inherent signal characteristics of EEG with deep learning technology to improve detection accuracy needs further research.

Empirical mode decomposition (EMD) is a non-stationary signal analysis method widely used in the study of epileptic EEG recognition (Mahjoub et al., 2020; Lu et al., 2023). EMD decomposes EEG signals into several linear combinations of intrinsic mode functions (IMF). However, due to the mode mixing problem in EMD, false components in the obtained IMF will adversely affect the EEG analysis. In our previous work, we proposed an improved EMD method named adaptively optimized masking empirical mode decomposition (AOMEMD) (Sun et al., 2024). AOMEMD can effectively alleviate the mode mixing problem of EMD so that the obtained IMFs can effectively capture the underlying physics of EEG. By applying the Hilbert transform (HT) to the IMFs, the Hilbert spectrum (HS) of the EEG can be constructed for high-resolution time-frequency representation of EEG signals. Compared with STFT and CWT methods, this method does not need to set the basis function in advance and has high adaptability and flexibility. Therefore, in this paper, time-frequency features of EEG are represented based on AOMEMD and HT.

The recurrence plot (RP) is a nonlinear time series analysis method that can reveal hidden dynamic characteristics in EEG signals in the form of images (Eckmann et al., 1987; Huang et al., 2023). The traditional RP is a binary symmetric square matrix, usually using the recurrence quantification analysis (RQA) method to extract the structural features of RP for classification recognition. Since the traditional RP cannot reflect detailed time series information, scholars have proposed various improved RP methods. Hatami et al. (2017) skipped the threshold segmentation step in the process of RP construction and combined the gray-level texture image of RP with CNN to classify the time series. Khosla et al. (2022) proposed an un-thresholded recurrence plot (URP) and used the fractal weighted local binary pattern (URP-FWLBP) method to extract the texture features to classify epileptic seizure types. Experiments show that the URP-FWLBP method is better than the traditional method based on RQA. Considering the nonlinear, dynamic, and complex EEG signal, this paper combines the time-frequency feature based on HT with the nonlinear and non-stationary features based on RP to classify epileptic EEG signals.

Therefore, in this paper, we propose a novel system combining nonlinear dynamic features of EEG and time-frequency features extracted by non-stationary time-frequency analysis methods with deep learning techniques to classify epileptic EEG signals automatically. The proposed system is based on a self-attention mechanism to fuse time-frequency features of the HS and nonlinear dynamic features of the grayscale recurrence plot (GRP) to detect epileptic EEG signals for single-channel EEG. So, we call the proposed system HG-SANet. Several classification tasks on the Bonn EEG database and the Bern-Barcelona EEG database verify the performance of the proposed system for the classification of epileptic EEG signals.

2 Materials and methods

In this section, the public dataset used in this paper is first introduced. Secondly, the proposed approach of seizure detection in EEG signals is elaborated. Finally, the experimental setup of this paper is introduced.

2.1 Dataset and data pre-processing

In this paper, two datasets are used. The first dataset is the Bonn EEG time series (Andrzejak et al., 2001). The dataset consists of five sets (denoted A, B, C, D, and E in the original reference) of single-channel EEG segments from healthy volunteers and epilepsy patients, with a signal sampling frequency of 173.61 Hz and a duration of 23.6 s per sample. In order to better distinguish the five subsets, the names of the five subsets are changed to A (denoted EO), B (denoted EC), C (denoted SOE), D (denoted SFE), and E (denoted ES). Each set has 100 recordings and is described in Table 1. Some samples are shown in Figure 1. All EEG signals are digitally band-pass filtered over a range of 0.53∼40 Hz. We used all the samples in this database for experiments to verify the effectiveness of the proposed method in epilepsy detection. We split the data to expand the size of the dataset (Varlı and Yılmaz, 2023). The data is divided into a segment of 512 sample points; the distance between segments is 128 sample points, the last one sample points of the data are deleted, and the final data is divided into 29 segments.

TABLE 1

Table 1. The details of five sets in the Bonn EEG time series.

FIGURE 1

Figure 1. EEG samples from the Bonn EEG database. (A) Example of set EO. (B) Example of set EC. (C) Example of set SOE. (D) Example of set SFE. (E) Example of set ES.

The second dataset is the Bern-Barcelona EEG database (Schindler et al., 2012). The dataset consists of focal and non-focal EEG segments during seizure-free periods from five epilepsy patients, with a signal sampling frequency of 1,024 Hz and a duration of 20 s per sample. Each class has 3,750 samples. If the channel is in the epileptogenic region, its label is focal; otherwise, its label is non-focal. The database is preprocessed as follows: (1) Samples are down-sampled to 512 Hz; (2) All EEG signals are digitally band-pass filtered over a range of 0.5∼150 Hz using a fourth-order Butterworth filter and phase distortions are minimized using forward filtering and backward filtering (Schindler et al., 2012). We used all the samples in this database for experiments to verify the effectiveness of the proposed method in epileptic focus localization. Some samples are shown in Figure 2. According to the previous works (Fasil and Rajesh, 2019), the data is divided into a non-overlapping segment of 1,024 sample points to expand the size of the dataset, and the final data is divided into 10 segments.

FIGURE 2

Figure 2. EEG samples from the Bern-Barcelona EEG database. (A) Example of focal EEG signals. (B) Example of non-focal EEG signals.

All the EEG signals in two datasets are normalized by the following Equation 1 to keep all data at the same scale, helping to improve recognition performance.

$\widetilde{\mathbf{\text{x}}}=\frac{\mathbf{\text{x}}-\mu }{\sigma }(1)$

where x is the input signal, μ is the mean of the signal, and σ is the standard deviation of the signal.

2.2 The proposed framework

The overview of the system based on the proposed HG-SANet is shown in Figure 3. The HG-SANet consists of three modules: EEG time-frequency feature extraction module based on HS and two-channel parallel convolutional neural network (HS-PCNet), nonlinear dynamic feature extraction module based on GRP and residual networks (GRP-ResNet), and multi-domain feature fusion module based on self-attention mechanism (MF-SANet). Below, we first introduce the construction method of HS and GRP and then introduce the network structure of each module.

FIGURE 3

Figure 3. The overview of the proposed epileptic seizure detection system. The cortical model in the figure is from the literature (Andrzejak et al., 2001).

2.2.1 AOMEMD-based Hilbert spectrum

In this part, we use AOMEMD and HT to construct Hilbert spectrum. For a single-channel EEG signal x(t), the AOMEMD is first used to decompose x(t) into a finite number of IMFs and a residue. Therefore, x(t) can be represented as Equation 2:

$x(t)={\sum }_{k=1}^{n_{imf}}{c}_k(t)+r(t)(2)$

where c_k(t) (k = 1, 2,…, n_imf) is the kth IMF and r(t) represents the residue. The frequency of the n_imf IMFs decreases from the first to the n_imfth in order. In this work, we use the AOMEMD without the optimization strategy, which can save computation time while maintaining performance (Sun et al., 2024). The AOMEMD obtains IMFs through the following sifting process and the details of EMD are referred to the work of Huang et al. (1998).

Step 1: Input the signal x(t). Initialize k = 1 and r_k–1(t) = x(t). The number of phases is n_p.

Step 2: Determine the amplitude ${\overline{a}}_k$ and frequency ${\overline{f}}_k$ of the kth group masking signal v_k(t) with resulted IMFs by applying EMD to r_k–1(t).

Step 3: Construct the kth group masking signal $v_{kj}(t)={\overline{a}}_{k}cos[2\mathrm{\pi }{\overline{f}}_{k}t+2\mathrm{\pi }(j-1)/n_p]$, (j = 1, 2,…,n_p). Obtain the kth IMF $c_k(t)=[{\sum }_{j=1}^{n_p}{\text{EMD}}_1(r_{k-1}(t)+v_{kj}(t))]/n_p$, where EMD₁ (⋅) represents to obtain the first IMF using EMD.

Step 4: Update r_k(t) = r_k–1(t)- c_k(t) and k = k+1. If r_k–1(t) fulfils termination criterion, r(t) = r_k–1(t); otherwise, go to step 2 and execute the loop.

For the obtained c_k(t) (k = 1, 2,…, n_imf) by AOMEMD, we use the HT to obtain the instantaneous frequency f_k(t) and instantaneous amplitude a_k(t) of c_k(t). The formula for y_k(t) obtained by applying the HT to c_k(t) is shown in Equation 3 (Huang et al., 1998):

$y_k\left(t\right)=\frac{1}{\mathrm{\pi }}\text{p}.\text{v}.{\int }_{-\mathrm{\infty }}^{+\mathrm{\infty }}\frac{c_k\left(\tau \right)}{\tau -t}\mathit{d}\tau (3)$

where p.v. is the cauchy principal value. Then, f_k(t) and a_k(t) are solved as shown in Equations 4, 5:

$f_k(t)=\frac{1}{2\mathrm{\pi }}\cdot \frac{d}{dt}(\arctan \frac{y_k\left(t\right)}{c_k\left(t\right)})(4)$

$a_k\left(t\right)=\sqrt{c_k^2\left(t\right)+y_k^2\left(t\right)}(5)$

Then the amplitude distribution of x(t) with frequency and time is the Hilbert spectrum (HS), denoted as HS(f, t), expressed as follows Equation 6:

$HS(f,t)=\mathrm{Re}({\sum }_{k=1}^{n_{imf}}{a}_k(t)e^{\text{i}{\int }_{-\mathrm{\infty }}^{t}2\mathrm{\pi }{f}_k(\tau )\mathit{d}\mathrm{\tau }})(6)$

Where Re represents the real part and i is the imaginary unit. HS(f, t) is a two-dimensional matrix with a time resolution equal to the sampling period (Molla and Hirose, 2007). Examples of HS are shown in Figure 4. As shown in Figure 4, the time-frequency distribution of samples from set EC and set ES is quite different.

FIGURE 4

Figure 4. Hilbert spectra from the set EC and set ES of the Bonn EEG database. (A) Hilbert spectrum from the set EC. (B) Hilbert spectrum from the set ES.

2.2.2 Grayscale recurrence plot

For a single-channel EEG signal x(t) of length T_EEG, the RP is computed as the following. First, according to Takens’ embedding theory (Takens, 1985), a phase space is reconstructed for x(t), and a phase point in this space is denoted as s_n and n = 1, 2, …, T_EEG – T_ps (m–1), where T_ps is the time delay and m is the embedding dimension. T_ps and m can be selected using mutual information (MI) and false nearest neighbor (FNN) methods, respectively (He et al., 2023). Second, the RP is defined according to Equation 7 below:

$RP(n,j)=\{\begin{array}{c}\hfill 1,\epsilon \ge ||s_n-s_j||\hfill \\ \hfill 0,\epsilon <||s_n-s_j||\hfill \end{array},$

$n,j=1,2,\mathrm{\dots },T_{EEG}-T_{ps}(m-1)(7)$

where ε is the distance threshold and || ⋅ || is the Euclidean norm. By assigning a black dot to the RP element (n, j) of RP(n, j) = 1 and a white dot to the RP element (n, j) of RP(n, j) = 0, a binary square image of an RP can be obtained, as shown in Figures 5A,C.

FIGURE 5

Figure 5. Examples of recurrence plots and grayscale recurrence plots from the set EC and set ES. (A) Recurrence plot from set EC. (B) Grayscale recurrence plot from set EC. (C) Recurrence plot from set ES. (D) Grayscale recurrence plot from set ES.

Binary square images constructed using the threshold method lose a lot of information, so we convert the RP to a grayscale intensity image (named grayscale RP, GRP). The examples of GRP are shown in Figures 5B,D. The GRP is defined according to Equation 8 below (Chen and Shi, 2019):

$GRP(n,j)=\frac{||s_n-s_j||-\min (||s_n-s_j||)}{\max \left(||s_n-s_j||\right)-\min (||s_n-s_j||)},$

$n,j=1,2,\mathrm{\dots },T_{EEG}-T_{ps}(m-1)(8)$

2.2.3 Network structure of the HG-SANet

In this section, the network structure in each module of the HG-SANet is described in detail.

The first HS-PCNet module inputs the HS built in section 2.2.1 into a parallel two-channel CNN network containing different convolutional kernels. CNN overcomes the limitation of insufficient feature extraction ability of machine learning methods through simultaneous shift calculation of convolutional kernel in the time and frequency dimensions of feature maps (Zhang et al., 2015). It has been used in time-frequency feature extraction of EEG signals (Sui et al., 2021). Therefore, in this section, we use CNN to further extract the high-level time-frequency features of HS. For HS, we design a parallel two-channel CNN network containing different types of convolutional kernels for feature extraction. Two types of convolution kernels are set as [N_kernel, 1] and [N_kernel, N_kernel]. As EEG signals comprise time-series data, we construct a convolution kernel of size [N_kernel, 1] to make feature extraction pay more attention to changes in the time domain. The convolution kernel of size [N_kernel, N_kernel] slides synchronously in the time domain and frequency domain dimensions of the HS to retain its original time-frequency characteristics. The time-frequency features are fully extracted by complementing the high-dimensional features of the two branches. The structure and details of the HS-PCNet module are shown in Figure 6A. The structure and parameter settings in each CNN block of the HS-PCNet module are shown in Table 2. In Table 2, the serial number corresponds to the serial number in Figure 6A. Each CNN block has a batch normalization layer and a ReLU activation layer between the 2D convolution (Conv 2D) and max pooling layers, which are omitted to save space. For the HS-PCNet module, the batch normalization layer normalizes the input data in small batches to speed up the training of the HS-PCNet and reduce the sensitivity to the network initialization. The max pooling layer performs downsampling by dividing the feature map into rectangular pooling regions and calculating the maximum value for each region, which helps reduce overfitting. The dropout layer makes the activation value of a certain neuron stop working with a certain probability, helping to prevent the HS-PCNet from overfitting (Krizhevsky et al., 2017).

FIGURE 6

Figure 6. The structure and details of the HS-PCNet module and GRP-ResNet module. (A) The overall structure of the HS-PCNet module. (B) The overall structure of the GRP-ResNet module.

TABLE 2

Table 2. The structure and parameter settings in each CNN block of HS-PCNet module.

A large number of studies have proved the advantage of residual networks in the field of image recognition (He et al., 2015). Therefore, the second GRP-ResNet module inputs the GRP in section 2.2.2 into a CNN with residual connections to fully learn the nonlinear dynamic features in the GRP. The convolutional module in the GRP-ResNet can use the receptive field of neurons to extract high-level local feature representation of the GRP, and the residual module allows cross-layer propagation, which can avoid overfitting caused by too many layers in the network, and will not lose important information in the feature (He et al., 2015). The overall structure of the GRP-ResNet module is shown in Figure 6B. The structure and parameter settings in each residual block and CNN block are shown in Table 3, and the serial number corresponds to the serial number in Figure 6B. In each block, there is a batch normalization layer after the 2D convolution (Conv 2D) layers, which is omitted to save space.

TABLE 3

Table 3. The structure and parameter settings in each residual block of GRP-ResNet module.

Research shows that the self-attention mechanism (Vaswani et al., 2017) can help the network select important features and assign higher weights to these important features to improve the performance of downstream tasks (Lin et al., 2022; Yang Q. et al., 2023). Therefore, in the third MF-SANet, a feature fusion module based on a multi-head self-attention mechanism is proposed to assign optimal weights to different types of features obtained by the HS-PCNet module and GRP-ResNet module to enhance the information extraction capability of HG-SANet further. The feature fusion formulas are calculated as follows. First, the features extracted from the HS-PCNet module and GRP-ResNet module are concatenated and the concatenated features are denoted as Feature_initial. In the self-attention mechanism, there are three kinds of important input queries, keys and values, denoted as QUE, KEY, and VAL, respectively. They are calculated as Equations 9–11 (Lin et al., 2022):

${\mathit{\text{QUE}}}_{\mathit{j}}=\mathit{\text{Feature}}\mathrm{\_}\mathit{\text{initial}}\times {\mathit{\text{W}}}_{\mathit{j}}^{\mathit{QUE}}(9)$

${\mathit{\text{KEY}}}_{\mathit{j}}=\mathit{\text{Feature}}\mathrm{\_}\mathit{\text{initial}}\times {\mathit{\text{W}}}_{\mathit{j}}^{\mathit{KEY}}(10)$

${\mathit{\text{VAL}}}_{\mathit{j}}=\mathit{\text{Feature}}\mathrm{\_}\mathit{\text{initial}}\times {\mathit{\text{W}}}_{\mathit{j}}^{\mathit{VAL}}(11)$

where j = 1, 2, …, N_head and N_head is the number of attention heads. ${\mathit{\text{W}}}_{\mathit{j}}^{\mathit{QUE}}$, ${\mathit{\text{W}}}_{\mathit{j}}^{\mathit{KEY}}$, and ${\mathit{\text{W}}}_{\mathit{j}}^{\mathit{VAL}}$are the parameter matrices. Then, the features of the final output are calculated as Equation 12:

${\mathit{\text{Feature}}}_{\mathit{final}}=\text{Concat}({\mathit{\text{HEAD}}}_1,{\mathit{\text{HEAD}}}_2,\mathrm{\dots },{\mathit{\text{HEAD}}}_{N_{head}}){\mathit{\text{W}}}^{\mathit{o}}(12)$

where W^o is a parameter matric and Concat(⋅) is the concatenating operation. The HEAD_j is calculated as Equation 13

${\mathit{\text{HEAD}}}_{\mathit{j}}=\mathrm{softmax}(\frac{{\mathit{\text{QUE}}}_{\mathit{j}}\mathit{\text{KEY}}_{\mathit{j}}{}^{\mathrm{T}}}{\sqrt{d_{KEY}}}){\mathit{\text{VAL}}}_{\mathit{j}}(13)$

where d_KEY is the dimension of keys.

In the classification layer based on the full connection layer, the activation function after the full connection layer is the softmax function.

2.3 Experiment configurations

2.3.1 Evaluation metrics

In this paper, epileptic EEG recognition is evaluated using precision (P), recall (R), accuracy (Acc), and specificity (SP) (Sriraam and Raghu, 2017; Gao et al., 2018). The sensitivity and recall are calculated using the same formula, so we no longer calculate sensitivity separately. Precision focuses on evaluating the percentage of true positive samples in all predicted positive samples. Recall focuses on the percentage of all positive samples that are successfully predicted to be positive. Accuracy is the proportion of correctly classified samples in total samples. The specificity is the proportion of all negative samples predicted correctly to all actual negative samples. These metrics are calculated as shown in Equations 14–17.

$P=\frac{N_{TP}}{N_{TP}+N_{FP}}(14)$

$R=\frac{N_{TP}}{N_{TP}+N_{FN}}(15)$

$Acc=\frac{N_{TP}+N_{TN}}{N_{TP}+N_{TN}+N_{FN}+N_{FP}}(16)$

$SP=\frac{N_{TN}}{N_{TN}+N_{FP}}(17)$

where N_TP is the number of true positive (TP) samples, N_TN is the number of true negative (TN) samples, N_FP is the number of false positive (FP) samples, and N_FN is the number of false negative (FN) samples.

In the decision stage, the HG-SANet gives prediction labels for all short segments of each sample. Finally, based on the prediction labels of short segments, the majority voting method is used to make the final prediction for the category of each test sample.

2.3.2 Model parameter setting

Parameters of the HG-SANet in the training process are set as follows. Adaptive moment estimation (Adam) optimizer is used to train the HG-SANet. The epoch used for training is 30, and the mini-batch size used for each training iteration is 32. The learning rate is 0.001. The cross-entropy loss function is used as the loss function. We reduce the overfitting of the HG-SANet by adding the regularization term of the weight to the loss function. The number of heads in the attention module is set to 2. In the testing process, the testing sample is input into the proposed system trained by the training set as shown in Figure 3 to obtain the final recognition result. The ten-fold cross-validation is used to obtain an unbiased evaluation of classification performance.

3 Results and discussion

3.1 Analysis of the proposed model

In this part, we designed several ablation experiments to analyze the effects of each module of the model. First, based on clinical applications and experiments conducted by scholars in the Bonn dataset (Ma et al., 2021), we selected three typical detection tasks to analyze our approach. The three typical tasks are: (1) Two-class detection task: distinguish between set EO and set ES, comparing the performance of methods to distinguish between healthy subjects and epileptic patients. (2) Two-class detection task: distinguish between set SOE and set ES, comparing the performance of methods to distinguish between non-epileptic interictal EEG and seizures in epileptic patients. (3) Three-class detection task: distinguish between normal (include set EO and EC), interictal activities (include set SOE and set SFE), and epileptic seizures (include set ES). This three-class task can be used not only to find epilepsy patients but also to automatically diagnose their symptoms, which is of great significance.

In order to verify the performance of each module, we designed the following experiments: (1) Use the RQA method to extract the structural features of RP (Pham, 2020) and input these features into a SVM to classify three-class detection task (denotes as RQA-SVM). (2) A fully connected classification layer is added to the back of the GRP-ResNet module to classify the three-class detection task (denoted as GRP-ResNet). (3) A fully connected classification layer is added to the back of the HS-PCNet module to classify the three-class detection task (denoted as HS-PCNet). (4) The Hilbert Spectrum of the HS-PCNet module is replaced with a CWT-based scalogram (denoted as CWT-PCNet). Then, a fully connected classification layer is added to the back of the CWT-PCNet module to classify the three-class detection task. The Morlet wavelet is used as the mother wavelet (Varlı and Yılmaz, 2023). CWT is an important method for EEG signal analysis. We designed the fourth experiment to compare AOMEMD method and CWT method. (5) The features extracted from the HS-PCNet module and GRP-ResNet module are concatenated and the concatenated features are input to a fully connected classification layer to classify three-class detection task, denotes as HG-SANet without self-attention mechanism (HG-SANet-wo). (6) Use the HG-SANet to classify all three typical tasks. The classification results are shown in Table 4 and Figure 7.

TABLE 4

Table 4. Classification results of the proposed HG-SANet for the three typical tasks.

FIGURE 7

Figure 7. Results of ablation experiments in the three-class detection task of the Bonn EEG time series. (A) Precision for each of the three classes. (B) Recall for each of the three classes. (C) Overall results of the three classes.

Figure 7 shows the results of the ablation experiments designed in this section for the three-class detection task. The average results of the ten-fold cross-validation method are shown in Figure 7. As shown in Figure 7, each module can detect seizures, and the HG-SANet gives the best results in terms of overall performance. The best result of all the 10-fold cross-validation results is 100%. Combining the nonlinear features based on GRP-ResNet with the time-frequency features based on HS-PCNet improves the average accuracy, precision, and recall of the model. Moreover, the average accuracy, precision, and recall of the fusion model with added attention mechanism are increased by 0.8%, 0.67%, and 0.7%, respectively, compared with the fusion model without added attention mechanism. The results in Figure 7 demonstrate the validity of the proposed HG-SANet. The performance of RQA-SVM is the worst. The dimension of the RQA features is only eight. The information expression ability of RQA features is limited. The performance of CWT-PCNet is worse than HS-PCNet. For set ES, the recall of CWT-PCNet is the worst, only 86%. In Table 4, we compare the average performance of the proposed HG-SANet under different classification tasks. As shown in Table 4, in the two-class detection task of identifying set EO and set ES, our method achieves 100% recognition rate.

3.2 Comparison with SOTA methods for the classification of epileptic EEG signals

To further validate the effectiveness of the proposed method, we compare the proposed HG-SANet with other state-of-the-art (SOTA) methods on the Bonn EEG time series and the Bern-Barcelona EEG database. The results of the Bonn EEG time series are shown in Table 5. All the comparison methods include deep learning methods and traditional machine learning methods. The results of the proposed HG-SANet in Table 5 are the mean of the 10-cross validation results. As shown in Table 5, the proposed HG-SANet performs best on all the tasks. The proposed HG-SANet has a high recall value, which indicates that the method proposed in this paper can detect the seizure signal as much as possible, which is of great significance for diagnosing the disease. The proposed model can distinguish not only the EEG data of epileptic patients and non-epileptic persons but also the EEG data from epileptic seizures and seizure-free intervals in epileptic patients. When conducting comparative experiments, it was also found that deep learning-based methods outperformed other types of methods.

TABLE 5

Table 5. Comparison of different methods on the Bonn EEG time series database.

The results of the Bern-Barcelona EEG database are shown in Table 6. A binary classification task is performed on this database (focal vs. non-focal). The comparison methods include deep learning, traditional machine learning, and statistical modeling methods. The results of the proposed HG-SANet in Table 6 are the mean of the 10-cross validation results. As seen from Table 6, the performance of the proposed method in epileptic focal location is better than that of all the compared methods. It is also seen on the Bern-Barcelona EEG database that deep learning methods outperform other methods. The results of the two datasets show that the proposed method can classify multiple brain states associated with epilepsy. The proposed method can be used in automatic epileptic seizure detection, the epileptic focal location, and other related applications in diagnosing epilepsy diseases.

TABLE 6

Table 6. Comparison of different methods on the Bern-Barcelona EEG database.

4 Conclusion

In this study, a novel model named HG-SANet is developed for the automated detection of epileptic EEG signals. This innovative model proposes a multi-channel parallel feature extraction module based on multi-domain features and a feature fusion module based on an attention mechanism. Through many experiments, the proposed network structure can capture the non-stationary nonlinear properties of epilepsy EEG well and realize the automatic and high-accuracy detection of epileptic seizures, epileptic focus localization, and EEG classification. The method proposed in this paper is of great significance to detecting and warning brain disease. In the future, we will research other epilepsy-related issues, such as seizure prediction, and further reduce the time complexity of the method and make the method better applied to real-time seizure prediction.

Data availability statement

The original contributions presented in the study are included in the article/supplementary material, further inquiries can be directed to the corresponding author.

Ethics statement

Ethical approval was not required for the study involving humans in accordance with the local legislation and institutional requirements. Written informed consent to participate in this study was not required from the participants or the participants’ legal guardians/next of kin in accordance with the national legislation and the institutional requirements.

Author contributions

CS: Methodology, Writing – original draft, Writing – review & editing, Conceptualization, Data curation, Formal analysis, Validation, Visualization. CX: Writing – review & editing, Methodology, Formal analysis. HoL: Data curation, Writing – review & editing, Formal analysis. HB: Validation, Writing – review & editing. LM: Supervision, Writing – review & editing, Conceptualization. HaL: Supervision, Writing – review & editing, Conceptualization, Funding acquisition, Resources.

Funding

The author(s) declare that financial support was received for the research, authorship, and/or publication of this article. This study was supported by the National Natural Science Foundation of China (Grant No. U20A20383), Basic and Applied Basic Research of Guangdong (Grant No. 2021B1515120052), and Shenzhen Foundational Research Funding (Grant No. JCYJ20200109150814370).

Acknowledgments

The authors are grateful for the reviewers who made constructive comments.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

Acharya, U. R., Vinitha Sree, S., Swapna, G., Martis, R. J., and Suri, J. S. (2013). Automated EEG analysis of epilepsy: A review. Knowledge Based Syst. 45, 147–165. doi: 10.1016/j.knosys.2013.02.014

Crossref Full Text | Google Scholar

Andrzejak, R. G., Lehnertz, K., Mormann, F., Rieke, C., David, P., and Elger, C. E. (2001). Indications of nonlinear deterministic and finite-dimensional structures in time series of brain electrical activity: Dependence on recording region and brain state. Phys. Rev. E 64:61907. doi: 10.1103/PhysRevE.64.061907

PubMed Abstract | Crossref Full Text | Google Scholar

Chen, G. Y., Lu, G. L., Shang, W., and Xie, Z. H. (2019). Automated change-point detection of EEG signals based on structural time-series analysis. IEEE Access 7, 180168–180180. doi: 10.1109/ACCESS.2019.2956768

Crossref Full Text | Google Scholar

Chen, W., and Shi, K. (2019). A deep learning framework for time series classification using relative position matrix and convolutional neural network. Neurocomputing 359, 384–394. doi: 10.1016/j.neucom.2019.06.032

Crossref Full Text | Google Scholar

Das, P., Manikandan, M. S., and Ramkumar, B. (2018). “Detection of epileptic seizure event in EEG signals using variational mode decomposition and mode spectral entropy,” in Proceedings of the IEEE 13th international conference on industrial and information systems (ICIIS), (Rupnagar), 42–47.

Google Scholar

Eckmann, J. P., Oliffson Kamphorst, S., and Ruelle, D. (1987). Recurrence plots of dynamical systems. Europhys. Lett. 4:973. doi: 10.1209/0295-5075/4/9/004

Crossref Full Text | Google Scholar

Fasil, O. K., and Rajesh, R. (2019). Time-domain exponential energy for epileptic EEG signal classification. Neurosci. Lett. 694, 1–8. doi: 10.1016/j.neulet.2018.10.062

PubMed Abstract | Crossref Full Text | Google Scholar

Fu, K., Qu, J., Chai, Y., and Zou, T. (2015). Hilbert marginal spectrum analysis for automatic seizure detection in EEG signals. Biomed. Signal Process. Control 18, 179–185. doi: 10.1016/j.bspc.2015.01.002

Crossref Full Text | Google Scholar

Gao, Z., Lu, G. L., Yan, P., Lyu, C., Li, X. Y., Shang, W., et al. (2018). Automatic change detection for real-time monitoring of EEG signals. Front. Physiol. 9:325. doi: 10.3389/fphys.2018.00325

PubMed Abstract | Crossref Full Text | Google Scholar

Hamavar, R., and Asl, B. M. (2021). Seizure onset detection based on detection of changes in brain activity quantified by evolutionary game theory model. Comput. Meth. Programs Biomed. 199:105899. doi: 10.1016/j.cmpb.2020.105899

PubMed Abstract | Crossref Full Text | Google Scholar

Hatami, N., Gavet, Y., and Debayle, J. (2017). “Classification of time-series images using deep convolutional neural networks,” in Proceedings of the international conference on machine vision, (Cham).

Google Scholar

He, K., Zhang, X., Ren, S., and Sun, J. (2015). “Deep residual learning for image recognition,” in Proceedings of the 2016 IEEE conference on computer vision and pattern recognition (CVPR), (Las Vegas, NV), 770–778.

Google Scholar

He, Q., Yu, F., Chang, J., and Ouyang, C. (2023). Fuzzy granular recurrence plot and quantification analysis: A novel method for classification. Pattern Recogn. 139:109456. doi: 10.1016/j.patcog.2023.109456

Crossref Full Text | Google Scholar

Huang, N. E., Shen, Z., Long, S. R., Wu, M. C., Shih, H. H., Zheng, Q., et al. (1998). The empirical mode decomposition and the Hilbert spectrum for nonlinear and non-stationary time series analysis. Proc. R. Soc. Lond. Ser. A Math. Phys. Eng. Sci. 454, 903–995. doi: 10.1098/rspa.1998.0193

PubMed Abstract | Crossref Full Text | Google Scholar

Huang, W., Yan, G., Chang, W., Zhang, Y., and Yuan, Y. (2023). EEG-based classification combining Bayesian convolutional neural networks with recurrence plot for motor movement/imagery. Pattern Recogn. 144:109838. doi: 10.1016/j.patcog.2023.109838

Crossref Full Text | Google Scholar

Jang, S., and Lee, S. (2020). Detection of epileptic seizures using wavelet transform, peak extraction and PSR from EEG signals. Symmetry 12:1239. doi: 10.3390/sym12081239

Crossref Full Text | Google Scholar

Khan, P., Khan, Y., Kumar, S., Khan, M. S., and Gandomi, A. H. (2021). HVD-LSTM based recognition of epileptic seizures and normal human activity. Comput. Biol. Med. 136:104684. doi: 10.1016/j.compbiomed.2021.104684

PubMed Abstract | Crossref Full Text | Google Scholar

Khosla, A., Khandnor, P., and Chand, T. (2022). EEG-based automatic multi-class classification of epileptic seizure types using recurrence plots. Expert Syst. 39:e12923. doi: 10.1111/exsy.12923

Crossref Full Text | Google Scholar

Krizhevsky, A., Sutskever, I., and Hinton, G. E. (2017). ImageNet classification with deep convolutional neural networks. Commun. ACM 60, 84–90. doi: 10.1145/3065386

Crossref Full Text | Google Scholar

Lin, S., Wang, Y., Zhang, L., Chu, Y., Liu, Y., Fang, Y., et al. (2022). MDF-SA-DDI: Predicting drug–drug interaction events based on multi-source drug fusion, multi-source feature fusion and transformer self-attention mechanism. Brief. Bioinf. 23:bbab421. doi: 10.1093/bib/bbab421

PubMed Abstract | Crossref Full Text | Google Scholar

Lu, X., Wang, T., Ye, M., Huang, S., Wang, M., and Zhang, J. (2023). Study on characteristic of epileptic multi-electroencephalograph base on Hilbert-Huang transform and brain network dynamics. Front. Neurosci. 17:1117340. doi: 10.3389/fnins.2023.1117340

PubMed Abstract | Crossref Full Text | Google Scholar

Lu, X., Zhang, J., Huang, S., Lu, J., Ye, M., and Wang, M. (2021). Detection and classification of epileptic EEG signals by the methods of nonlinear dynamics. Chaos Solitons Fract. 151:111032. doi: 10.1016/j.chaos.2021.111032

Crossref Full Text | Google Scholar

Ma, M., Cheng, Y., Wei, X., Chen, Z., and Zhou, Y. (2021). Research on epileptic EEG recognition based on improved residual networks of 1-D CNN and indRNN. BMC Med. Inform. Decis. Mak. 21:100. doi: 10.1186/s12911-021-01438-5

PubMed Abstract | Crossref Full Text | Google Scholar

Mahjoub, C., Le Bouquin, Jeannès, R., Lajnef, T., and Kachouri, A. (2020). Epileptic seizure detection on EEG signals using machine learning techniques and advanced preprocessing methods. Biomed. Tech. 65, 33–50. doi: 10.1515/bmt-2019-0001

PubMed Abstract | Crossref Full Text | Google Scholar

Molla, M. K. I., and Hirose, K. (2007). Single-mixture audio source separation by subspace decomposition of Hilbert spectrum. IEEE Trans. Audio Speech Lang. Process. 15, 893–900. doi: 10.1109/TASL.2006.885254

Crossref Full Text | Google Scholar

Noachtar, S., and Rémi, J. (2009). The role of EEG in epilepsy: A critical review. Epilepsy Behav. 15, 22–33. doi: 10.1016/j.yebeh.2009.02.035

PubMed Abstract | Crossref Full Text | Google Scholar

Peng, H., Li, C., Chao, J., Wang, T., Zhao, C., Huo, X., et al. (2021). A novel automatic classification detection for epileptic seizure based on dictionary learning and sparse representation. Neurocomputing 424, 179–192. doi: 10.1016/j.neucom.2019.12.010

Crossref Full Text | Google Scholar

Peng, R., Jiang, J., Kuang, G., Du, H., Wu, D., and Shao, J. (2022). EEG-based automatic epilepsy detection: Review and outlook. Acta Autom. Sin. 48, 335–350. doi: 10.16383/j.aas.c200745

Crossref Full Text | Google Scholar

Pham, T. D. (2020). “Recurrence plots,” in Proceedings of the Fuzzy recurrence plots and networks with applications in biomedicine, (Cham: Springer), 9–16. doi: 10.1007/978-3-030-37530-0_2

Crossref Full Text | Google Scholar

Ren, Q. R., Sun, X. F., Fu, X. Q., Zhang, S. D., Yuan, Y. Y., Wu, H., et al. (2023). A review of automatic detection of epilepsy based on EEG signals. J. Semicond. 44:121401. doi: 10.1088/1674-4926/44/12/121401

Crossref Full Text | Google Scholar

San-Segundo, R., Gil-Martín, M., D’Haro-Enríquez, L. F., and Pardo, J. M. (2019). Classification of epileptic EEG recordings using signal transforms and convolutional neural networks. Comput. Biol. Med. 109, 148–158. doi: 10.1016/j.compbiomed.2019.04.031

PubMed Abstract | Crossref Full Text | Google Scholar

Sazgar, M., and Young, M. G. (2019). “Seizures and epilepsy,” in Proceedings of the absolute epilepsy and EEG rotation review, (Cham: Springer), 9–46. doi: 10.1007/978-3-030-03511-2_2

Crossref Full Text | Google Scholar

Schindler, K., Rummel, C., and Andrzejak, R. G. (2012). Nonrandomness, nonlinear dependence, and nonstationarity of electroencephalographic recordings from epilepsy patients. Phys. Rev. E 86:46206. doi: 10.1103/PhysRevE.86.046206

PubMed Abstract | Crossref Full Text | Google Scholar

Sharma, R., Pachori, R. B., and Acharya, U. R. (2015). Application of entropy measures on intrinsic mode functions for the automated identification of focal electroencephalogram signals. Entropy 17, 669–691. doi: 10.3390/e17020669

Crossref Full Text | Google Scholar

Sriraam, N., and Raghu, S. (2017). Classification of focal and non focal epileptic seizures using multi-features and SVM classifier. J. Med. Syst. 41:160. doi: 10.1007/s10916-017-0800-x

PubMed Abstract | Crossref Full Text | Google Scholar

Sui, L., Zhao, X., Zhao, Q., Tanaka, T., Cao, J., and Fornaro, M. (2021). Hybrid convolutional neural network for localization of epileptic focus based on iEEG. Neural Plast. 2021:6644365. doi: 10.1155/2021/6644365

PubMed Abstract | Crossref Full Text | Google Scholar

Sun, C., Li, H., Xu, C., Ma, L., and Li, H. (2024). Adaptively optimized masking EMD for separating intrinsic oscillatory modes of nonstationary signals. IEEE Signal Process. Lett. 31, 216–220. doi: 10.1109/LSP.2023.3347146

Crossref Full Text | Google Scholar

Takens, F. (1985). On the numerical determination of the dimension of an attractor. Lect. Notes Math. 1125, 99–106. doi: 10.1007/BFb00756

Crossref Full Text | Google Scholar

Türk, Ö, and Özerdem, M. S. (2019). Epilepsy detection by using scalogram based convolutional neural network from EEG signals. Brain Sci. 9:115. doi: 10.3390/brainsci9050115

PubMed Abstract | Crossref Full Text | Google Scholar

Ullah, I., Hussain, M., Qazi, E., and Aboalsamh, H. (2018). An automated system for epilepsy detection using EEG brain signals based on deep learning approach. Expert Syst. Appl. 107, 61–71. doi: 10.1016/j.eswa.2018.04.021

Crossref Full Text | Google Scholar

Varlı, M., and Yılmaz, H. (2023). Multiple classification of EEG signals and epileptic seizure diagnosis with combined deep learning. J. Comput. Sci. 67:101943. doi: 10.1016/j.jocs.2023.101943

Crossref Full Text | Google Scholar

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., et al. (2017). “Attention is all you need,” in Proceedings of the 31st international conference on neural information processing systems (NIPS’17), (Red Hook, NY: Curran Associates Inc), 6000–6010.

Google Scholar

Wang, L. N., Xue, W. N., Li, Y., Luo, M. L., Huang, J., Cui, W. G., et al. (2017). Automatic epileptic seizure detection in EEG signals using multi-domain feature extraction and nonlinear analysis. Entropy 19:222. doi: 10.3390/e19060222

Crossref Full Text | Google Scholar

Yang, Q., Tang, B., Shen, Y., and Li, Q. (2023). Self-attention parallel fusion network for wind turbine gearboxes fault diagnosis. IEEE Sens. J. 23, 23210–23220. doi: 10.1109/JSEN.2023.3308971

Crossref Full Text | Google Scholar

Yang, Y., Li, F., Luo, J., Qin, X., and Huang, D. (2023). Epileptic focus localization using transfer learning on multi-modal EEG. Front. Comput. Neurosci. 17:1294770. doi: 10.3389/fncom.2023.1294770

PubMed Abstract | Crossref Full Text | Google Scholar

Zeng, M., Zhao, C., and Meng, Q. (2019). Detecting seizures from EEG signals using the entropy of visibility heights of hierarchical neighbors. IEEE Access 7, 7889–7896. doi: 10.1109/ACCESS.2019.2890895

Crossref Full Text | Google Scholar

Zhang, H., Mcloughlin, I., and Song, Y. (2015). “Robust sound event recognition using convolutional neural networks,” in Proceedings of the 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), (South Brisbane, QLD), 559–563. doi: 10.1109/ICASSP.2015.7178031

Crossref Full Text | Google Scholar

Zhao, W., Zhao, W., Wang, W., Jiang, X., Zhang, X., Peng, Y., et al. (2020). A novel deep neural network for robust detection of seizures using EEG signals. Comput. Math. Method Med. 2020:9689821. doi: 10.1155/2020/9689821

PubMed Abstract | Crossref Full Text | Google Scholar

Zhao, X., Solé-Casals, J., Zhao, Q., Cao, J., and Tanaka, T. (2021). “Multi-feature fusion for epileptic focus localization based on tensor representation,” in Proceedings of the 2021 Asia-pacific signal and information processing association annual summit and conference (APSIPA ASC), (Tokyo), 1323–1327.

Google Scholar