Emotion recognition based on multimodal physiological electrical signals

Zhuozheng Wang Yihan Wang ^*

• Beijing University of Technology, Beijing, China

With the increasing severity of mental health problems, the application of emotion recognition technology in mental health diagnosis and intervention has gradually gained attention.In this paper, we propose a multimodal emotion recognition method based on the fusion of electroencephalography (EEG) and electrocardiography (ECG) signals, aiming to achieve an accurate classification of emotional states.Aiming at the three dimensions of emotions (affect, arousal, and dominance), a composite neural network (Att-1DCNN-GRU) model combining a one-dimensional convolutional neural network (1DCNN) with an attentional mechanism and a gated recurrent unit (GRU) is designed in this study.The model extracts time-domain, frequency-domain and nonlinear features by fusing EEG and ECG signals, and uses the random forest method for feature screening to improve the accuracy and robustness of emotion recognition.In the experimental part, the proposed model is validated using the DREAMER dataset, and the results show that the model achieves high classification accuracy in all three emotion dimensions, VALENCE, AROUSAL and DOMINANCE, especially in the VALENCE dimension, where the accuracy reaches 95.95%.The fusion model significantly improves the recognition effect compared to the traditional methods using EEG or ECG signals alone.In addition, this study was validated on the DEAP dataset to further verify the generalisation ability and cross-dataset adaptability of the model.Through a series of comparison and ablation experiments, this study demonstrates the advantages of multimodal signal fusion in emotion recognition and shows the great potential of deep learning methods in processing complex physiological signals.The experimental results show that the Att-1DCNN-GRU model has a strong emotion recognition capability and provides valuable technical support for affective computing and mental health management.