EEG-based Emotion recognition holds significant promise for applications in human-computer interaction, medicine, and neuroscience. While deep learning has shown potential in this field, current approaches usually rely on large-scale high-quality labeled datasets, limiting the performance of deep learning. Self-supervised learning offers a solution by automatically generating labels, but its inter-subject generalizability remains under-explored. For this reason, our interest lies in offering a self-supervised learning paradigm with better inter-subject generalizability. Inspired by recent efforts in combining low-level and high-level tasks in deep learning, we propose a cascaded self-supervised architecture for EEG emotion recognition. Then, we introduce a low-level task, time-to-frequency reconstruction (TFR). This task leverages the inherent time-frequency relationship in EEG signals. Our architecture integrates it with the high-level contrastive learning modules, performing self-supervised learning for EEG-based emotion recognition. Experiment on DEAP and DREAMER datasets demonstrates superior performance of our method over similar works. The outcome results also highlight the indispensability of the TFR task and the robustness of our method to label scarcity, validating the effectiveness of the proposed method.
Both the temporal dynamics and spatial correlations of Electroencephalogram (EEG), which contain discriminative emotion information, are essential for the emotion recognition. However, some redundant information within the EEG signals would degrade the performance. Specifically,the subjects reach prospective intense emotions for only a fraction of the stimulus duration. Besides, it is a challenge to extract discriminative features from the complex spatial correlations among a number of electrodes. To deal with the problems, we propose a transformer-based model to robustly capture temporal dyna…
Deep Learning has impacted various fields especially in bio-medical applications. Deep learning algorithms work well with both structured and unstructured data. Especially, convolutional neural network work well with signal-based data like EEG data. These types of data may or may not follow a pattern in their data. Algorithms like CNN help feature engineering and simplistic interpretation of the data. These algorithms are also better in comparison to other algorithms when generalised to a data belonging to larger data set.
In this study, the Multivariate Empirical Mode Decomposition (MEMD) is applied to multichannel EEG to obtain scale-aligned intrinsic mode functions (IMFs) as input features for emotion detection. The IMFs capture local signal variation related to emotion changes. Among the extracted IMFs, the high oscillatory ones are found to be significant for the intended task. The Marginal Hilbert spectrum (MHS) is computed from the selected IMFs. A 3D convolutional neural network (CNN) is adopted to perform emotion detection with spatial-temporal-spectral feature representations that are constructed by st…
Affective computing is an important branch of artificial intelligence, and with the rapid development of brain computer interface technology, emotion recognition based on EEG signals has received broad attention. It is still a great challenge to effectively explore the multi-dimensional information in the EEG data in spite of a large number of deep learning methods. In this paper, we propose a deep model called Attention-based Multiple Dimensions EEG Transformer (AMDET), which can exploit the complementarity among the spectral-spatial-temporal features of EEG data by employing the multi-dimens…
The spatial correlations and the temporal contexts are indispensable in Electroencephalogram (EEG)-based emotion recognition. However, the learning of complex spatial correlations among several channels is a challenging problem. Besides, the temporal contexts learning is beneficial to emphasize the critical EEG frames because the subjects only reach the prospective emotion during part of stimuli. Hence, we propose a novel Spatial-Temporal Information Learning Network (STILN) to extract the discriminative features by capturing the spatial correlations and temporal contexts. Specifically, the ge…
Accurate forecasting of an electroencephalogram (EEG) time series is crucial for the correct diagnosis of neurological disorders such as seizures and epilepsy. Since the EEG time series is chaotic, most traditional machine learning algorithms have failed to forecast its next steps accurately. Thus, we suggest a model, which has formed by taking inspiration from the neural structures that underlie feelings (emotional states), to forecast EEG time series. The model, which is referred to as emotion-inspired deep structure (EiDS), can be used to predict both short- and long-term of EEG time series…
University of Electronic Science and Technology of China（电子科技大学）
Emotion recognition based on EEG (electroencephalography) has been widely used in human-computer interaction, distance education and health care. However, the conventional methods ignore the adjacent and symmetrical characteristics of EEG signals, which also contain salient information related to emotion. In this paper, a spatial folding ensemble network (SFE-Net) is presented for EEG feature extraction and emotion recognition. Firstly, for the undetected area between EEG electrodes, an improved Bicubic-EEG interpolation algorithm is developed for EEG channels information completion, which all…
Khalifa University（哈利法大学）；Università degli Studi di Milano（米兰大学）
The major aim of this paper is to explain the data poisoning attacks using label-flipping during the training stage of the electroencephalogram (EEG) signal-based human emotion evaluation systems deploying Machine Learning models from the attackers' perspective. Human emotion evaluation using EEG signals has consistently attracted a lot of research attention. The identification of human emotional states based on EEG signals is effective to detect potential internal threats caused by insider individuals. Nevertheless, EEG signal-based human emotion evaluation systems have shown several vulnerab…
Electroencephalogram (EEG) is one of the most reliable physiological signal for emotion detection. Being non-stationary in nature, EEGs are better analysed by spectro temporal representations. Standard features like Discrete Wavelet Transformation (DWT) can represent temporal changes in spectral dynamics of an EEG, but is insufficient to extract information other way around, i.e. spectral changes in temporal dynamics. On the other hand, Empirical mode decomposition (EMD) based features can be useful to bridge the above mentioned gap. Towards this direction, we extract two novel features on top…
The neuroscience study has revealed the discrepancy of emotion expression between left and right hemispheres of human brain. Inspired by this study, in this paper, we propose a novel bi-hemispheric discrepancy model (BiHDM) to learn the asymmetric differences between two hemispheres for electroencephalograph (EEG) emotion recognition. Concretely, we first employ four directed recurrent neural networks (RNNs) based on two spatial orientations to traverse electrode signals on two separate brain regions, which enables the model to obtain the deep representations of all the EEG electrodes' signals…
Recognizing the feelings of human beings plays a critical role in our daily communication. Neuroscience has demonstrated that different emotion states present different degrees of activation in different brain regions, EEG frequency bands and temporal stamps. In this paper, we propose a novel structure to explore the informative EEG features for emotion recognition. The proposed module, denoted by PST-Attention, consists of Positional, Spectral and Temporal Attention modules to explore more discriminative EEG features. Specifically, the Positional Attention module is to capture the activate re…
Northwestern Polytechnic University（西北工业大学）；Shaanxi Normal University（陕西师范大学）
One of the most significant challenges of EEG-based emotion recognition is the cross-subject EEG variations, leading to poor performance and generalizability. This paper proposes a novel EEG-based emotion recognition model called the domain adversarial graph attention model (DAGAM). The basic idea is to generate a graph to model multichannel EEG signals using biological topology. Graph theory can topologically describe and analyze relationships and mutual dependency between channels of EEG. Then, unlike other graph convolutional networks, self-attention pooling is applied to benefit salient EE…
Use of the electroencephalogram (EEG) and machine learning approaches to recognize emotions can facilitate affective human computer interactions. However, the type of EEG data constitutes an obstacle for cross-individual EEG feature modelling and classification. To address this issue, we propose a deep-learning framework denoted as a dynamic entropy-based pattern learning (DEPL) to abstract informative indicators pertaining to the neurophysiological features among multiple individuals. DEPL enhanced the capability of representations generated by a deep convolutional neural network by modelling…
Shenzhen University（深圳大学）；Kyoto University（京都大学）；Peng Cheng Laboratory（鹏城实验室）；ATR Neural Information Analysis Laboratories（ATR神经信息分析实验室）
How to effectively and efficiently extract valid and reliable features from high-dimensional electroencephalography (EEG), particularly how to fuse the spatial and temporal dynamic brain information into a better feature representation, is a critical issue in brain data analysis. Most current EEG studies work in a task driven manner and explore the valid EEG features with a supervised model, which would be limited by the given labels to a great extent. In this paper, we propose a practical hybrid unsupervised deep convolutional recurrent generative adversarial network based EEG feature charact…
Xidian University（西安电子科技大学）；Southeast University（东南大学）；Beijing Institute of Mechanical Equipment（北京机械设备研究所）
Studies in the area of neuroscience have revealed the relationship between emotional patterns and brain functional regions, demonstrating that dynamic relationships between different brain regions are an essential factor affecting emotion recognition determined through electroencephalography (EEG). Moreover, in EEG emotion recognition, we can observe that clearer boundaries exist between coarse-grained emotions than those between fine-grained emotions, based on the same EEG data; this indicates the concurrence of large coarse- and small fine-grained emotion variations. Thus, the progressive cl…
An Enhanced Source-Free Unsupervised Domain Adaptation Framework for Cross-Dataset EEG Emotion Recognition via Predictive Coding and Test-Time Training
University of Canberra（堪培拉大学）；Kuwait College of Science and Technology（科威特科技学院）
While Parkinson's disease (PD) is typically characterized by motor disorder, there is evidence of diminished emotion perception in PD patients. This study examines the utility of affective Electroencephalography (EEG) signals to understand emotional differences between PD vs Healthy Controls (HC), and for automated PD detection. Employing traditional machine learning and deep learning methods, we explore (a) dimensional and categorical emotion recognition, and (b) PD vs HC classification from emotional EEG signals. Our results reveal that PD patients comprehend arousal better than valence, and…
Convolutional Neural Networks(CNNs) has achieved remarkable performance breakthrough in a variety of tasks. Recently, CNNs based methods that are fed with hand-extracted EEG features gradually produce a powerful performance on the EEG data based emotion recognition task. In this paper, we propose a novel convolutional layer allowing to adaptively extract effective data-driven spectrogram-like features from raw EEG signals, which we reference as scaling layer. Further, it leverages convolutional kernels scaled from one data-driven pattern to exposed a frequency-like dimension to address the sho…
Graph Neural Networks in EEG-based Emotion Recognition: A Survey
Nanyang Technological University, Singapore（南洋理工大学）；Xi’an Jiaotong University, Xi’an, China（西安交通大学）；Imperial College London, London, UK（伦敦帝国理工学院）；Amazon, Seattle, WA, USA（亚马逊）；Carnegie Mellon University, Pittsburgh, PA, USA（卡内基梅隆大学）；Uber Technologies, Inc., USA（Uber Technologies, Inc.）；School of Computer Science, Central China Normal University, Wuhan, China（中央财经大学计算机学院）；Institute of Automation, Chinese Academy of Sciences, Beijing, China（中国科学院自动化研究所）
EEG signals in emotion recognition absorb special attention owing to their high temporal resolution and their information about what happens in the brain. Different regions of brain work together to process information and meanwhile the activity of brain changes during the time. Therefore, the investigation of the connection between different brain areas and their temporal patterns plays an important role in neuroscience. In this study, we investigate the emotion classification performance using functional connectivity features in different frequency bands and compare them with the classificat…
EEG signals have been reported to be informative and reliable for emotion recognition in recent years. However, the inter-subject variability of emotion-related EEG signals still poses a great challenge for the practical applications of EEG-based emotion recognition. Inspired by recent neuroscience studies on inter-subject correlation, we proposed a Contrastive Learning method for Inter-Subject Alignment (CLISA) to tackle the cross-subject emotion recognition problem. Contrastive learning was employed to minimize the inter-subject differences by maximizing the similarity in EEG signal represen…
The data scarcity problem in emotion recognition from electroencephalography (EEG) leads to difficulty in building an affective model with high accuracy using machine learning algorithms or deep neural networks. Inspired by emerging deep generative models, we propose three methods for augmenting EEG training data to enhance the performance of emotion recognition models. Our proposed methods are based on two deep generative models, variational autoencoder (VAE) and generative adversarial network (GAN), and two data augmentation strategies. For the full usage strategy, all of the generated data …
EEG-based emotion recognition often requires sufficient labeled training samples to build an effective computational model. Labeling EEG data, on the other hand, is often expensive and time-consuming. To tackle this problem and reduce the need for output labels in the context of EEG-based emotion recognition, we propose a semi-supervised pipeline to jointly exploit both unlabeled and labeled data for learning EEG representations. Our semi-supervised framework consists of both unsupervised and supervised components. The unsupervised part maximizes the consistency between original and reconstruc…
Previous electroencephalogram (EEG) emotion recognition relies on single-task learning, which may lead to overfitting and learned emotion features lacking generalization. In this paper, a graph-based multi-task self-supervised learning model (GMSS) for EEG emotion recognition is proposed. GMSS has the ability to learn more general representations by integrating multiple self-supervised tasks, including spatial and frequency jigsaw puzzle tasks, and contrastive learning tasks. By learning from multiple tasks simultaneously, GMSS can find a representation that captures all of the tasks thereby d…
In the recent past, deep learning-based approaches have significantly improved the classification accuracy when compared to classical signal processing and machine learning based frameworks. But most of them were subject-dependent studies which were not able to generalize on the subject-independent tasks due to the inter-subject variability present in EEG data. In this work, a novel deep learning framework capable of doing subject-independent emotion recognition is presented, consisting of two parts. First, an unsupervised Long Short-Term Memory (LSTM) with channel-attention autoencoder is pro…
School of ECE, National Technical University of Athens（雅典国家技术大学电气与计算机工程学院）
Emotion Recognition from EEG signals has long been researched as it can assist numerous medical and rehabilitative applications. However, their complex and noisy structure has proven to be a serious barrier for traditional modeling methods. In this paper, we employ multifractal analysis to examine the behavior of EEG signals in terms of presence of fluctuations and the degree of fragmentation along their major frequency bands, for the task of emotion recognition. In order to extract emotion-related features we utilize two novel algorithms for EEG analysis, based on Multiscale Fractal Dimension…
Lanzhou University（兰州大学）；Beijing Institute of Technology（北京理工大学）；Inner Mongolia University（内蒙古大学）；Shandong Academy of Intelligent Computing Technology（山东智能计算技术研究院）；Chinese Academy of Sciences（中国科学院）
Cross-dataset emotion recognition as an extremely challenging task in the field of EEG-based affective computing is influenced by many factors, which makes the universal models yield unsatisfactory results. Facing the situation that lacks EEG information decoding research, we first analyzed the impact of different EEG information(individual, session, emotion and trial) for emotion recognition by sample space visualization, sample aggregation phenomena quantification, and energy pattern analysis on five public datasets. Based on these phenomena and patterns, we provided the processing methods a…
Zhejiang University（浙江大学）；HwaMei Hospital, University of Chinese Academy of Sciences（中国科学院大学华美医院）；Ningbo Institute of Life and Health Industry, University of Chinese Academy of Sciences（中国科学院大学宁波生命与健康产业研究院）
Electroencephalography (EEG) based emotion recognition has demonstrated tremendous improvement in recent years. Specifically, numerous domain adaptation (DA) algorithms have been exploited in the past five years to enhance the generalization of emotion recognition models across subjects. The DA methods assume that calibration data (although unlabeled) exists in the target domain (new user). However, this assumption conflicts with the application scenario that the model should be deployed without the time-consuming calibration experiments. We argue that domain generalization (DG) is more reason…
A controversial issue in artificial intelligence is human emotion recognition. This paper presents a fuzzy parallel cascades (FPC) model for predicting the continuous subjective appraisal of the emotional content of music by time-varying spectral content of EEG signals. The EEG, along with an emotional appraisal of 15 subjects, was recorded during listening to seven musical excerpts. The emotional appraisement was recorded along the valence and arousal emotional axes as a continuous signal. The FPC model was composed of parallel cascades with each cascade containing a fuzzy logic-based system.…
EEGMatch: Learning with Incomplete Labels for Semi-Supervised EEG-based Cross-Subject Emotion Recognition
Shenzhen University（深圳大学）；Guangdong Provincial Key Laboratory of Biomedical Measurements and Ultrasound Imaging（广东省生物医学测量与超声成像重点实验室）；City University of Hong Kong（香港城市大学）；Hong Kong Center for Cerebro-Cardiovascular Health Engineering（香港心脑血管健康工程中心）
Ping An Technology (Shenzhen) Co., Ltd.（平安科技（深圳）有限公司）
Using deep learning methods to classify EEG signals can accurately identify people's emotions. However, existing studies have rarely considered the application of the information in another domain's representations to feature selection in the time-frequency domain. We propose a classification network of EEG signals based on the cross-domain feature fusion method, which makes the network more focused on the features most related to brain activities and thinking changes by using the multi-domain attention mechanism. In addition, we propose a two-step fusion method and apply these methods to the …
A Supervised Information Enhanced Multi-Granularity Contrastive Learning Framework for EEG Based Emotion Recognition
Key Laboratory of Computing Power Network and Information Security, Ministry of Education, Shandong Computer Science Center (National Supercomputer Center in Jinan), Qilu University of Technology (Shandong Academy of Sciences)（齐鲁工业大学（山东省科学院）山东省计算中心（国家超级计算济南中心）算力网络与信息安全教育部重点实验室）；Shandong Provincial Key Laboratory of Computer Networks, Shandong Fundamental Research Center for Computer Science（山东省计算机网络重点实验室、山东省计算机科学基础研究中心）；Institute of Engineering Medicine, Beijing Institute of Technology（北京理工大学医学工程学院）
Adaptive Progressive Attention Graph Neural Network for EEG Emotion Recognition
Key Laboratory of Intelligent Perception and Image Understanding of Ministry of Education（教育部智能感知与图像理解重点实验室）；School of Artificial Intelligence（人工智能学院）；Xidian University（西安电子科技大学）
Qilu University of Technology (Shandong Academy of Sciences)（齐鲁工业大学（山东省科学院））；Zhengzhou University of Light Industry（郑州轻工业大学）；Nanjing University（南京大学）；Aalto University（阿尔托大学）；Beijing Institute of Technology（北京理工大学）
Emotion recognition technology through analyzing the EEG signal is currently an essential concept in Artificial Intelligence and holds great potential in emotional health care, human-computer interaction, multimedia content recommendation, etc. Though there have been several works devoted to reviewing EEG-based emotion recognition, the content of these reviews needs to be updated. In addition, those works are either fragmented in content or only focus on specific techniques adopted in this area but neglect the holistic perspective of the entire technical routes. Hence, in this paper, we review…
A Comprehensive Survey on EEG-Based Emotion Recognition: A Graph-Based Perspective
College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算与数据科学学院）；School of Computer Science, Central China Normal University（华中师范大学计算机学院）；University of Science and Technology of China（中国科学技术大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
University of Trento（特伦托大学）；Temple University（天普大学）
Current models on Explainable Artificial Intelligence (XAI) have shown an evident and quantified lack of reliability for measuring feature-relevance when statistically entangled features are proposed for training deep classifiers. There has been an increase in the application of Deep Learning in clinical trials to predict early diagnosis of neuro-developmental disorders, such as Autism Spectrum Disorder (ASD). However, the inclusion of more reliable saliency-maps to obtain more trustworthy and interpretable metrics using neural activity features is still insufficiently mature for practical app…
Nanyang Technological University（南洋理工大学）；Duke University - National University of Singapore Medical School（杜克大学-新加坡国立大学医学院）；Singapore General Hospital（新加坡中央医院）
In this paper, we propose a deep learning framework, TSception, for emotion detection from electroencephalogram (EEG). TSception consists of temporal and spatial convolutional layers, which learn discriminative representations in the time and channel domains simultaneously. The temporal learner consists of multi-scale 1D convolutional kernels whose lengths are related to the sampling rate of the EEG signal, which learns multiple temporal and frequency representations. The spatial learner takes advantage of the asymmetry property of emotion responses at the frontal brain area to learn the discr…
University of Shanghai for Science and Technology（上海理工大学）
Technique of emotion recognition enables computers to classify human affective states into discrete categories. However, the emotion may fluctuate instead of maintaining a stable state even within a short time interval. There is also a difficulty to take the full use of the EEG spatial distribution due to its 3-D topology structure. To tackle the above issues, we proposed a locally temporal-spatial pattern learning graph attention network (LTS-GAT) in the present study. In the LTS-GAT, a divide-and-conquer scheme was used to examine local information on temporal and spatial dimensions of EEG p…
School of Biomedical Engineering, Medical School, Shenzhen University（深圳大学医学院生物医学工程学院）；Guangdong Provincial Key Laboratory of Biomedical Measurements and Ultrasound Imaging（广东省生物医学测量与超声成像重点实验室）；Institute of Computing and Intelligence, Harbin Institute of Technology（哈尔滨工业大学计算与智能研究院）；Peng Cheng Laboratory（鹏城实验室）；Shenzhen Mental Health Center, Shenzhen Kangning Hospital（深圳市精神卫生中心（深圳市康宁医院））
MSGM: A Multi-Scale Spatiotemporal Graph Mamba for EEG Emotion Recognition
School of Electronics and Communication Engineering, Sun Yat-sen University（中山大学电子与通信工程学院）；College of Communication Engineering, Jilin University（吉林大学通信工程学院）；School of Pharmaceutical Sciences (Shenzhen), Sun Yat-sen University（中山大学药学院（深圳））；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
A novel Fourier Adjacency Transformer for advanced EEG emotion recognition
Faculty of Information Engineering and Automation, Kunming University of Science and Technology（昆明理工大学信息工程与自动化学院）；School of Artificial Intelligence and Advanced Computing, Xi’an Jiaotong-Liverpool University（西交利物浦大学人工智能与先进计算学院）
