One of the most significant challenges of EEG-based emotion recognition is the cross-subject EEG variations, leading to poor performance and generalizability. This paper proposes a novel EEG-based emotion recognition model called the domain adversarial graph attention model (DAGAM). The basic idea is to generate a graph to model multichannel EEG signals using biological topology. Graph theory can topologically describe and analyze relationships and mutual dependency between channels of EEG. Then, unlike other graph convolutional networks, self-attention pooling is applied to benefit salient EEG feature extraction from the graph, which effectively improves the performance. Finally, after graph pooling, the domain adversarial based on the graph is employed to identify and handle EEG variation across subjects, efficiently reaching good generalizability. We conduct extensive evaluations on two benchmark datasets (SEED and SEED IV) and obtain state-of-the-art results in subject-independent emotion recognition. Our model boosts the SEED accuracy to 92.59% (4.69% improvement) with the lowest standard deviation of 3.21% (2.92% decrements) and SEED IV accuracy to 80.74% (6.90% improvement) with the lowest standard deviation of 4.14% (3.88% decrements) respectively.
University of Electronic Science and Technology of China（电子科技大学）
Emotion recognition based on EEG (electroencephalography) has been widely used in human-computer interaction, distance education and health care. However, the conventional methods ignore the adjacent and symmetrical characteristics of EEG signals, which also contain salient information related to emotion. In this paper, a spatial folding ensemble network (SFE-Net) is presented for EEG feature extraction and emotion recognition. Firstly, for the undetected area between EEG electrodes, an improved Bicubic-EEG interpolation algorithm is developed for EEG channels information completion, which all…
Khalifa University（哈利法大学）；Università degli Studi di Milano（米兰大学）
The major aim of this paper is to explain the data poisoning attacks using label-flipping during the training stage of the electroencephalogram (EEG) signal-based human emotion evaluation systems deploying Machine Learning models from the attackers' perspective. Human emotion evaluation using EEG signals has consistently attracted a lot of research attention. The identification of human emotional states based on EEG signals is effective to detect potential internal threats caused by insider individuals. Nevertheless, EEG signal-based human emotion evaluation systems have shown several vulnerab…
Electroencephalogram (EEG) is one of the most reliable physiological signal for emotion detection. Being non-stationary in nature, EEGs are better analysed by spectro temporal representations. Standard features like Discrete Wavelet Transformation (DWT) can represent temporal changes in spectral dynamics of an EEG, but is insufficient to extract information other way around, i.e. spectral changes in temporal dynamics. On the other hand, Empirical mode decomposition (EMD) based features can be useful to bridge the above mentioned gap. Towards this direction, we extract two novel features on top…
Use of the electroencephalogram (EEG) and machine learning approaches to recognize emotions can facilitate affective human computer interactions. However, the type of EEG data constitutes an obstacle for cross-individual EEG feature modelling and classification. To address this issue, we propose a deep-learning framework denoted as a dynamic entropy-based pattern learning (DEPL) to abstract informative indicators pertaining to the neurophysiological features among multiple individuals. DEPL enhanced the capability of representations generated by a deep convolutional neural network by modelling…
Convolutional Neural Networks(CNNs) has achieved remarkable performance breakthrough in a variety of tasks. Recently, CNNs based methods that are fed with hand-extracted EEG features gradually produce a powerful performance on the EEG data based emotion recognition task. In this paper, we propose a novel convolutional layer allowing to adaptively extract effective data-driven spectrogram-like features from raw EEG signals, which we reference as scaling layer. Further, it leverages convolutional kernels scaled from one data-driven pattern to exposed a frequency-like dimension to address the sho…
Graph Neural Networks in EEG-based Emotion Recognition: A Survey
Nanyang Technological University, Singapore（南洋理工大学）；Xi’an Jiaotong University, Xi’an, China（西安交通大学）；Imperial College London, London, UK（伦敦帝国理工学院）；Amazon, Seattle, WA, USA（亚马逊）；Carnegie Mellon University, Pittsburgh, PA, USA（卡内基梅隆大学）；Uber Technologies, Inc., USA（Uber Technologies, Inc.）；School of Computer Science, Central China Normal University, Wuhan, China（中央财经大学计算机学院）；Institute of Automation, Chinese Academy of Sciences, Beijing, China（中国科学院自动化研究所）
The data scarcity problem in emotion recognition from electroencephalography (EEG) leads to difficulty in building an affective model with high accuracy using machine learning algorithms or deep neural networks. Inspired by emerging deep generative models, we propose three methods for augmenting EEG training data to enhance the performance of emotion recognition models. Our proposed methods are based on two deep generative models, variational autoencoder (VAE) and generative adversarial network (GAN), and two data augmentation strategies. For the full usage strategy, all of the generated data …
EEG-based emotion recognition often requires sufficient labeled training samples to build an effective computational model. Labeling EEG data, on the other hand, is often expensive and time-consuming. To tackle this problem and reduce the need for output labels in the context of EEG-based emotion recognition, we propose a semi-supervised pipeline to jointly exploit both unlabeled and labeled data for learning EEG representations. Our semi-supervised framework consists of both unsupervised and supervised components. The unsupervised part maximizes the consistency between original and reconstruc…
Shenzhen University（深圳大学）；Kyoto University（京都大学）；Peng Cheng Laboratory（鹏城实验室）；ATR Neural Information Analysis Laboratories（ATR神经信息分析实验室）
How to effectively and efficiently extract valid and reliable features from high-dimensional electroencephalography (EEG), particularly how to fuse the spatial and temporal dynamic brain information into a better feature representation, is a critical issue in brain data analysis. Most current EEG studies work in a task driven manner and explore the valid EEG features with a supervised model, which would be limited by the given labels to a great extent. In this paper, we propose a practical hybrid unsupervised deep convolutional recurrent generative adversarial network based EEG feature charact…
Zhejiang University（浙江大学）；HwaMei Hospital, University of Chinese Academy of Sciences（中国科学院大学华美医院）；Ningbo Institute of Life and Health Industry, University of Chinese Academy of Sciences（中国科学院大学宁波生命与健康产业研究院）
Electroencephalography (EEG) based emotion recognition has demonstrated tremendous improvement in recent years. Specifically, numerous domain adaptation (DA) algorithms have been exploited in the past five years to enhance the generalization of emotion recognition models across subjects. The DA methods assume that calibration data (although unlabeled) exists in the target domain (new user). However, this assumption conflicts with the application scenario that the model should be deployed without the time-consuming calibration experiments. We argue that domain generalization (DG) is more reason…
University of Canberra（堪培拉大学）；Kuwait College of Science and Technology（科威特科技学院）
While Parkinson's disease (PD) is typically characterized by motor disorder, there is evidence of diminished emotion perception in PD patients. This study examines the utility of affective Electroencephalography (EEG) signals to understand emotional differences between PD vs Healthy Controls (HC), and for automated PD detection. Employing traditional machine learning and deep learning methods, we explore (a) dimensional and categorical emotion recognition, and (b) PD vs HC classification from emotional EEG signals. Our results reveal that PD patients comprehend arousal better than valence, and…
Adaptive Progressive Attention Graph Neural Network for EEG Emotion Recognition
Key Laboratory of Intelligent Perception and Image Understanding of Ministry of Education（教育部智能感知与图像理解重点实验室）；School of Artificial Intelligence（人工智能学院）；Xidian University（西安电子科技大学）
Previous electroencephalogram (EEG) emotion recognition relies on single-task learning, which may lead to overfitting and learned emotion features lacking generalization. In this paper, a graph-based multi-task self-supervised learning model (GMSS) for EEG emotion recognition is proposed. GMSS has the ability to learn more general representations by integrating multiple self-supervised tasks, including spatial and frequency jigsaw puzzle tasks, and contrastive learning tasks. By learning from multiple tasks simultaneously, GMSS can find a representation that captures all of the tasks thereby d…
A Comprehensive Survey on EEG-Based Emotion Recognition: A Graph-Based Perspective
College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算与数据科学学院）；School of Computer Science, Central China Normal University（华中师范大学计算机学院）；University of Science and Technology of China（中国科学技术大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
University of Trento（特伦托大学）；Temple University（天普大学）
Current models on Explainable Artificial Intelligence (XAI) have shown an evident and quantified lack of reliability for measuring feature-relevance when statistically entangled features are proposed for training deep classifiers. There has been an increase in the application of Deep Learning in clinical trials to predict early diagnosis of neuro-developmental disorders, such as Autism Spectrum Disorder (ASD). However, the inclusion of more reliable saliency-maps to obtain more trustworthy and interpretable metrics using neural activity features is still insufficiently mature for practical app…
Xidian University（西安电子科技大学）；Southeast University（东南大学）；Beijing Institute of Mechanical Equipment（北京机械设备研究所）
Studies in the area of neuroscience have revealed the relationship between emotional patterns and brain functional regions, demonstrating that dynamic relationships between different brain regions are an essential factor affecting emotion recognition determined through electroencephalography (EEG). Moreover, in EEG emotion recognition, we can observe that clearer boundaries exist between coarse-grained emotions than those between fine-grained emotions, based on the same EEG data; this indicates the concurrence of large coarse- and small fine-grained emotion variations. Thus, the progressive cl…
School of ECE, National Technical University of Athens（雅典国家技术大学电气与计算机工程学院）
Emotion Recognition from EEG signals has long been researched as it can assist numerous medical and rehabilitative applications. However, their complex and noisy structure has proven to be a serious barrier for traditional modeling methods. In this paper, we employ multifractal analysis to examine the behavior of EEG signals in terms of presence of fluctuations and the degree of fragmentation along their major frequency bands, for the task of emotion recognition. In order to extract emotion-related features we utilize two novel algorithms for EEG analysis, based on Multiscale Fractal Dimension…
EEG signals have been reported to be informative and reliable for emotion recognition in recent years. However, the inter-subject variability of emotion-related EEG signals still poses a great challenge for the practical applications of EEG-based emotion recognition. Inspired by recent neuroscience studies on inter-subject correlation, we proposed a Contrastive Learning method for Inter-Subject Alignment (CLISA) to tackle the cross-subject emotion recognition problem. Contrastive learning was employed to minimize the inter-subject differences by maximizing the similarity in EEG signal represen…
Lanzhou University（兰州大学）；Beijing Institute of Technology（北京理工大学）；Inner Mongolia University（内蒙古大学）；Shandong Academy of Intelligent Computing Technology（山东智能计算技术研究院）；Chinese Academy of Sciences（中国科学院）
Cross-dataset emotion recognition as an extremely challenging task in the field of EEG-based affective computing is influenced by many factors, which makes the universal models yield unsatisfactory results. Facing the situation that lacks EEG information decoding research, we first analyzed the impact of different EEG information(individual, session, emotion and trial) for emotion recognition by sample space visualization, sample aggregation phenomena quantification, and energy pattern analysis on five public datasets. Based on these phenomena and patterns, we provided the processing methods a…
MSGM: A Multi-Scale Spatiotemporal Graph Mamba for EEG Emotion Recognition
School of Electronics and Communication Engineering, Sun Yat-sen University（中山大学电子与通信工程学院）；College of Communication Engineering, Jilin University（吉林大学通信工程学院）；School of Pharmaceutical Sciences (Shenzhen), Sun Yat-sen University（中山大学药学院（深圳））；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
EEGMatch: Learning with Incomplete Labels for Semi-Supervised EEG-based Cross-Subject Emotion Recognition
Shenzhen University（深圳大学）；Guangdong Provincial Key Laboratory of Biomedical Measurements and Ultrasound Imaging（广东省生物医学测量与超声成像重点实验室）；City University of Hong Kong（香港城市大学）；Hong Kong Center for Cerebro-Cardiovascular Health Engineering（香港心脑血管健康工程中心）
Ping An Technology (Shenzhen) Co., Ltd.（平安科技（深圳）有限公司）
Using deep learning methods to classify EEG signals can accurately identify people's emotions. However, existing studies have rarely considered the application of the information in another domain's representations to feature selection in the time-frequency domain. We propose a classification network of EEG signals based on the cross-domain feature fusion method, which makes the network more focused on the features most related to brain activities and thinking changes by using the multi-domain attention mechanism. In addition, we propose a two-step fusion method and apply these methods to the …
A Supervised Information Enhanced Multi-Granularity Contrastive Learning Framework for EEG Based Emotion Recognition
Key Laboratory of Computing Power Network and Information Security, Ministry of Education, Shandong Computer Science Center (National Supercomputer Center in Jinan), Qilu University of Technology (Shandong Academy of Sciences)（齐鲁工业大学（山东省科学院）山东省计算中心（国家超级计算济南中心）算力网络与信息安全教育部重点实验室）；Shandong Provincial Key Laboratory of Computer Networks, Shandong Fundamental Research Center for Computer Science（山东省计算机网络重点实验室、山东省计算机科学基础研究中心）；Institute of Engineering Medicine, Beijing Institute of Technology（北京理工大学医学工程学院）
School of Integrated Technology, Yonsei University（延世大学综合技术学院）
Electroencephalography (EEG) is a useful way to implicitly monitor the users perceptual state during multimedia consumption. One of the primary challenges for the practical use of EEG-based monitoring is to achieve a satisfactory level of accuracy in EEG classification. Connectivity between different brain regions is an important property for the classification of EEG. However, how to define the connectivity structure for a given task is still an open problem, because there is no ground truth about how the connectivity structure should be in order to maximize the classification performance. In…
Qilu University of Technology (Shandong Academy of Sciences)（齐鲁工业大学（山东省科学院））；Zhengzhou University of Light Industry（郑州轻工业大学）；Nanjing University（南京大学）；Aalto University（阿尔托大学）；Beijing Institute of Technology（北京理工大学）
Emotion recognition technology through analyzing the EEG signal is currently an essential concept in Artificial Intelligence and holds great potential in emotional health care, human-computer interaction, multimedia content recommendation, etc. Though there have been several works devoted to reviewing EEG-based emotion recognition, the content of these reviews needs to be updated. In addition, those works are either fragmented in content or only focus on specific techniques adopted in this area but neglect the holistic perspective of the entire technical routes. Hence, in this paper, we review…
Both the temporal dynamics and spatial correlations of Electroencephalogram (EEG), which contain discriminative emotion information, are essential for the emotion recognition. However, some redundant information within the EEG signals would degrade the performance. Specifically,the subjects reach prospective intense emotions for only a fraction of the stimulus duration. Besides, it is a challenge to extract discriminative features from the complex spatial correlations among a number of electrodes. To deal with the problems, we propose a transformer-based model to robustly capture temporal dyna…
Graph Learning for Cross-Subject, Cross-Population EEG Emotion Decoding and Model-Derived Spatial-Spectral Neural Signatures
Chongqing University of Technology（重庆理工大学）；The Hong Kong Polytechnic University（香港理工大学）；Chongqing Polytechnic University of Electronic Technology（重庆电子工程职业技术大学）；Affiliated Lianyungang Hospital of Xuzhou Medical University（徐州医科大学附属连云港医院）
Accurate forecasting of an electroencephalogram (EEG) time series is crucial for the correct diagnosis of neurological disorders such as seizures and epilepsy. Since the EEG time series is chaotic, most traditional machine learning algorithms have failed to forecast its next steps accurately. Thus, we suggest a model, which has formed by taking inspiration from the neural structures that underlie feelings (emotional states), to forecast EEG time series. The model, which is referred to as emotion-inspired deep structure (EiDS), can be used to predict both short- and long-term of EEG time series…
Nanyang Technological University（南洋理工大学）；Duke University - National University of Singapore Medical School（杜克大学-新加坡国立大学医学院）；Singapore General Hospital（新加坡中央医院）
In this paper, we propose a deep learning framework, TSception, for emotion detection from electroencephalogram (EEG). TSception consists of temporal and spatial convolutional layers, which learn discriminative representations in the time and channel domains simultaneously. The temporal learner consists of multi-scale 1D convolutional kernels whose lengths are related to the sampling rate of the EEG signal, which learns multiple temporal and frequency representations. The spatial learner takes advantage of the asymmetry property of emotion responses at the frontal brain area to learn the discr…
School of Biomedical Engineering, Medical School, Shenzhen University（深圳大学医学院生物医学工程学院）；Guangdong Provincial Key Laboratory of Biomedical Measurements and Ultrasound Imaging（广东省生物医学测量与超声成像重点实验室）；Institute of Computing and Intelligence, Harbin Institute of Technology（哈尔滨工业大学计算与智能研究院）；Peng Cheng Laboratory（鹏城实验室）；Shenzhen Mental Health Center, Shenzhen Kangning Hospital（深圳市精神卫生中心（深圳市康宁医院））
Deep Learning has impacted various fields especially in bio-medical applications. Deep learning algorithms work well with both structured and unstructured data. Especially, convolutional neural network work well with signal-based data like EEG data. These types of data may or may not follow a pattern in their data. Algorithms like CNN help feature engineering and simplistic interpretation of the data. These algorithms are also better in comparison to other algorithms when generalised to a data belonging to larger data set.
The neuroscience study has revealed the discrepancy of emotion expression between left and right hemispheres of human brain. Inspired by this study, in this paper, we propose a novel bi-hemispheric discrepancy model (BiHDM) to learn the asymmetric differences between two hemispheres for electroencephalograph (EEG) emotion recognition. Concretely, we first employ four directed recurrent neural networks (RNNs) based on two spatial orientations to traverse electrode signals on two separate brain regions, which enables the model to obtain the deep representations of all the EEG electrodes' signals…
Recognizing the feelings of human beings plays a critical role in our daily communication. Neuroscience has demonstrated that different emotion states present different degrees of activation in different brain regions, EEG frequency bands and temporal stamps. In this paper, we propose a novel structure to explore the informative EEG features for emotion recognition. The proposed module, denoted by PST-Attention, consists of Positional, Spectral and Temporal Attention modules to explore more discriminative EEG features. Specifically, the Positional Attention module is to capture the activate re…
University of Shanghai for Science and Technology（上海理工大学）
Technique of emotion recognition enables computers to classify human affective states into discrete categories. However, the emotion may fluctuate instead of maintaining a stable state even within a short time interval. There is also a difficulty to take the full use of the EEG spatial distribution due to its 3-D topology structure. To tackle the above issues, we proposed a locally temporal-spatial pattern learning graph attention network (LTS-GAT) in the present study. In the LTS-GAT, a divide-and-conquer scheme was used to examine local information on temporal and spatial dimensions of EEG p…
A novel Fourier Adjacency Transformer for advanced EEG emotion recognition
Faculty of Information Engineering and Automation, Kunming University of Science and Technology（昆明理工大学信息工程与自动化学院）；School of Artificial Intelligence and Advanced Computing, Xi’an Jiaotong-Liverpool University（西交利物浦大学人工智能与先进计算学院）
This paper proposes a novel two-stage framework for emotion recognition using EEG data that outperforms state-of-the-art models while keeping the model size small and computationally efficient. The framework consists of two stages; the first stage involves constructing efficient models named EEGNet, which is inspired by the state-of-the-art efficient architecture and employs inverted-residual blocks that contain depthwise separable convolutional layers. The EEGNet models on both valence and arousal labels achieve the average classification accuracy of 90%, 96.6%, and 99.5% with only 6.4k, 14k,…
