Among the different modalities to assess emotion, electroencephalogram (EEG), representing the electrical brain activity, achieved motivating results over the last decade. Emotion estimation from EEG could help in the diagnosis or rehabilitation of certain diseases. In this paper, we propose a dual model considering two different representations of EEG feature maps: 1) a sequential based representation of EEG band power, 2) an image-based representation of the feature vectors. We also propose an innovative method to combine the information based on a saliency analysis of the image-based model to promote joint learning of both model parts. The model has been evaluated on four publicly available datasets: SEED-IV, SEED, DEAP and MPED. The achieved results outperform results from state-of-the-art approaches for three of the proposed datasets with a lower standard deviation that reflects higher stability. For sake of reproducibility, the codes and models proposed in this paper are available at https://github.com/VDelv/Emotion-EEG.
EEG Emotion Copilot: Optimizing Lightweight LLMs for Emotional EEG Interpretation with Assisted Medical Record Generation
Laboratory of Digital Image and Intelligent Computation, Shanghai Maritime University, Shanghai 201306, China（数字图像与智能计算实验室，上海 Maritime 大学）；Department of Neurology, Affiliated Lianyungang Hospital of Xuzhou Medical University, Lianyungang 222002, China（神经内科，徐州医学院附属连云港医院）；Department of Chinese and Bilingual Studies, The Hong Kong Polytechnic University, Hong Kong, SAR, China（中文学与双语研究部，香港理工大学）
DuA: Dual Attentive Transformer in Long-Term Continuous EEG Emotion Analysis
School of Biomedical Engineering, Medical School, Shenzhen University（深圳大学医学院生物医学工程学院）；Guangdong Provincial Key Laboratory of Biomedical Measurements and Ultrasound Imaging（广东省生物医学测量与超声成像重点实验室）
Lanzhou University（兰州大学）；Beijing Institute of Technology（北京理工大学）；Inner Mongolia University（内蒙古大学）；Shandong Academy of Intelligent Computing Technology（山东智能计算技术研究院）；Chinese Academy of Sciences（中国科学院）
Cross-dataset emotion recognition as an extremely challenging task in the field of EEG-based affective computing is influenced by many factors, which makes the universal models yield unsatisfactory results. Facing the situation that lacks EEG information decoding research, we first analyzed the impact of different EEG information(individual, session, emotion and trial) for emotion recognition by sample space visualization, sample aggregation phenomena quantification, and energy pattern analysis on five public datasets. Based on these phenomena and patterns, we provided the processing methods a…
Halmstad University（哈尔姆斯塔德大学）；Ulster University Belfast（贝尔法斯特阿尔斯特大学）
One of the challenges in virtual environments is the difficulty users have in interacting with these increasingly complex systems. Ultimately, endowing machines with the ability to perceive users emotions will enable a more intuitive and reliable interaction. Consequently, using the electroencephalogram as a bio-signal sensor, the affective state of a user can be modelled and subsequently utilised in order to achieve a system that can recognise and react to the user's emotions. This paper investigates features extracted from electroencephalogram signals for the purpose of affective state model…
Multi-dataset Joint Pre-training of Emotional EEG Enables Generalizable Affective Computing
Department of Biomedical Engineering, Southern University of Science and Technology（生物医学工程系，南方科技大学）；School of Data Science, University of California, San Diego（数据科学学院，加州大学圣地亚哥分校）；School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen（科学与工程学院，香港中文大学（深圳））
LibEMER: A novel benchmark and algorithms library for EEG-based Multimodal Emotion Recognition
XJTU-POLIMI Joint School, Xi'an Jiaotong University School of Computer Science；Technology, Xi'an Jiaotong University MIGU Video Co., Ltd., Shanghai, China
Sentiment analysis is crucial for extracting social signals from social media content. Due to the prevalence of images in social media, image sentiment analysis is receiving increasing attention in recent years. However, most existing systems are black-boxes that do not provide insight on how image content invokes sentiment and emotion in the viewers. Psychological studies have confirmed that salient objects in an image often invoke emotions. In this work, we investigate more fine-grained and more comprehensive interaction between visual saliency and visual sentiment. In particular, we partiti…
EEG-Driven Image Reconstruction with Saliency-Guided Diffusion Models
Ivannikov Institute for System Programming of the Russian Academy of Sciences（俄罗斯科学院伊万诺夫系统编程研究所）；Research Center for Trusted Artificial Intelligence（可信人工智能研究中心）；AI Talent Hub, ITMO University（ITMO大学人工智能人才中心）；Moscow, Russia（俄罗斯莫斯科）
State University of New York at Binghamton（纽约州立大学宾汉姆顿分校）
Emotion is an experience associated with a particular pattern of physiological activity along with different physiological, behavioral and cognitive changes. One behavioral change is facial expression, which has been studied extensively over the past few decades. Facial behavior varies with a person's emotion according to differences in terms of culture, personality, age, context, and environment. In recent years, physiological activities have been used to study emotional responses. A typical signal is the electroencephalogram (EEG), which measures brain activity. Most of existing EEG-based em…
Xidian University（西安电子科技大学）；Southeast University（东南大学）；Beijing Institute of Mechanical Equipment（北京机械设备研究所）
Studies in the area of neuroscience have revealed the relationship between emotional patterns and brain functional regions, demonstrating that dynamic relationships between different brain regions are an essential factor affecting emotion recognition determined through electroencephalography (EEG). Moreover, in EEG emotion recognition, we can observe that clearer boundaries exist between coarse-grained emotions than those between fine-grained emotions, based on the same EEG data; this indicates the concurrence of large coarse- and small fine-grained emotion variations. Thus, the progressive cl…
Joint NTU-UBC Research Centre of Excellence in Active Living for the Elderly (LILY)（南洋理工大学-不列颠哥伦比亚大学活跃老年生活卓越联合研究中心（LILY））；Nanyang Technological University（南洋理工大学）；Alibaba-NTU Singapore Joint Research Institute（阿里巴巴-南洋理工大学新加坡联合研究院）；School of Computer Science and Engineering, Nanyang Technological University（南洋理工大学计算机科学与工程学院）
Electroencephalography (EEG) measures the neuronal activities in different brain regions via electrodes. Many existing studies on EEG-based emotion recognition do not fully exploit the topology of EEG channels. In this paper, we propose a regularized graph neural network (RGNN) for EEG-based emotion recognition. RGNN considers the biological topology among different brain regions to capture both local and global relations among different EEG channels. Specifically, we model the inter-channel relations in EEG signals via an adjacency matrix in a graph neural network where the connection and spa…
In real-world applications such as emotion recognition from recorded brain activity, data are captured from electrodes over time. These signals constitute a multidimensional time series. In this paper, Echo State Network (ESN), a recurrent neural network with a great success in time series prediction and classification, is optimized with different neural plasticity rules for classification of emotions based on electroencephalogram (EEG) time series. Actually, the neural plasticity rules are a kind of unsupervised learning adapted for the reservoir, i.e. the hidden layer of ESN. More specifical…
The existed methods for electroencephalograph (EEG) emotion recognition always train the models based on all the EEG samples indistinguishably. However, some of the source (training) samples may lead to a negative influence because they are significant dissimilar with the target (test) samples. So it is necessary to give more attention to the EEG samples with strong transferability rather than forcefully training a classification model by all the samples. Furthermore, for an EEG sample, from the aspect of neuroscience, not all the brain regions of an EEG sample contains emotional information t…
Audio Acoustics Lab - Faculty of Electronics, Telecommunications and Informatics, Gdańsk University of Technology (GUT), Poland（电子电信与信息学院，格但斯克技术大学（GUT））；School of Information Technology, Halmstad University, Sweden（信息科技学院，哈马碧大学）
The subject of this work is to check how different types of music affect human emotions. While listening to music, a subjective survey and brain activity measurements were carried out using an EEG helmet. The aim is to demonstrate the impact of different music genres on emotions. The research involved a diverse group of participants of different gender and musical preferences. This had the effect of capturing a wide range of emotional responses to music. After the experiment, a relationship analysis of the respondents' questionnaires with EEG signals was performed. The analysis revealed connec…
Cross-domain EEG-based Emotion Recognition with Contrastive Learning
School of Software Engineering, Xi'an Jiaotong University, China（西安交通大学软件工程学院）；School of Computer Science and Technology, Xi'an Jiaotong University, China（西安交通大学计算机科学与技术学院）
Australian National University（澳大利亚国立大学）；Sun Yat-sen University（中山大学）；DATA61, CSIRO（联邦科学与工业研究组织数据61所）
Human eyes concentrate different facial regions during distinct cognitive activities. We study utilising facial visual saliency maps to classify different facial expressions into different emotions. Our results show that our novel method of merely using facial saliency maps can achieve a descent accuracy of 65\%, much higher than the chance level of $1/7$. Furthermore, our approach is of semi-supervision, i.e., our facial saliency maps are generated from a general saliency prediction algorithm that is not explicitly designed for face images. We also discovered that the classification accuracie…
PhysioSync: Temporal and Cross-Modal Contrastive Learning Inspired by Physiological Synchronization for EEG-Based Emotion Recognition
School of Instrument Science and Opto-electronics Engineering, Hefei University of Technology（仪器科学与光电工程学院，合肥工业大学）；School of Computer Science and Information Engineering, Hefei University of Technology（计算机科学与信息工程学院，合肥工业大学）
Recognizing the feelings of human beings plays a critical role in our daily communication. Neuroscience has demonstrated that different emotion states present different degrees of activation in different brain regions, EEG frequency bands and temporal stamps. In this paper, we propose a novel structure to explore the informative EEG features for emotion recognition. The proposed module, denoted by PST-Attention, consists of Positional, Spectral and Temporal Attention modules to explore more discriminative EEG features. Specifically, the Positional Attention module is to capture the activate re…
University of Shanghai for Science and Technology（上海理工大学）
Technique of emotion recognition enables computers to classify human affective states into discrete categories. However, the emotion may fluctuate instead of maintaining a stable state even within a short time interval. There is also a difficulty to take the full use of the EEG spatial distribution due to its 3-D topology structure. To tackle the above issues, we proposed a locally temporal-spatial pattern learning graph attention network (LTS-GAT) in the present study. In the LTS-GAT, a divide-and-conquer scheme was used to examine local information on temporal and spatial dimensions of EEG p…
Partial Label Learning for Emotion Recognition from EEG
National Institute of Standards and Technology（美国国家标准与技术研究院）；Rice University（莱斯大学）；Colorado State University（科罗拉多州立大学）；University of Colorado（科罗拉多大学）；National Research Institute for Metals（国立金属材料研究所）
See the Emotion: A Facial Emoji Proxy Modeling for EEG Emotion Recognition
Hefei University of Technology（合肥工业大学）；PLA Information Engineering University（中国人民解放军信息工程大学）；University of Science and Technology of China（中国科学技术大学）；MBZUAI
Lab of Digital Image and Intelligent Computation（数字图像与智能计算实验室）；Shanghai Maritime University（上海 Maritime 大学）；Department of Chinese and Bilingual Studies（中文与双语研究部）；The Hong Kong Polytechnic University（香港理工大学）；Department of Neurology（神经病学部）；Affiliated Lianyungang Hospital of Xuzhou Medical University（徐州医学院附属连云港医院）；The Institute of Computing and Intelligence（计算与智能研究所）；Harbin Institute of Technology Shenzhen（哈尔滨工业大学深圳研究院）
FreqDGT: Frequency-Adaptive Dynamic Graph Networks with Transformer for Cross-subject EEG Emotion Recognition
Department of Chinese and Bilingual Studies, The Hong Kong Polytechnic University（中文与双语研究系，香港理工大学）；School of Information Engineering, Shanghai Maritime University（信息工程学院，上海海洋大学）
Evaluation in EEG Emotion Recognition: State-of-the-Art Review and Unified Framework
Muskhelishvili Institute of Computational Mathematics, Georgian Technical University（穆赫列什维利计算数学研究所，格鲁吉亚技术大学）；German Research Center for Artificial Intelligence（德国人工智能研究中心）；The Biorobotics Institute, Scuola Superiore Sant’Anna（生物机器人研究所，圣安娜高等学院）；INRIA, Universite Côte d’Azur（INRIA，法国蔚蓝海岸大学）
Commuting Distance Regularization for Timescale-Dependent Label Inconsistency in EEG Emotion Recognition
School of Mathematics (Zhuhai) Sun Yat-sen University（中山大学数学学院（珠海））；ODEN Institute for Computational Engineering & Sciences University of Texas at Austin（德克萨斯大学奥斯汀分校ODEN计算工程与科学研究所）；Shenzhen Institute of Advanced Technology Chinese Academy of Sciences（中国科学院深圳先进技术研究院）
Sun Yat-sen University（中山大学）；University of Texas at Austin（德克萨斯大学奥斯汀分校）
In this work, a parameter-efficient attention module is presented for emotion classification using a limited, or relatively small, number of electroencephalogram (EEG) signals. This module is called the Monotonicity Constrained Attention Module (MCAM) due to its capability of incorporating priors on the monotonicity when converting features' Gram matrices into attention matrices for better feature refinement. Our experiments have shown that MCAM's effectiveness is comparable to state-of-the-art attention modules in boosting the backbone network's performance in prediction while requiring less …
