Emotions play a crucial role in human behavior and decision-making, making emotion recognition a key area of interest in human-computer interaction (HCI). This study addresses the challenges of emotion recognition by integrating facial expression analysis with electroencephalogram (EEG) signals, introducing a novel multimodal framework-Milmer. The proposed framework employs a transformer-based fusion approach to effectively integrate visual and physiological modalities. It consists of an EEG preprocessing module, a facial feature extraction and balancing module, and a cross-modal fusion module. To enhance visual feature extraction, we fine-tune a pre-trained Swin Transformer on emotion-related datasets. Additionally, a cross-attention mechanism is introduced to balance token representation across modalities, ensuring effective feature integration. A key innovation of this work is the adoption of a multiple instance learning (MIL) approach, which extracts meaningful information from multiple facial expression images over time, capturing critical temporal dynamics often overlooked in previous studies. Extensive experiments conducted on the DEAP dataset demonstrate the superiority of the proposed framework, achieving a classification accuracy of 96.72% in the four-class emotion recognition task. Ablation studies further validate the contributions of each module, highlighting the significance of advanced feature extraction and fusion strategies in enhancing emotion recognition performance. Our code are available at https://github.com/liangyubuaa/Milmer.
Emotion-LLaMAv2 and MMEVerse: A New Framework and Benchmark for Multimodal Emotion Understanding
Shenzhen Technology University(深圳技术大学);Shenzhen University of Information Technology(深圳信息科技学院);University of Washington(华盛顿大学);Foundation Model Team, Meituan(美团基础模型团队);National University of Singapore(新加坡国立大学);Tongji University(同济大学)
MCN-CL: Multimodal Cross-Attention Network and Contrastive Learning for Multimodal Emotion Recognition
School of Computer and Information Engineering, Anhui University of Finance and Economics, Anhui, 233030, China(计算机与信息工程学院,安徽财经大学,安徽,233030,中国);State Key Laboratory of Cognitive Science and Mental Health, Institute of Psychology, Chinese Academy of Sciences, Beijing, 100045, China(认知科学与心理健康国家重点实验室,心理学研究所,中国科学院,北京,100045,中国)
Investigation of Multimodal Features, Classifiers and Fusion Methods for Emotion Recognition
自动情感识别是一项具有挑战性的任务。在本文中,我们介绍了我们在2018年野外情绪识别(Emotion Recognition in the Wild,EMOTIW)挑战赛中基于音频视频的子挑战赛的努力,该挑战赛要求参与者从六种普遍情绪(愤怒,厌恶,恐惧,快乐,悲伤和惊讶)和中性中为视频片段分配一个单一的情绪标签。所提出的多模态情感识别系统将音频、视频和文本信息考虑在内。除了手工特征,我们还通过迁移学习从深度神经网络(DNN)中提取瓶颈特征。对时间分类器和非时间分类器进行评估,以获得最佳的单峰情感分类结果。然后提取可能性并将其传递到束搜索融合(BS-融合)中。我们在2018年的挑战赛中测试了我们的方法,并获得了令人鼓舞的结果。与基线系统相比,有了明显的改进。我们在测试数据集上实现了60.34%的准确率,仅比获胜者低1.5%。这表明我们的方法非常有竞争力。
EmoMeta: A Multimodal Dataset for Fine-grained Emotion Classification in Chinese Metaphors
School of Software(软件学院);Dalian University of Technology(大连理工大学);School of Foreign Languages and School of Software(外语学院和软件学院);Faculty of Business Administration(商学院);University of Macau(澳门大学);School of Computing Technologies(计算技术学院);RMIT University(皇家墨尔本理工大学)
RoHyDR: Robust Hybrid Diffusion Recovery for Incomplete Multimodal Emotion Recognition
School of Computer Science and Engineering(计算机科学与工程学院);South China University of Technology(华南理工大学);School of Future Technology(未来技术学院);University of Oxford(牛津大学);Pengcheng Laboratory(鹏城实验室);Department of Computer Science(计算机科学系)
XEmoGPT: An Explainable Multimodal Emotion Recognition Framework with Cue-Level Perception and Reasoning
School of Computer Science and Engineering, University of Electronic Science and Technology of China(电子科技大学计算机科学与工程学院);th Research Institute, China Electronics Technology Group Corporation(中国电子科技集团第五十四研究所)
VISTANet: VIsual Spoken Textual Additive Net for Interpretable Multimodal Emotion Recognition
Center for Machine Vision and Signal Analysis, University of Oulu(机器视觉与信号分析中心,奥卢大学);Indian Institute of Technology Roorkee(印度理工学院罗尔基分校);State Key Laboratory of Blockchain and Data Security, Zhejiang University(区块链与数据安全国家重点实验室,浙江大学)
Leveraging Label Potential for Enhanced Multimodal Emotion Recognition
Xinjiang University(新疆大学);School of Computer Science and Technology(计算机科学与技术学院);Xinjiang Multimodal Intelligent Processing and Information Security Engineering Technology Research Center(新疆多模态智能处理与信息安全工程技术研究中心)
The MuSe 2021 Multimodal Sentiment Analysis Challenge: Sentiment, Emotion, Physiological-Emotion, and Stress
University of Augsburg(奥格斯堡大学);University of Ulm(乌尔姆大学);Nanyang Technological University(南洋理工大学);University of Oulu(奥卢大学);Imperial College London(帝国理工学院)
Noise-Resistant Multimodal Transformer for Emotion Recognition
China University of Geosciences (Wuhan)(中国地质大学(武汉));JD Explore Academy, JD.com(京东集团京东探索研究院);Australian Catholic University(澳大利亚天主教大学);The University of Sydney(悉尼大学)
Multimodal Video Emotion Recognition with Reliable Reasoning Priors
Lenovo Research(联想研究院);School of Artificial Intelligence, UCAS(中国科学院大学人工智能学院);Institute of Automation, CAS(中国科学院自动化研究所);Macau University of Science and Technology(澳门科学理工学院);School of Computer Science and Technology, UCAS(中国科学院大学计算机科学与技术学院)
EmoAgent-R1: Towards Multimodal Emotion Understanding with Reinforcement Learning-based Dynamic Agent Specialization
Guangdong University of Technology(广东工业大学);Southern University of Science and Technology(南方科技大学);Xi’an Jiaotong University(西安交通大学);Guizhou Provincial Laboratory of Big Data, State Key Laboratory of Public Big Data, Guizhou University(贵州大学大数据省级重点实验室、公共大数据国家重点实验室)
EmotionHallucer: Evaluating Emotion Hallucinations in Multimodal Large Language Models
Lappeenranta-Lahti University of Technology LUT(拉佩兰塔-拉赫蒂技术大学);University of Oulu(奥卢大学);Southeast University(东南大学);Shanghai Jiao Tong University(上海交通大学);Brno University of Technology(布拉格技术大学)
Beyond Emotion Recognition: A Multi-Turn Multimodal Emotion Understanding and Reasoning Benchmark
Hefei University of Technology(合肥工业大学);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));University of Science and Technology of China(中国科学技术大学);Institute of Artificial Intelligence (IAI), Hefei Comprehensive National Science Center(人工智能研究院(IAI),合肥综合性国家科学中心)
Enriching Multimodal Sentiment Analysis through Textual Emotional Descriptions of Visual-Audio Content
Tianjin University(天津大学);Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)(广东人工智能与数字经济实验室(深圳));Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院)