Multimodal emotion recognition is an important research topic in artificial intelligence, whose main goal is to integrate multimodal clues to identify human emotional states. Current works generally assume accurate labels for benchmark datasets and focus on developing more effective architectures. However, emotion annotation relies on subjective judgment. To obtain more reliable labels, existing datasets usually restrict the label space to some basic categories, then hire plenty of annotators and use majority voting to select the most likely label. However, this process may result in some correct but non-candidate or non-majority labels being ignored. To ensure reliability without ignoring subtle emotions, we propose a new task called ``Explainable Multimodal Emotion Recognition (EMER)''. Unlike traditional emotion recognition, EMER takes a step further by providing explanations for these predictions. Through this task, we can extract relatively reliable labels since each label has a certain basis. Meanwhile, we borrow large language models (LLMs) to disambiguate unimodal clues and generate more complete multimodal explanations. From them, we can extract richer emotions in an open-vocabulary manner. This paper presents our initial attempt at this task, including introducing a new dataset, establishing baselines, and defining evaluation metrics. In addition, EMER can serve as a benchmark task to evaluate the audio-video-text understanding performance of multimodal LLMs.
Emotional expressions form a key part of user behavior on today's digital platforms. While multimodal emotion recognition techniques are gaining research attention, there is a lack of deeper understanding on how visual and non-visual features can be used to better recognize emotions in certain contexts, but not others. This study analyzes the interplay between the effects of multimodal emotion features derived from facial expressions, tone and text in conjunction with two key contextual factors: i) gender of the speaker, and ii) duration of the emotional episode. Using a large public dataset o…
XEmoGPT: An Explainable Multimodal Emotion Recognition Framework with Cue-Level Perception and Reasoning
School of Computer Science and Engineering, University of Electronic Science and Technology of China（电子科技大学计算机科学与工程学院）；th Research Institute, China Electronics Technology Group Corporation（中国电子科技集团第五十四研究所）
Noise-Resistant Multimodal Transformer for Emotion Recognition
China University of Geosciences (Wuhan)（中国地质大学（武汉））；JD Explore Academy, JD.com（京东集团京东探索研究院）；Australian Catholic University（澳大利亚天主教大学）；The University of Sydney（悉尼大学）
Ping An Life Insurance Company of China, Ltd.（中国平安人寿保险股份有限公司）；Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）
Emotion Recognition in Conversations (ERC) has considerable prospects for developing empathetic machines. For multimodal ERC, it is vital to understand context and fuse modality information in conversations. Recent graph-based fusion methods generally aggregate multimodal information by exploring unimodal and cross-modal interactions in a graph. However, they accumulate redundant information at each layer, limiting the context understanding between modalities. In this paper, we propose a novel Multimodal Dynamic Fusion Network (MM-DFN) to recognize emotions by fully understanding multimodal co…
Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning
Shenzhen Technology University（深圳技术大学）；Carnegie Mellon University（卡内基梅隆大学）；Alibaba Group（阿里巴巴集团）；National University of Singapore（新加坡国立大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
EmoEUS: Uncertainty Supervision for Multimodal Emotion Recognition in Conversation
Dept. of Electrical and Electronic Engineering, The Hong Kong Polytechnic University（香港理工大学电子与电气工程系）；Speech, Language, and Cognition Laboratory, The University of Hong Kong（香港大学语音、语言与认知实验室）
College of Computer and Mathematics, Central South University of Forestry and Technology（计算机与数学学院，中央南林业科技大学）；Department of Computer Science, State University of New York（计算机科学系，纽约州立大学）
This paper addresses the question of emotion classification. The task consists in predicting emotion labels (taken among a set of possible labels) best describing the emotions contained in short video clips. Building on a standard framework -- lying in describing videos by audio and visual features used by a supervised classifier to infer the labels -- this paper investigates several novel directions. First of all, improved face descriptors based on 2D and 3D Convo-lutional Neural Networks are proposed. Second, the paper explores several fusion methods, temporal and multimodal, including a nov…
Sync-TVA: A Graph-Attention Framework for Multimodal Emotion Recognition with Cross-Modal Fusion
James Watt School of Engineering, University of Glasgow（格拉斯哥大学詹姆斯·瓦特工程学院）；University of Bristol（布里斯托大学）；School of Engineering Mathematics and Technology, University of Bristol（布里斯托大学工程数学与技术学院）；School of Computing Science, University of Glasgow（格拉斯哥大学计算科学学院）；Department of Civil, Environmental & Geomatic Engineering, University College London (UCL)（伦敦大学学院（UCL）土木、环境与测绘工程系）
