Automatic emotion recognition (AER) based on enriched multimodal inputs, including text, speech, and visual clues, is crucial in the development of emotionally intelligent machines. Although complex modality relationships have been proven effective for AER, they are still largely underexplored because previous works predominantly relied on various fusion mechanisms with simply concatenated features to learn multimodal representations for emotion classification. This paper proposes a novel hierarchical fusion graph convolutional network (HFGCN) model that learns more informative multimodal representations by considering the modality dependencies during the feature fusion procedure. Specifically, the proposed model fuses multimodality inputs using a two-stage graph construction approach and encodes the modality dependencies into the conversation representation. We verified the interpretable capabilities of the proposed method by projecting the emotional states to a 2D valence-arousal (VA) subspace. Extensive experiments showed the effectiveness of our proposed model for more accurate AER, which yielded state-of-the-art results on two public datasets, IEMOCAP and MELD.
Sync-TVA: A Graph-Attention Framework for Multimodal Emotion Recognition with Cross-Modal Fusion
James Watt School of Engineering, University of Glasgow（格拉斯哥大学詹姆斯·瓦特工程学院）；University of Bristol（布里斯托大学）；School of Engineering Mathematics and Technology, University of Bristol（布里斯托大学工程数学与技术学院）；School of Computing Science, University of Glasgow（格拉斯哥大学计算科学学院）；Department of Civil, Environmental & Geomatic Engineering, University College London (UCL)（伦敦大学学院（UCL）土木、环境与测绘工程系）
HeLo: Heterogeneous Multi-Modal Fusion with Label Correlation for Emotion Distribution Learning
College of Artificial Intelligence（人工智能学院）；Nanjing University of Aeronautics and Astronautics（南京航空航天大学）；School of Computer Science and Technology（计算机科学与技术学院）；Harbin Institute of Technology（哈尔滨工业大学）；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学深圳分校）
DRKF: Decoupled Representations with Knowledge Fusion for Multimodal Emotion Recognition
School of Computer Science and Engineering, University of Electronic Science and Technology of China（电子科技大学计算机科学与工程学院）；School of Information and Software Engineering, University of Electronic Science and Technology of China（电子科技大学信息与软件工程学院）
本文讨论了情感分类的问题。该任务包括预测最能描述短视频剪辑中包含的情绪的情绪标签（从一组可能的标签中选取）。建立在一个标准的框架-躺在描述视频的音频和视觉特征使用的监督分类器来推断标签-本文研究了几个新的方向。首先，提出了基于二维和三维卷积神经网络的改进人脸描述子。其次，本文探讨了几种融合方法，时间和多模态，包括一种新的分层方法相结合的功能和分数。此外，我们仔细审查了流水线的不同阶段，并设计了一个适应任务的CNN架构;这一点很重要，因为与问题的难度相比，训练集的大小很小，使得泛化困难。该模型在2017年的Emotion in the Wild挑战赛中排名第四，准确率为58.8%。
Multi-modal Attention for Speech Emotion Recognition
Institute of Data Science, NUS（新加坡国立大学数据科学研究所）；Graduate School for Integrative Sciences and Engineering, NUS（新加坡国立大学综合科学与工程研究生院）；National University of Singapore (NUS)（新加坡国立大学）；Osaka University（大阪大学）
Learning to Prefer Reliably: Error-Augmented Emotion Preference Optimization with Calibrated Fusion
Hong Kong Polytechnic University（香港理工大学）；Brno University of Technology（布尔诺理工大学）；Tsinghua University（清华大学）；National Taiwan University（台湾大学）；Kyoto University（京都大学）
MF-AED-AEC: Speech Emotion Recognition by Leveraging Multimodal Fusion, Asr Error Detection, and Asr Error Correction
Graduate School of Informatics, Nagoya University（名古屋大学信息学研究科）；School of Computer Science and Technology, Hainan University（海南大学计算机科学与技术学院）；Information Technology Center, Nagoya University（名古屋大学信息技术中心）
A cross-modal fusion network based on self-attention and residual structure for multimodal emotion recognition
Beijing University of Posts and Telecommunications（北京邮电大学）；East China Normal University（华东师范大学）；Shanghai University of International Business and Economics（上海对外经贸大学）
Emotion Collider: Dual Hyperbolic Mirror Manifolds for Sentiment Recovery via Anti Emotion Reflection
University of Macau（澳门大学）；Zhejiang University（浙江大学）；Tsinghua University（清华大学）；Tongji University（同济大学）；University of Southampton（南安普顿大学）；University of Bologna（博洛尼亚大学）；Minzu University of China（中央民族大学）
Emotion-Driven Personalized Recommendation for AI-Generated Content Using Multi-Modal Sentiment and Intent Analysis
University of Electronic Science and Technology of China（电子科技大学）；Columbia Business School, Columbia University（哥伦比亚大学哥伦比亚商学院）；New York University（纽约大学）
EmoEUS: Uncertainty Supervision for Multimodal Emotion Recognition in Conversation
Dept. of Electrical and Electronic Engineering, The Hong Kong Polytechnic University（香港理工大学电子与电气工程系）；Speech, Language, and Cognition Laboratory, The University of Hong Kong（香港大学语音、语言与认知实验室）
CMATH: Cross-Modality Augmented Transformer with Hierarchical Variational Distillation for Multimodal Emotion Recognition in Conversation
Chongqing University of Technology（重庆理工大学）；Fuzhou University（福州大学）；China Automotive Engineering Research Institute Co., Ltd.（中国汽车工程研究院股份有限公司）；Chongqing Changan Automobile Co., Ltd.（重庆长安汽车股份有限公司）
