Multimodal signals are more powerful than unimodal data for emotion recognition since they can represent emotions more comprehensively. In this paper, we introduce deep canonical correlation analysis (DCCA) to multimodal emotion recognition. The basic idea behind DCCA is to transform each modality separately and coordinate different modalities into a hyperspace by using specified canonical correlation analysis constraints. We evaluate the performance of DCCA on five multimodal datasets: the SEED, SEED-IV, SEED-V, DEAP, and DREAMER datasets. Our experimental results demonstrate that DCCA achieves state-of-the-art recognition accuracy rates on all five datasets: 94.58% on the SEED dataset, 87.45% on the SEED-IV dataset, 84.33% and 85.62% for two binary classification tasks and 88.51% for a four-category classification task on the DEAP dataset, 83.08% on the SEED-V dataset, and 88.99%, 90.57%, and 90.67% for three binary classification tasks on the DREAMER dataset. We also compare the noise robustness of DCCA with that of existing methods when adding various amounts of noise to the SEED-V dataset. The experimental results indicate that DCCA has greater robustness. By visualizing feature distributions with t-SNE and calculating the mutual information between different modalities before and after using DCCA, we find that the features transformed by DCCA from different modalities are more homogeneous and discriminative across emotions.
RoHyDR: Robust Hybrid Diffusion Recovery for Incomplete Multimodal Emotion Recognition
School of Computer Science and Engineering（计算机科学与工程学院）；South China University of Technology（华南理工大学）；School of Future Technology（未来技术学院）；University of Oxford（牛津大学）；Pengcheng Laboratory（鹏城实验室）；Department of Computer Science（计算机科学系）
本文讨论了情感分类的问题。该任务包括预测最能描述短视频剪辑中包含的情绪的情绪标签（从一组可能的标签中选取）。建立在一个标准的框架-躺在描述视频的音频和视觉特征使用的监督分类器来推断标签-本文研究了几个新的方向。首先，提出了基于二维和三维卷积神经网络的改进人脸描述子。其次，本文探讨了几种融合方法，时间和多模态，包括一种新的分层方法相结合的功能和分数。此外，我们仔细审查了流水线的不同阶段，并设计了一个适应任务的CNN架构;这一点很重要，因为与问题的难度相比，训练集的大小很小，使得泛化困难。该模型在2017年的Emotion in the Wild挑战赛中排名第四，准确率为58.8%。
LMR-CBT: Learning Modality-fused Representations with CB-Transformer for Multimodal Emotion Recognition from Unaligned Multimodal Sequences
Beijing University of Posts and Telecommunications（北京邮电大学）；East China Normal University（华东师范大学）；Shanghai University of International Business and Economics（上海对外经贸大学）
Touch and Tell: Multimodal Decoding of Human Emotions and Social Gestures for Robots
Faculty of Engineering and Architecture（工程与建筑学院）；IDLab-AIRO, Ghent University – imec（IDLab-AIRO，根特大学–imec）；Department of Engineering Science, University of Oxford（工程科学系，牛津大学）
EmoEUS: Uncertainty Supervision for Multimodal Emotion Recognition in Conversation
Dept. of Electrical and Electronic Engineering, The Hong Kong Polytechnic University（香港理工大学电子与电气工程系）；Speech, Language, and Cognition Laboratory, The University of Hong Kong（香港大学语音、语言与认知实验室）
Affective Multimodal Agents with Proactive Knowledge Grounding for Emotionally Aligned Marketing Dialogue
Hunan Police Academy, Department of Criminal Investigation（湖南警察学院犯罪侦查系）；Business College, California State University（加州州立大学长滩分校商学院）；Northwestern University（西北大学）；AI Agent Lab, Vokram Group（Vokram集团AI代理实验室）；Beijing University of Technology（北京理工大学）；Zheng Zhou University of Light Industry（郑州轻工业大学）
An Audio-Video Deep and Transfer Learning Framework for Multimodal Emotion Recognition in the wild
Ulm University（乌尔姆大学）；ITMO University（圣彼得堡国立信息技术机械与光学大学）；St. Petersburg Federal Research Center of the Russian Academy of Sciences（俄罗斯科学院圣彼得堡联邦研究中心）；Utrecht University（乌得勒支大学）
Study of Emotion Concept Formation by Integrating Vision, Physiology, and Word Information using Multilayered Multimodal Latent Dirichlet Allocation
Graduate School of Engineering Science, Osaka University（大阪大学工程科学研究生院）；Graduate School of Science and Technology, Nara Institute of Science and Technology（奈良科学技术大学院大学科学技术研究生院）
