Deep Imbalanced Learning for Multimodal Emotion Recognition in Conversations
Central South University of Forestry and Technology（中南林业科技大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；State University of New York, New Paltz（纽约州立大学新帕尔茨分校）
对话多模态情感识别（Multimodal Emotion Recognition in Conversations，MERC）的主要任务是识别文本、音频、图像和视频等模态中的情感，这是实现机器智能的重要发展方向。然而，MERC中的许多数据天然呈现出情感类别分布不平衡的问题，而研究人员忽视了不平衡数据对情感识别的负面影响。为解决这一问题，我们从数据增强、损失敏感性和采样策略三个方面进行系统分析，并提出了类别边界增强表示学习（Class Boundary Enhanced Representation Learning，CBERL）模型。具体而言，我们首先设计了一个多模态生成对抗网络，以解决原始数据中{emotion}类别分布不平衡的问题。其次，提出了一种深度联合变分自编码器，用于融合跨模态的互补语义信息，并获得具有判别性的特征表示。最后，我们实现了一个带有掩码重构和分类优化的多任务图神经网络，以解决类别边界学习中的过拟合和欠拟合问题，并实现跨模态情感识别。我们在IEMOCAP和MELD基准数据集上进行了大量实验，结果表明CBERL在情感识别有效性方面取得了一定的性能提升。特别是在少数类恐惧和厌恶情感标签上，我们的模型将准确率和F1值提高了10%至20%。
英文摘要：
The main task of Multimodal Emotion Recognition in Conversations (MERC) is to identify the emotions in modalities, e.g., text, audio, image and video, which is a significant development direction for realizing machine intelligence. However, many data in MERC naturally exhibit an imbalanced distribution of emotion categories, and researchers ignore the negative impact of imbalanced data on emotion recognition. To tackle this problem, we systematically analyze it from three aspects: data augmentation, loss sensitivity, and sampling strategy, and propose the Class Boundary Enhanced Representation Learning (CBERL) model. Concretely, we first design a multimodal generative adversarial network to address the imbalanced distribution of {emotion} categories in raw data. Secondly, a deep joint variational autoencoder is proposed to fuse complementary semantic information across modalities and obtain discriminative feature representations. Finally, we implement a multi-task graph neural network with mask reconstruction and classification optimization to solve the problem of overfitting and underfitting in class boundary learning, and achieve cross-modal emotion recognition. We have conducted extensive experiments on the IEMOCAP and MELD benchmark datasets, and the results show that CBERL has achieved a certain performance improvement in the effectiveness of emotion recognition. Especially on the minority class fear and disgust emotion labels, our model improves the accuracy and F1 value by 10% to 20%.
Emotion-Anchored Contrastive Learning Framework for Emotion Recognition in Conversation
National Key Laboratory for Novel Software Technology, Nanjing University（南京大学计算机软件新技术国家重点实验室）；School of Artificial Intelligence, Nanjing University（南京大学人工智能学院）
Speaker-Guided Encoder-Decoder Framework for Emotion Recognition in Conversation
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）
GatedxLSTM: A Multimodal Affective Computing Approach for Emotion Recognition in Conversations
Imperial College London（伦敦帝国学院）；Technical University of Munich（慕尼黑工业大学）；Konrad Zuse School of Excellence in Reliable AI（康拉德·祖斯可靠人工智能卓越学院）；Munich Data Science Institute（慕尼黑数据科学研究所）；Munich Center for Machine Learning（慕尼黑机器学习中心）
MSP-Conversation: A Corpus for Naturalistic, Time-Continuous Emotion Recognition
Erik Jonsson School of Engineering and Computer Science, University of Texas at Dallas（埃里克·乔纳森工程与计算机科学学院，德克萨斯大学达拉斯分校）；Language Technologies Institute, Carnegie Mellon University（语言技术研究所，卡内基梅隆大学）；AT&T Labs Research（AT&T实验室研究）
