Multimodal Emotion Recognition in Conversations (ERC) is a typical multimodal learning task in exploiting various data modalities concurrently. Prior studies on effective multimodal ERC encounter challenges in addressing modality imbalances and optimizing learning across modalities. Dealing with these problems, we present a novel framework named Ada2I, which consists of two inseparable modules namely Adaptive Feature Weighting (AFW) and Adaptive Modality Weighting (AMW) for feature-level and modality-level balancing respectively via leveraging both Inter- and Intra-modal interactions. Additionally, we introduce a refined disparity ratio as part of our training optimization strategy, a simple yet effective measure to assess the overall discrepancy of the model's learning process when handling multiple modalities simultaneously. Experimental results validate the effectiveness of Ada2I with state-of-the-art performance compared to baselines on three benchmark datasets, particularly in addressing modality imbalances.
Orthogonal Disentanglement with Projected Feature Alignment for Multimodal Emotion Recognition in Conversation
School of Computer Science Sichuan University(四川大学计算机学院);Faculty of Computing Harbin Institute of Technology(哈尔滨工业大学计算学部);College of Computer Science and Technology Harbin Engineering University(哈尔滨工程大学计算机科学与技术学院)
Angle-Optimized Partial Disentanglement for Multimodal Emotion Recognition in Conversation
Faculty of Computing, Harbin Institute of Technology(哈尔滨工业大学计算学部);School of Astronautics, Beihang University(北京航空航天大学宇航学院);Group of Intelligent Signal Processing (GISP), College of Computer Science and Technology, Harbin Engineering University(哈尔滨工程大学计算机科学与技术学院智能信号处理研究组)
CMATH: Cross-Modality Augmented Transformer with Hierarchical Variational Distillation for Multimodal Emotion Recognition in Conversation
Chongqing University of Technology(重庆理工大学);Fuzhou University(福州大学);China Automotive Engineering Research Institute Co., Ltd.(中国汽车工程研究院股份有限公司);Chongqing Changan Automobile Co., Ltd.(重庆长安汽车股份有限公司)
College of Computer and Mathematics, Central South University of Forestry and Technology(计算机与数学学院,中央南林业科技大学);Department of Computer Science, State University of New York(计算机科学系,纽约州立大学)
Multi-Modality in Music: Predicting Emotion in Music from High-Level Audio Features and Lyrics
University of Copenhagen(哥本哈根大学)
本文旨在测试多模态方法的音乐情感识别(MER)是否比单模态的高层次的歌曲特征和歌词。我们使用从Spotify API检索的11个歌曲特征,组合歌词特征,包括情感,TF-IDF和Anew,以预测Deezer情绪检测数据集(DMDD)(Delbouys et al.,2018),具有4种不同的回归模型。我们发现,在11个高级歌曲特征中,主要有5个对性能有贡献,多模态特征在预测效价时比单独的音频做得更好。我们使我们的代码可供使用。
DRKF: Decoupled Representations with Knowledge Fusion for Multimodal Emotion Recognition
School of Computer Science and Engineering, University of Electronic Science and Technology of China(电子科技大学计算机科学与工程学院);School of Information and Software Engineering, University of Electronic Science and Technology of China(电子科技大学信息与软件工程学院)
EmoEUS: Uncertainty Supervision for Multimodal Emotion Recognition in Conversation
Dept. of Electrical and Electronic Engineering, The Hong Kong Polytechnic University(香港理工大学电子与电气工程系);Speech, Language, and Cognition Laboratory, The University of Hong Kong(香港大学语音、语言与认知实验室)