arXiv 2112.016972021-12-03cs.CVcs.CLcs.LGcs.SDeess.AS

LMR-CBT：基于CB-Transformer的模态融合表示学习，用于非对齐多模态序列的多模态情感识别

LMR-CBT: Learning Modality-fused Representations with CB-Transformer for Multimodal Emotion Recognition from Unaligned Multimodal Sequences

Beijing University of Posts and Telecommunications（北京邮电大学）；East China Normal University（华东师范大学）；Shanghai University of International Business and Economics（上海对外经贸大学）

Ziwang Fu, Feng Liu, Hanyang Wang, Siyuan Shen, Jiahao Zhang, Jiayin Qi, Xiangling Fu, Aimin Zhou

备注：: 9 pages ,Figure 2, Table 5

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

摘要：

学习模态融合表示和处理不对齐的多模态序列在多模态情感识别中是有意义和具有挑战性的。现有的方法使用定向成对注意或消息中心来融合语言、视觉和听觉模态。然而，这些方法在融合特征时引入了信息冗余，并且没有考虑模态的互补性而效率低下。在本文中，我们提出了一个有效的神经网络学习模态融合表示与CB-Transformer（LMR-CBT）的多模态情感识别从非对齐的多模态序列。具体来说，我们首先分别对三种模态进行特征提取，得到序列的局部结构。然后，我们设计了一种新的跨模态块的Transformer（CB-Transformer），它可以实现不同模态的互补学习，主要分为局部时间学习，跨模态特征融合和全局自注意表示。此外，我们拼接融合的功能与原始功能的分类序列的情感。最后，我们在三个具有挑战性的数据集IEMOCAP，CMU-MOSI和CMU-MOSEI上进行了单词对齐和未对齐的实验。实验结果表明，我们提出的方法在这两种设置的优越性和效率。与主流方法相比，我们的方法达到了国家的最先进的最少数量的参数。

英文摘要：

Learning modality-fused representations and processing unaligned multimodal sequences are meaningful and challenging in multimodal emotion recognition. Existing approaches use directional pairwise attention or a message hub to fuse language, visual, and audio modalities. However, those approaches introduce information redundancy when fusing features and are inefficient without considering the complementarity of modalities. In this paper, we propose an efficient neural network to learn modality-fused representations with CB-Transformer (LMR-CBT) for multimodal emotion recognition from unaligned multimodal sequences. Specifically, we first perform feature extraction for the three modalities respectively to obtain the local structure of the sequences. Then, we design a novel transformer with cross-modal blocks (CB-Transformer) that enables complementary learning of different modalities, mainly divided into local temporal learning,cross-modal feature fusion and global self-attention representations. In addition, we splice the fused features with the original features to classify the emotions of the sequences. Finally, we conduct word-aligned and unaligned experiments on three challenging datasets, IEMOCAP, CMU-MOSI, and CMU-MOSEI. The experimental results show the superiority and efficiency of our proposed method in both settings. Compared with the mainstream methods, our approach reaches the state-of-the-art with a minimum number of parameters.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。