arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2212.10030cs.AI
收藏

InterMulti:基于文本主导的分层高阶融合的多视角多模态交互情感分析

InterMulti:Multi-view Multimodal Interactions with Text-dominated Hierarchical High-order Fusion for Emotion Analysis

Virtual Human Group, Netease Fuxi AI Lab(网易伏羲人工智能实验室虚拟人组);College of Computer Science, Hangzhou Dianzi University(杭州电子科技大学计算机学院)

Feng Qiu, Wanzeng Kong, Yu Ding

摘要:

人类善于从多模态信号中解读对话者的情绪,如语音内容、语调和面部表情。然而,机器可能很难理解各种情绪,因为很难从多模态信号之间的复杂交互中有效解码情绪。在本文中,我们提出了一个多模态情感分析框架,InterMulti,从不同的角度捕捉复杂的多模态交互,并从多模态信号中识别情感。我们提出的框架将不同模态的信号分解为三种多模态交互表示,包括模态完整的交互表示,模态共享的交互表示和三种模态特定的交互表示。此外,为了平衡不同模态的贡献并学习更多信息的潜在交互表示,我们开发了一种新的文本主导的分层高阶融合(THHF)模块。THHF模块将上述三种表征合理地整合为一个综合的多模态交互表征。在广泛使用的数据集上的大量实验结果,(即)MOSEI,MOSI和IEMOCAP,证明我们的方法优于国家的最先进的。

英文摘要:

Humans are sophisticated at reading interlocutors' emotions from multimodal signals, such as speech contents, voice tones and facial expressions. However, machines might struggle to understand various emotions due to the difficulty of effectively decoding emotions from the complex interactions between multimodal signals. In this paper, we propose a multimodal emotion analysis framework, InterMulti, to capture complex multimodal interactions from different views and identify emotions from multimodal signals. Our proposed framework decomposes signals of different modalities into three kinds of multimodal interaction representations, including a modality-full interaction representation, a modality-shared interaction representation, and three modality-specific interaction representations. Additionally, to balance the contribution of different modalities and learn a more informative latent interaction representation, we developed a novel Text-dominated Hierarchical High-order Fusion(THHF) module. THHF module reasonably integrates the above three kinds of representations into a comprehensive multimodal interaction representation. Extensive experimental results on widely used datasets, (i.e.) MOSEI, MOSI and IEMOCAP, demonstrate that our method outperforms the state-of-the-art.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑