arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2501.12424cs.LGcs.AIcs.IR
收藏

多模态协作学习用于情感分析

Multi-Modality Collaborative Learning for Sentiment Analysis

Shanmin Wang, Chengguang Liu, Qingshan Liu

AI总结:

本文提出多模态协作学习框架,通过解耦模块和策略模型提升跨模态情感特征学习,实验证明在四个数据库上性能显著提升。

摘要:

多模态情感分析(MSA)通过整合视觉、音频和文本模态来识别视频中个体的情感状态。尽管现有方法已取得进展,但固有的模态异质性限制了跨模态交互情感特征的有效捕捉。在本文中,我们引入了多模态协作学习(MMCL)框架,以促进跨模态交互,并分别从模态通用和模态特定的表示中捕捉增强和互补的特征。具体而言,我们设计了一个无参数解耦模块,并通过跨模态元素的语义评估将单模态分解为模态通用和模态特定的组件。对于模态特定的表示,受强化学习中动作-奖励机制的启发,我们设计了策略模型,在联合奖励的指导下自适应地挖掘互补的情感特征。对于模态通用的表示,采用模块内注意力机制来突出关键组件,在模态间发挥增强作用。实验结果包括在四个数据库上的优越性评估、每个模块的有效性验证以及互补特征的评估,证明MMCL成功地在跨模态中学习了协作特征,并显著提高了性能。代码可在https://github.com/smwanghhh/MMCL上获得。

英文摘要:

Multimodal sentiment analysis (MSA) identifies individuals' sentiment states in videos by integrating visual, audio, and text modalities. Despite progress in existing methods, the inherent modality heterogeneity limits the effective capture of interactive sentiment features across modalities. In this paper, by introducing a Multi-Modality Collaborative Learning (MMCL) framework, we facilitate cross-modal interactions and capture enhanced and complementary features from modality-common and modality-specific representations, respectively. Specifically, we design a parameter-free decoupling module and separate uni-modality into modality-common and modality-specific components through semantics assessment of cross-modal elements. For modality-specific representations, inspired by the act-reward mechanism in reinforcement learning, we design policy models to adaptively mine complementary sentiment features under the guidance of a joint reward. For modality-common representations, intra-modal attention is employed to highlight crucial components, playing enhanced roles among modalities. Experimental results, including superiority evaluations on four databases, effectiveness verification of each module, and assessment of complementary features, demonstrate that MMCL successfully learns collaborative features across modalities and significantly improves performance. The code can be available at https://github.com/smwanghhh/MMCL.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑