arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.28575cs.AI
收藏

一种冲突感知惩罚与统计损失框架,用于平衡模态并增强多模态情感分析的稳定性

A Conflict-Aware Penalty and Statistical Loss Framework for Balancing Modalities and Enhancing Stability in Multimodal Sentiment Analysis

School of Computer Science, South China Normal University(华南师范大学计算机学院)

Jianheng Dai, Jiazhang Liang, Sijie Mai

AI总结:

针对多模态情感分析中文本模态主导导致梯度冲突的问题,提出冲突感知惩罚和统计损失框架,实现模态平衡与训练稳定,在CMU-MOSI上取得最优性能。

摘要:

多模态情感分析(MSA)融合文本、声学和视觉流来推断情感。由于预训练文本编码器的表达能力远强于声学和视觉编码器,文本模态往往主导优化过程,抑制较弱模态并引发梯度范数冲突,从而破坏训练稳定性。为解决此问题,我们提出一种冲突感知惩罚(CP),在每一步训练中检测并惩罚梯度范数冲突,以及一种统计损失(SL),使预测分布统计量与经验输入统计量对齐。关键的是,CP防止主导模态梯度干扰SL目标,从而在统一框架内实现协同训练,该框架包含自适应模态编码、门控跨模态融合和单模态辅助头。在CMU-MOSI上的实验表明,该方法达到了最先进的性能,消融研究证实了每个组件的有效性。

英文摘要:

Multimodal Sentiment Analysis (MSA) fuses text, acoustic, and visual streams to infer sentiment. Because pre-trained text encoders are far more expressive than their acoustic and visual counterparts, the text modality tends to dominate optimization, suppressing weaker modalities and inducing gradient norm conflicts that destabilize training. To address this, we propose a Conflict-aware Penalty (CP) that detects and penalizes gradient norm conflicts at each training step, and a Statistical Loss (SL) that aligns predicted distribution statistics with empirical input statistics. Crucially, CP prevents dominant modality gradients from interfering with the SL objective, enabling synergistic training within a unified framework incorporating adaptive modality encoding, gated cross-modal fusion, and unimodal auxiliary heads. Experiments on CMU-MOSI demonstrate state-of-the-art performance, with ablation studies confirming the effectiveness of each component.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑