可靠性感知的跨样本增强用于鲁棒多模态情感分析
Reliability-aware Cross-sample Enhancement for Robust Multimodal Sentiment Analysis
查看机构详情
- Sun Yat-sen University（中山大学）
- South China Normal University（华南师范大学）
机构由 AI 辅助整理，请以论文原文为准。
浏览论文内容
中文总结 AI 辅助
提出可靠性感知跨样本增强框架，通过自适应变分信息瓶颈和跨样本增强策略，解决多模态情感分析中的噪声与模态缺失问题，提升鲁棒性。
中文摘要 AI 辅助
多模态情感分析（MSA）旨在从文本、音频和视觉等多种模态中推断人类情感。在实践中，输入常常受到噪声和模态缺失的干扰，这会导致性能下降。现有方法通常孤立地处理这些挑战，限制了其在现实场景中的有效性。为解决这一局限性，我们提出了一个可靠性感知的跨样本增强（RCE）框架。具体而言，RCE首先引入自适应变分信息瓶颈来建模模态级别的不确定性，并执行质量感知的信息压缩，从而抑制不可靠模态中的冗余噪声。此外，我们设计了一种可靠性感知的跨样本增强策略，从大型候选池中检索高置信度、语义一致的邻居，以丰富和校准当前表示，有效缓解由模态缺失引起的信息不足。在此基础上，RCE通过多级可靠性感知融合机制整合跨模态交互，自适应地聚合来自不同模态和增强阶段的信息，从而产生更鲁棒的多模态表示。大量实验表明，RCE在完整、含噪和模态缺失设置下均持续优于最先进的方法。
英文摘要
Multimodal Sentiment Analysis (MSA) aims to infer human emotions from multiple modalities such as text, audio, and vision. In practice, inputs are often corrupted by noise and missing modalities, which degrades performance. Existing methods typically address these challenges in isolation, limiting their effectiveness in realistic settings. To address this limitation, we propose a Reliability-aware Cross-sample Enhancement (RCE) framework. Specifically, RCE first introduces an adaptive variational information bottleneck to model modality-wise uncertainty and perform quality-aware information compression, thereby suppressing redundant noise in unreliable modalities. Furthermore, we design a reliability-aware cross-sample enhancement strategy that retrieves high-confidence, semantically consistent neighbors from a large candidate pool to enrich and calibrate current representations, effectively alleviating information deficiency caused by missing modalities. Building upon this, RCE integrates cross-modal interactions with a multilevel reliability-aware fusion mechanism to adaptively aggregate information across modalities and enhancement stages, leading to more robust multimodal representations. Extensive experiments demonstrate that RCE consistently outperforms state-of-the-art methods across full, noisy, and missing-modality settings.
