多模态情感推理中情感冲突的基准测试与桥接
Benchmarking and Bridging Emotion Conflicts for Multimodal Emotion Reasoning
AI总结：
针对多模态大模型在情感冲突中过度依赖音频的问题，提出CA-MER基准与参数高效框架MoSEAR，通过模态专家与注意力重分配平衡多模态信息，在冲突与一致样本上均达到SOTA。
摘要：
尽管现有的多模态大语言模型（MLLMs）在多模态情感推理中表现出色，但它们往往忽略了涉及情感冲突的场景，即来自不同模态的情感线索不一致。为了填补这一空白，我们首先引入了CA-MER，这是一个旨在检验MLLMs在现实情感冲突下表现的新基准。它包含三个子集：视频对齐、音频对齐和一致，其中仅有一种或所有模态反映了真实情感。然而，对我们的CA-MER的评估表明，当前最先进的情感MLLMs在情感冲突中系统性地过度依赖音频信号，忽视了来自视觉模态的关键线索。为了缓解这种偏差，我们提出了MoSEAR，这是一个促进模态平衡集成的参数高效框架。MoSEAR由两个模块组成：(1)MoSE，具有正则化门控机制的模态特定专家，用于减少微调头中的模态偏差；以及(2)AR，一种注意力重分配机制，在推理期间重新平衡冻结骨干网络中的模态贡献。我们的框架具有两个关键优势：它缓解了情感冲突并提升了一致性样本上的性能，且不会在音频和视觉模态之间产生权衡。在多个基准（包括MER2023、EMER、DFEW和我们的CA-MER）上的实验表明，MoSEAR实现了最先进的性能，特别是在模态冲突条件下。
英文摘要：
Despite their strong performance in multimodal emotion reasoning, existing Multimodal Large Language Models (MLLMs) often overlook the scenarios involving emotion conflicts, where emotional cues from different modalities are inconsistent. To fill this gap, we first introduce CA-MER, a new benchmark designed to examine MLLMs under realistic emotion conflicts. It consists of three subsets: video-aligned, audio-aligned, and consistent, where only one or all modalities reflect the true emotion. However, evaluations on our CA-MER reveal that current state-of-the-art emotion MLLMs systematically over-rely on audio signal during emotion conflicts, neglecting critical cues from visual modality. To mitigate this bias, we propose MoSEAR, a parameter-efficient framework that promotes balanced modality integration. MoSEAR consists of two modules: (1)MoSE, modality-specific experts with a regularized gating mechanism that reduces modality bias in the fine-tuning heads; and (2)AR, an attention reallocation mechanism that rebalances modality contributions in frozen backbones during inference. Our framework offers two key advantages: it mitigates emotion conflicts and improves performance on consistent samples-without incurring a trade-off between audio and visual modalities. Experiments on multiple benchmarks-including MER2023, EMER, DFEW, and our CA-MER-demonstrate that MoSEAR achieves state-of-the-art performance, particularly under modality conflict conditions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
