融合细粒度音频视觉证据以实现鲁棒的多模态情绪推理
Integrating Fine-Grained Audio-Visual Evidence for Robust Multimodal Emotion Reasoning
AI总结：
SABER-LLM通过构建大规模情绪推理数据集和结构化证据分解范式，实现鲁棒的多模态情绪推理。
摘要：
多模态情绪分析正从静态分类转向生成性推理。超越简单的标签预测，鲁棒的情感推理必须综合细粒度信号，如面部微表情和语调，以解码复杂社会情境中的潜在因果关系。然而，当前的多模态大语言模型（MLLMs）在细粒度感知方面面临显著限制，主要由于数据稀缺和跨模态融合不足。因此，这些模型往往表现出单模态主导，导致在复杂多模态交互中出现幻觉，特别是在视觉和声学线索微妙、模糊甚至矛盾（例如在讽刺场景中）时。为了解决这个问题，我们引入了SABER-LLM框架，旨在实现鲁棒的多模态推理。首先，我们构建了SABER，一个包含60万视频片段的大规模情绪推理数据集，标注有新颖的六维模式，共同捕捉音频视觉线索和因果逻辑。其次，我们提出了结构化证据分解范式，该范式在证据提取和推理之间强制实施“感知-然后推理”的分离，以缓解单模态主导。通过一致性感知的直接偏好优化，进一步增强了感知复杂场景的能力，该方法在模糊或冲突的感知条件下明确鼓励模态间的对齐。在EMER、EmoBench-M和SABER-Test上的实验表明，SABER-LLM在开放源代码基线中显著优于，并在解码复杂情绪动态方面与闭源模型具有竞争力。数据集和模型可在https://github.com/zxzhao0/SABER-LLM上获得。
英文摘要：
Multimodal emotion analysis is shifting from static classification to generative reasoning. Beyond simple label prediction, robust affective reasoning must synthesize fine-grained signals such as facial micro-expressions and prosodic which shifts to decode the latent causality within complex social contexts. However, current Multimodal Large Language Models (MLLMs) face significant limitations in fine-grained perception, primarily due to data scarcity and insufficient cross-modal fusion. As a result, these models often exhibit unimodal dominance which leads to hallucinations in complex multimodal interactions, particularly when visual and acoustic cues are subtle, ambiguous, or even contradictory (e.g., in sarcastic scenery). To address this, we introduce SABER-LLM, a framework designed for robust multimodal reasoning. First, we construct SABER, a large-scale emotion reasoning dataset comprising 600K video clips, annotated with a novel six-dimensional schema that jointly captures audiovisual cues and causal logic. Second, we propose the structured evidence decomposition paradigm, which enforces a "perceive-then-reason" separation between evidence extraction and reasoning to alleviate unimodal dominance. The ability to perceive complex scenes is further reinforced by consistency-aware direct preference optimization, which explicitly encourages alignment among modalities under ambiguous or conflicting perceptual conditions. Experiments on EMER, EmoBench-M, and SABER-Test demonstrate that SABER-LLM significantly outperforms open-source baselines and achieves robustness competitive with closed-source models in decoding complex emotional dynamics. The dataset and model are available at https://github.com/zxzhao0/SABER-LLM.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
