AVERE: 通过偏好优化提升音频视觉情感推理
AVERE: Improving Audiovisual Emotion Reasoning with Preference Optimization
AI总结：
AVERE通过偏好优化技术提升音频视觉情感推理性能，解决情感与无关线索的虚假关联和幻觉问题，提升多模态模型在情感理解中的表现。
摘要：
情感理解对于构建具有社会智能的智能体至关重要。尽管最近的多模态大语言模型在该任务上表现出色，但仍然存在两个关键挑战——情感与无关音频视觉线索之间的虚假关联，以及由语言模型基础架构中的文本先验驱动的音频视觉线索幻觉。为了量化和理解这些问题，我们引入了EmoReAlM基准，用于评估MLLMs在线索-情感关联、幻觉和模态一致性方面的性能。我们随后提出了AVEm-DPO，一种偏好优化技术，使模型响应与音频视觉输入和以情感为中心的查询对齐。具体而言，我们构建了显示虚假关联或幻觉的响应偏好，以及由文本提示引导的音频视觉输入对。我们还包含一个正则化项，惩罚对文本先验的依赖，从而减轻模态特定的线索幻觉。在DFEW、RAVDESS和EMER上的实验结果表明，我们的方法在零样本设置中显著提高了参考基线模型的性能，相对性能提升为6-19%。通过提供严格基准和稳健的优化框架，这项工作使MLLMs在情感理解和社交AI中的原则性评估和改进成为可能。代码、模型和基准将在https://avere-iclr.github.io发布。
英文摘要：
Emotion understanding is essential for building socially intelligent agents. Although recent multimodal large language models have shown strong performance on this task, two key challenges remain - spurious associations between emotions and irrelevant audiovisual cues, and hallucinations of audiovisual cues driven by text priors in the language model backbone. To quantify and understand these issues, we introduce EmoReAlM, a benchmark designed to evaluate MLLMs for cue-emotion associations, hallucinations and modality agreement. We then propose AVEm-DPO, a preference optimization technique that aligns model responses with both audiovisual inputs and emotion-centric queries. Specifically, we construct preferences over responses exhibiting spurious associations or hallucinations, and audiovisual input pairs guided by textual prompts. We also include a regularization term that penalizes reliance on text priors, thereby mitigating modality-specific cue hallucinations. Experimental results on DFEW, RAVDESS and EMER demonstrate that our method significantly improves the performance of the reference baseline models with 6-19% of relative performance gains in zero-shot settings. By providing both a rigorous benchmark and a robust optimization framework, this work enables principled evaluation and improvement of MLLMs for emotion understanding and social AI. Code, models and benchmark will be released at https://avere-iclr.github.io.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
