当眼睛与耳朵产生分歧：多模态大语言模型能否识别视听混淆？
When Eyes and Ears Disagree: Can MLLMs Discern Audio-Visual Confusion?
AI总结：
针对MLLMs因视觉主导推理难以识别视觉存在但听觉缺失的混淆对象问题，本文提出AV-ConfuseBench基准和RL-CoMM模型，通过引入LALM参考与置信度优化，在有限数据下将准确率提升10~30%。
摘要：
多模态大语言模型（MLLMs）能否识别视觉存在但听觉缺失的混淆对象？为研究此问题，我们引入新基准AV-ConfuseBench，通过修改视频中物体的对应声音模拟“视听混淆”场景，例如静音发声物体并询问MLLMs“是否存在静音物体的声音”。实验结果显示，Qwen2.5-Omni和Gemini 2.5等MLLMs因视觉主导推理难以区分不存在的音频。基于此观察，我们提出基于Qwen2.5-Omni基础的强化学习协作多MLLM模型RL-CoMM，包含两个阶段：1）引入外部大音频语言模型（LALM）作为参考模型生成纯音频推理，缓解视觉主导歧义；设计分步推理奖励函数，使MLLMs借助纯音频参考自我提升视听推理能力。2）引入以答案为中心的置信度优化，减少潜在异构推理差异的不确定性。在视听问答和视听幻觉任务的大量实验表明，RL-CoMM在有限训练数据下比基线模型准确率提升10~30%。更多信息：https://github.com/rikeilong/AVConfusion。
英文摘要：
Can Multimodal Large Language Models (MLLMs) discern confused objects that are visually present but audio-absent? To study this, we introduce a new benchmark, AV-ConfuseBench, which simulates an ``Audio-Visual Confusion'' scene by modifying the corresponding sound of an object in the video, e.g., mute the sounding object and ask MLLMs Is there a/an muted-object sound''. Experimental results reveal that MLLMs, such as Qwen2.5-Omni and Gemini 2.5, struggle to discriminate non-existent audio due to visually dominated reasoning. Motivated by this observation, we introduce RL-CoMM, a Reinforcement Learning-based Collaborative Multi-MLLM that is built upon the Qwen2.5-Omni foundation. RL-CoMM includes two stages: 1) To alleviate visually dominated ambiguities, we introduce an external model, a Large Audio Language Model (LALM), as the reference model to generate audio-only reasoning. Then, we design a Step-wise Reasoning Reward function that enables MLLMs to self-improve audio-visual reasoning with the audio-only reference. 2) To ensure an accurate answer prediction, we introduce Answer-centered Confidence Optimization to reduce the uncertainty of potential heterogeneous reasoning differences. Extensive experiments on audio-visual question answering and audio-visual hallucination show that RL-CoMM improves the accuracy by 10~30\% over the baseline model with limited training data. Follow: https://github.com/rikeilong/AVConfusion.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
