先分后融：通过模态特定的推理链减轻音频视觉大语言模型中的跨模态干扰
Separate First, Fuse Later: Mitigating Cross-Modal Interference in Audio-Visual LLMs Reasoning with Modality-Specific Chain-of-Thought
AI总结：
本文提出SFFL框架，通过模态特定的推理链减少跨模态干扰，提升音频视觉问答的准确性和鲁棒性，实验显示在通用AVQA基准和跨模态幻觉基准上分别提升5.16%和11.17%。
摘要：
音频和视觉为音频视觉问答提供了互补的证据，但当前的音频视觉大语言模型可能受到跨模态干扰：一种模态的信息会误导另一种模态的解释，从而导致幻觉。我们将其归因于中间推理过程中不受控制的跨模态交互。为缓解这一问题，我们提出Separate First, Fuse Later（SFFL），一种音频视觉推理框架，旨在减少跨模态干扰。SFFL强制执行模态特定的推理链，生成单独的音频和视觉推理轨迹，并整合证据以回答问题。我们通过在不同模态输入设置下的数据管道构建模态偏好标签。我们使用这些标签作为强化学习中的辅助奖励，以鼓励实例依赖的模态偏好。我们进一步引入了模态特定的推理机制，在分离推理阶段保持模态隔离，同时在证据融合阶段允许完全访问跨模态信息。实验显示在准确性和鲁棒性上均有所提升，在通用AVQA基准上平均相对增益为5.16%，在跨模态幻觉基准上为11.17%。
英文摘要：
Audio and vision provide complementary evidence for audio-visual question answering, yet current audio-visual large language models may suffer from cross-modal interference: information from one modality misguides the interpretation of another, thereby inducing hallucinations. We attribute this issue to uncontrolled cross-modal interactions during intermediate reasoning. To mitigate this, we propose Separate First, Fuse Later (SFFL), an audio-visual reasoning framework designed to reduce cross-modal interference. SFFL enforces modality-specific chain-of-thought reasoning, producing separate audio and visual reasoning traces and integrating evidence for answering. We construct modality-preference labels via a data pipeline under different modality input settings. We use these labels as an auxiliary reward in reinforcement learning to encourage a instance-dependent preference for modality cues when answering. We further introduce a modality-specific reasoning mechanism that preserves modality isolation during the separated reasoning stage while enabling full access to cross-modal information at the evidence fusion stage. Experiments demonstrate consistent improvements in both accuracy and robustness, yielding an average relative gain of 5.16\% on general AVQA benchmarks and 11.17\% on a cross-modal hallucination benchmark.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
