COPO：面向MLLM幻觉的因果导向策略优化
COPO: Causal-Oriented Policy Optimization for Hallucinations of MLLMs
AI总结：
COPO通过引入因果完整性奖励和GRPO框架，解决多模态大语言模型的幻觉问题，通过令牌级因果约束确保输出的正确性和证据基础。
摘要：
尽管多模态大语言模型（MLLMs）展现出了令人印象深刻的性能，但它们可能会产生幻觉。实证研究表明，MLLMs相比纯文本语言模型更倾向于关注与任务无关的背景区域，暗示了虚假的背景-答案相关性。我们主张并分析认为（i）基于结果的奖励可以是导致虚假相关性的关键因素，（ii）虚假相关性可以是导致幻觉的关键因素。基于这些结果，我们提出了因果导向的策略优化（COPO）以减轻这些虚假相关性，从而解决幻觉问题。它对每个推理标记施加令牌级的充分性和必要性约束，从而确保正确且有证据支持的输出。具体而言，我们首先通过新提出的因果完整性奖励评估每个标记的因果贡献。该奖励随后用于在GRPO优化框架内构建一个因果导向的优势函数，鼓励模型关注对准确生成具有因果充分性和必要性的标记。在各种基准测试中的实验结果展示了COPO的优势。
英文摘要：
Despite Multimodal Large Language Models (MLLMs) having shown impressive capabilities, they may suffer from hallucinations. Empirically, we find that MLLMs attend disproportionately to task-irrelevant background regions compared with text-only LLMs, implying spurious background-answer correlations. We claim and analyze that (i) outcome-based rewards can be an important factor leading to spurious correlations, and (ii) spurious correlations can be an important factor leading to hallucinations. Based on these results, we propose Causal-Oriented Policy Optimization (COPO) to mitigate these spurious correlations, thus addressing the issue of hallucinations. It imposes token-level sufficiency and necessity constraints to measure each inference token's causal contribution, thus ensuring correct and evidence-grounded output. Specifically, we first evaluate each token's causal contribution via a newly proposed causal completeness reward. This reward is then used to construct a causally informed advantage function within the GRPO optimization framework, encouraging the model to focus on tokens that are causally sufficient and necessary for accurate generation. Experimental results across various benchmarks demonstrate the advantages of COPO.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
