arXivDaily arXiv每日学术速递 周一至周五更新
​通知:联邦节假日调休:哥伦布日 / 原住民日(Columbus Day / Indigenous Peoples' Day),今日(13日)无更新。
arXiv 2606.23206cs.CVcs.CL

CFPO:用于多模态推理的反事实策略优化

CFPO: Counterfactual Policy Optimization for Multimodal Reasoning

Beijing University of Posts and Telecommunications(北京邮电大学)

Zhangyuan Yu, Wanran Sun, Guangjing Yang, Xiaohu Wu, Qicheng Lao

AI总结:

提出反事实策略优化框架,通过跨模态反事实增强机制强制视觉与文本推理的因果一致性,显著提升多模态推理的准确性。

摘要:

大型视觉语言模型(LVLMs)在多模态推理中展现出卓越能力。然而,当前的强化学习(RL)范式缺乏明确的反事实增强和因果学习机制。这一根本缺陷导致严重的接地失败,表现为倾向于忽略视觉证据而依赖语言先验,或在长链推理过程中出现幻觉漂移。为解决这一根本原因,我们提出反事实策略优化(CFPO),一种强制视觉感知与文本推理之间因果一致性的新颖框架。CFPO引入跨模态反事实增强机制,通过最大化模型预测与抑制关键视觉线索的反事实状态预测之间的差异来正则化策略。该方法可无缝集成到GRPO和DAPO等标准算法中,无需外部奖励模型或额外监督。大量实验表明,CFPO显著提升了推理保真度,在标准RL基线上获得3.17%-6.25%的一致提升,比最先进的感知感知方法(PAPO)高出1.32%-2.13%。代码可在以下网址获取:https://this.url。

英文摘要:

Large Vision-Language Models (LVLMs) have demonstrated remarkable capabilities in multimodal reasoning. However, prevailing reinforcement learning (RL) paradigms lack explicit counterfactual enhancement and causal learning mechanisms. This fundamental deficiency results in severe grounding failures, manifesting as a tendency to ignore visual evidence in favor of language priors or exhibiting hallucination drift during long chain-of-thought reasoning. To address this root cause, we propose CounterFactual Policy Optimization (CFPO), a novel framework that enforces causal consistency between visual perception and textual reasoning. CFPO introduces a cross-modal counterfactual enhancement mechanism, which regularizes the policy by maximizing the discrepancy between the model's predictions and those from a counterfactual state where critical visual cues are suppressed. This approach seamlessly integrates with standard algorithms like GRPO and DAPO without requiring external reward models or additional supervision. Extensive experiments demonstrate that CFPO significantly improves reasoning fidelity, achieving consistent gains of 3.17%-6.25% over standard RL baselines and 1.32%-2.13% over the state-of-the-art perception-aware method (PAPO). Code is available at https://github.com/Raven-July/CFPO.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑