CORA: 通过一致性导向的推理对齐分析与弥合多模态RLVR中的思考-答案差距
CORA: Analyzing and bridging thinking-answer gap in Multimodal RLVR via Consistency-Oriented Reasoning Alignment
AI总结：
本文分析多模态RLVR中思考与答案的语义不一致问题，提出CORA方法，通过轻量级一致性奖励模型引入语义一致性，并采用混合奖励优势分裂稳定优化，提升推理忠实度。
摘要：
可验证奖励的强化学习（RLVR）已成功激发大语言模型的推理能力，推动其向多模态场景扩展。现有方法主要关注提升推理轨迹的视觉覆盖和缓解视觉幻觉，但低估了推理过程与最终答案之间的语义不一致性。本文深入研究了大型视觉语言模型（LVLMs）中RLVR的思考-答案不一致性，通过对组相对策略优化（GRPO）训练过程中收集的轨迹以及RLVR后评估输出的分析，表明该问题在训练期间持续存在，并在推理时仍然存在。受此分析启发，我们提出一致性导向的推理对齐（CORA），通过轻量级即插即用的一致性奖励模型将思考-答案语义一致性引入RLVR，并进一步结合混合奖励优势分裂（HRAS）以稳定协调任务和一致性优化。在代表性多模态推理基准和主流LVLMs上的大量实验表明，CORA在提升任务性能的同时有效缓解了思考-答案不一致性，从而产生更忠实的推理轨迹。
英文摘要：
Reinforcement learning with verifiable rewards (RLVR) has successfully elicited the reasoning capabilities of large language models, motivating its extension to multimodal scenarios. Existing methods primarily focus on improving the visual coverage of reasoning traces and mitigating visual hallucinations, but underestimate the semantic inconsistency between the reasoning process and the final answer. In this paper, we delve into thinking-answer inconsistency in RLVR for large vision-language models (LVLMs), showing thorough analyses of rollouts collected throughout Group Relative Policy Optimization (GRPO) training process and post-RLVR evaluation outputs that this issue persists during training and remains present during inference. Motivated by the analysis, we propose Consistency-Oriented Reasoning Alignment (CORA), which introduces thinking-answer semantic consistency into RLVR through a lightweight plug-and-play consistency reward model, and further incorporates Hybrid Reward Advantage Splitting (HRAS) to stably coordinate task and consistency optimization. Extensive experiments across representative multimodal reasoning benchmarks and mainstream LVLMs show that CORA improves task performance while effectively mitigating thinking-answer inconsistency, leading to more faithful reasoning traces.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
