优先选择最佳:通过奖励超越答案正确性来激励可靠的多模态推理
Prioritizing the Best: Incentivizing Reliable Multimodal Reasoning by Rewarding Beyond Answer Correctness
AI总结:
本文提出Groupwise Ranking Reward方法,通过在单次遍历中对相同提示下的验证通过轨迹进行排名并重新分配奖励,有效缓解多模态强化学习中的推理-答案不一致问题,提升可靠性条件下的准确率。
摘要:
可验证奖励的强化学习(RLVR)通过奖励可验证的最终答案来提升多模态推理能力。然而,答案正确的轨迹仍可能依赖于不完整的推导、薄弱的证据或与结论矛盾的陈述。这种答案正确性与推理有效性之间的差距,即推理-答案不一致,推动了多模态RL中的轨迹监督。我们比较了两种主要方法:奖励模型(RMs)和生成奖励(GRs)。RMs高效且在训练初期表现良好,但其收益随着策略分布变化而减弱;GRs能提升性能,但可能产生不稳定的奖励和计算成本较高的问题。因此,我们提出组内排名奖励,通过一次遍历对相同提示下的验证通过轨迹进行排名并重新分配奖励。组内比较比GRs更有效地分离强弱正确的轨迹,且具有更低的判断开销。实验表明,RLVR加剧了推理-答案不一致,而轨迹监督可缓解此问题。组内排名奖励在整体上表现最佳,将可靠性条件下的准确率从47.4%提升至54.7%。
英文摘要:
Reinforcement Learning with Verifiable Rewards (RLVR) improves multimodal reasoning by rewarding verifiable final answers. Yet answer-correct trajectories may still rely on incomplete derivations, weak evidence, or statements that contradict their conclusions. This gap between answer correctness and reasoning validity, which we call reasoning-answer inconsistency, motivates trajectory supervision in multimodal RL. We compare two main approaches: reward models (RMs), and Generative Rewards (GRs). RMs are efficient and help early in training, but their gains weaken as the policy distribution shifts; GRs improve performance, but may give unstable rewards and computationally expensive. We therefore propose Groupwise Ranking Reward, which ranks verifier-passed trajectories for the same prompt in one pass and redistributes reward accordingly. Groupwise comparison better separates stronger and weaker correct trajectories with lower judge overhead than GRs. Experiments show that RLVR aggravates reasoning-answer inconsistency, while trajectory supervision alleviates it. Groupwise Ranking Reward performs best overall, improving reliability-conditioned accuracy from 47.4% to 54.7% over RLVR.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。