结果奖励并不能保证可验证或因果重要的推理
Outcome Rewards Do Not Guarantee Verifiable or Causally Important Reasoning
AI总结：
本文通过实验发现，尽管RLVR提升了任务准确率，但并未可靠提升推理的因果重要性和充分性，通过预训练可改善此问题，辅助奖励能进一步提升推理质量。
摘要：
链式思维推理中的可验证奖励强化学习（RLVR）已成为语言模型后训练的标准部分。一个常见假设是，通过RLVR训练的推理链可靠地代表了模型得出答案的方式。本文开发了两个指标来批判性地检验这一假设：因果重要性推理（CIR），衡量推理标记对最终答案的累积影响；推理充分性（SR），衡量验证者是否能基于推理本身得出明确答案。通过Qwen2.5模型系列和ReasoningGym任务的实验，我们发现：（1）尽管RLVR提高了任务准确率，但并未可靠提高CIR或SR，对模型性能中推理的作用提出质疑；（2）在RLVR前进行少量SFT可缓解低CIR和SR；（3）无需SFT，通过在基于结果的奖励上应用辅助CIR/SR奖励可进一步提升CIR和SR。这种联合奖励在准确率上与RLVR相当，同时导致因果重要且充分的推理。这些结果表明，RLVR并不总是使模型以人们通常认为的方式依赖推理，但通过简单修改后训练流程可以解决此问题。
英文摘要：
Reinforcement Learning from Verifiable Rewards (RLVR) on chain-of-thought reasoning has become a standard part of language model post-training recipes. A common assumption is that the reasoning chains trained through RLVR reliably represent how a model gets to its answer. In this paper, we develop two metrics for critically examining this assumption: Causal Importance of Reasoning (CIR), which measures the cumulative effect of reasoning tokens on the final answer, and Sufficiency of Reasoning (SR), which measures whether a verifier can arrive at an unambiguous answer based on the reasoning alone. Through experiments with the Qwen2.5 model series and ReasoningGym tasks, we find that: (1) while RLVR does improve task accuracy, it does not reliably improve CIR or SR, calling the role of reasoning in model performance into question; (2) a small amount of SFT before RLVR can be a remedy for low CIR and SR; and (3) CIR and SR can be improved even without SFT by applying auxiliary CIR/SR rewards on top of the outcome-based reward. This joint reward matches the accuracy of RLVR while also leading to causally important and sufficient reasoning. These results show that RLVR does not always lead models to rely on reasoning in the way that is commonly thought, but this issue can be remedied with simple modifications to the post-training procedure.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
