arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.08489cs.LGcs.CL
收藏

超越正确性:通过迁移学习实现鲁棒推理

Beyond Correctness: Learning Robust Reasoning via Transfer

Microsoft Research(微软研究院)

Hyunseok Lee, Soheil Abbasloo, Jihoon Tack, Jinwoo Shin

AI总结:

通过迁移奖励的强化学习方法,提升大语言模型推理的鲁棒性和效率,实现更稳定和可推广的推理过程。

摘要:

可验证奖励的强化学习(RLVR)最近增强了大语言模型(LLM)的推理能力,但其专注于最终答案的正确性留有一个关键空白:它不保证推理过程本身的鲁棒性。我们采用一种简单的哲学观点,即鲁棒推理应在产生它的思维之外仍然有用,并将推理视为一种必须经受截断、重新解释和延续的含义转移形式。基于这一原则,我们引入了可迁移奖励的强化学习(RLTR),通过迁移奖励来操作鲁棒性,该奖励测试了一个模型的局部推理前缀能否引导另一个模型得到正确答案。这促使LLM生成稳定、可解释且真正可推广的推理。我们的方法在提高采样一致性的同时提高了最终答案的准确性,并在显著较少的训练步骤中达到与RLVR相当的性能。例如,在MATH500上,RLTR相比RLVR在Maj@64上实现了+3.6%的提升,并在约2.5倍更少的训练步骤中达到与RLVR相似的平均准确率,提供了更可靠的推理和显著更高效的样本使用。

英文摘要:

Reinforcement Learning with Verifiable Rewards (RLVR) has recently strengthened LLM reasoning, but its focus on final answer correctness leaves a critical gap: it does not ensure the robustness of the reasoning process itself. We adopt a simple philosophical view, robust reasoning should remain useful beyond the mind that produced it, and treat reasoning as a form of meaning transfer that must survive truncation, reinterpretation, and continuation. Building on this principle, we introduce Reinforcement Learning with Transferable Reward (RLTR), which operationalizes robustness via transfer reward that tests whether a partial reasoning prefix from one model can guide a separate model to the correct answer. This encourages LLMs to produce reasoning that is stable, interpretable, and genuinely generalizable. Our approach improves sampling consistency while improving final answer accuracy, and it reaches comparable performance in substantially fewer training steps. For example, on MATH500, RLTR achieves a +3.6%p gain in Maj@64 compared to RLVR and matches RLVR's average accuracy with roughly 2.5x fewer training steps, providing both more reliable reasoning and significantly more sample efficient.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑