arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.01223cs.LGcs.CL
收藏

在强化学习中学习难题问题:基于参考的微调

Learn Hard Problems During RL with Reference Guided Fine-tuning

Carnegie Mellon University(卡内基梅隆大学)

Yangzhen Wu, Shanda Li, Zixin Wen, Xin Zhou, Ameet Talwalkar, Yiming Yang, Wenhao Huang, Tianle Cai

AI总结:

ReGFT通过利用人类参考解决方案合成积极轨迹,提升RL在数学推理中的性能和训练效率。

摘要:

强化学习(RL)用于数学推理时常常面临奖励稀疏性的问题:对于难题,大语言模型无法采样任何正确的轨迹,从而阻止RL获得有意义的积极反馈。同时,问题通常会伴随人类撰写的参考解决方案(例如来自AoPS的问题),但直接在这些解决方案上进行微调没有效果,因为模型往往无法模仿超出自身推理分布的人类证明。我们引入了参考引导微调(ReGFT),一种简单而有效的方法,利用人类撰写的参考解决方案来合成难题上的积极轨迹,并在RL训练前进行训练。对于每个问题,我们为模型提供部分参考解决方案,让其生成自己的推理轨迹,确保最终的轨迹仍留在模型的推理空间中,同时受益于参考指导。在这些参考引导的轨迹上进行微调会增加可解决的问题数量,并产生在RL中获得更多积极奖励的检查点。在三个基准(AIME24、AIME25、BeyondAIME)上,ReGFT一致提高了监督准确性,加速了DAPO训练,并提高了RL的最终性能平台。我们的结果表明,ReGFT有效地克服了奖励稀疏性,并解锁了更强的基于RL的数学推理。

英文摘要:

Reinforcement learning (RL) for mathematical reasoning can suffer from reward sparsity: for challenging problems, LLM fails to sample any correct trajectories, preventing RL from receiving meaningful positive feedback. At the same time, there often exist human-written reference solutions along with the problem (e.g., problems from AoPS), but directly fine-tuning on these solutions offers no benefit because models often cannot imitate human proofs that lie outside their own reasoning distribution. We introduce Reference-Guided Fine-Tuning (ReGFT), a simple and effective method that utilizes human-written reference solutions to synthesize positive trajectories on hard problems and train on them before RL. For each problem, we provide the model with a partial reference solution and let it generate its own reasoning trace, ensuring the resulting trajectories remain in the model's reasoning space while still benefiting from reference guidance. Fine-tuning on these reference-guided trajectories increases the number of solvable problems and produces a checkpoint that receives more positive rewards during RL. Across three benchmarks (AIME24, AIME25, BeyondAIME), ReGFT consistently improves supervised accuracy, accelerates DAPO training, and raises the final performance plateau of RL. Our results show that ReGFT effectively overcomes reward sparsity and unlocks stronger RL-based mathematical reasoning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑