发表机构
Amazon（亚马逊）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本研究提出教师引导的课程学习，利用更强模型的推理轨迹解锁RLVR中不可解问题的训练价值，仅用128个问题达到16倍数据效率，并推出单调前沿课程（MFC）进一步提升性能。
AI 中文摘要
基于可验证奖励的强化学习（RLVR）在提升大型语言模型的数学推理能力方面已展现出显著成功。然而，对于超出模型当前能力的问题，其展开过程普遍失败且不产生学习信号，尽管这些问题是信息量最大的训练前沿，却在结构上被浪费了。我们证明，这些原本惰性的问题可以通过教师引导的课程学习来解锁：来自更强模型的局部推理轨迹创建了一个分级难度景观，而反向链接课程逐步撤回引导，直到模型独立解决问题。仅使用128个不可解问题进行训练，在九个基准的平均表现上，对两种基础模型均达到或超过了在完整2000个问题语料库上训练的GRPO（约16倍数据效率），同时大幅扩展了以大规模k下pass@k衡量的推理边界。此外，我们识别出在仅不可解问题训练机制中尤为严重的分布偏移成本，并提出了单调前沿课程（MFC）方法，该方法单调地推动训练朝向无引导求解，持续优于现有课程学习方法。
英文摘要
Reinforcement Learning with Verifiable Rewards (RLVR) has shown remarkable success in improving the mathematical reasoning of large language models. Yet problems beyond the model's current capability, where rollouts uniformly fail and no learning signal is produced, are structurally wasted despite marking the most informative training frontier. We show that these otherwise-inert problems can be unlocked via teacher-guided curriculum learning: partial reasoning traces from a stronger model create a graded difficulty landscape, and a backward-chaining curriculum progressively withdraws guidance until the model solves problems unaided. Training on only 128 unsolvable problems matches or exceeds GRPO trained on a full 2,000-problem corpus (~16x data efficiency) on the nine-benchmark average for both base models, while substantially expanding the reasoning boundary measured by pass@k at large k. Furthermore, we identify a distribution-shift cost that is particularly acute in the unsolvable-only regime and propose Monotone Frontier Curriculum (MFC), a method that monotonically drives training toward unguided solving, consistently outperforming existing curriculum methods.
Comments20 pages, 9 figures. Accepted to Findings of the Association for Computational Linguistics: EMNLP 2026
