DRIFT: 基于难度路由的自我蒸馏与节奏门控探索及成功缓冲训练
DRIFT: Difficulty Routing Self-DIstillation with Rhythm-Gated Exploration and Success BuFfer Training
AI总结：
提出DRIFT框架，通过难度路由和节奏门控动态分配自蒸馏与强化学习信号，结合成功缓冲和两阶段课程学习，提升大模型在复杂推理任务中的自我进化稳定性，在五个基准上平均得分79.5%，超越GRPO和SDPO。
摘要：
使大型语言模型在没有外部专家监督的情况下实现稳定的自我改进仍然是复杂推理任务中的核心挑战。现有的自蒸馏和强化学习方法缺乏明确机制来跟踪问题级别的学习进度并相应调整优化策略。因此，训练可能过度优化简单问题，从困难问题中获得弱监督，并且未能充分探索边界情况。为了解决这些问题，我们提出了DRIFT，一种用于大型语言模型的在线自我进化策略优化框架。DRIFT通过联合使用难度路由和节奏门控来调节模型的自我改进过程。前者在问题级别识别模型的学习状态并动态分配自蒸馏和强化学习信号，而后者在令牌级别细化策略更新，将探索集中在关键推理位置。通过进一步引入成功缓冲和两阶段课程学习策略，DRIFT保留了高质量的历史经验，同时逐步引导模型从可靠的行为获取走向稳定的策略进化。在五个基准和三个模型规模上的评估中，DRIFT在所有评估指标上均超越了GRPO和SDPO的峰值性能。在五个基准的平均得分上，DRIFT达到79.5%，优于GRPO 9.5%和SDPO 7.5%，创造了新的最先进结果。值得注意的是，在ToolUse上，DRIFT达到了79.2%的准确率，比GRPO提高了13.5%，比SDPO提高了10.7%，创造了新的最先进水平，并大幅优于所有同期方法。
英文摘要：
Enabling large language models to achieve stable self-improvement without external expert supervision remains a central challenge in complex reasoning tasks. Existing self-distillation and reinforcement learning methods lack explicit mechanisms for tracking problem-level learning progress and adapting optimization strategies accordingly. Consequently, training may over-optimize easy problems, receive weak supervision from hard problems, and fail to sufficiently explore borderline cases. To resolve these issues, we propose DRIFT, an online self-evolution policy optimization framework for large language models. DRIFT regulates the model's self-improvement process through the joint use of Difficulty Routing and Rhythm Gating. The former identifies the model's learning state at the problem level and dynamically allocates self-distillation and reinforcement learning signals, while the latter refines policy updates at the token level, concentrating exploration on critical reasoning positions. By further incorporating a success buffer and a two-stage curriculum learning strategy, DRIFT preserves high-quality historical experience while progressively guiding the model from reliable behavior acquisition toward stable policy evolution. Evaluated across five benchmarks and three model scales, DRIFT surpasses the peak performance of both GRPO and SDPO across all evaluated metrics. On the average score over the five benchmarks, DRIFT achieves 79.5$\%$, outperforming GRPO by 9.5$\%$ and SDPO by 7.5$\%$, establishing a new state-of-the-art result. Notably, on ToolUse, DRIFT reaches an accuracy of 79.2$\%$, improving over GRPO by 13.5$\%$ and SDPO by 10.7$\%$, setting a new state-of-the-art and substantially outperforming all concurrent methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
