Goldilocks RL: 调整任务难度以摆脱稀疏奖励进行推理
Goldilocks RL: Tuning Task Difficulty to Escape Sparse Rewards for Reasoning
AI总结：
Goldilocks RL通过预测学生模型的难度来优化数据采样，利用GRPO训练提升模型性能，克服稀疏奖励带来的样本效率低问题。
摘要：
强化学习已成为解锁语言模型推理能力的强大范式。然而，依赖稀疏奖励使这一过程高度样本不高效，因为模型必须在广阔搜索空间中以极少反馈导航。经典课程学习旨在通过按复杂度排序数据来缓解这一问题，但先前工作主要针对小数据集，无法直接迁移到现代语言模型训练的大型设置。此外，特定模型的正确排序往往不明确。为此，我们提出Goldilocks，一种新颖的教师驱动数据采样策略，旨在预测每个问题对学生模型的难度。教师模型选择适中难度的问题，即既不太简单也不太困难（Goldilocks原则），同时用GRPO训练学生模型。通过利用学生在已见样本上的表现，教师持续适应学生不断变化的能力。在OpenMathReasoning数据集上，Goldilocks数据采样在相同计算预算下提升了使用标准GRPO训练的模型性能。
英文摘要：
Reinforcement learning has emerged as a powerful paradigm for unlocking reasoning capabilities in language models. However, relying on sparse rewards makes this process highly sample-inefficient, as models must navigate vast search spaces with minimal feedback. While classic curriculum learning aims to mitigate this by ordering data based on complexity, prior works have primarily targeted small datasets and do not directly transfer to the large-scale settings typical of modern language model training. Furthermore, the right ordering for a specific model is often unclear. To address this, we propose Goldilocks, an adaptive data-selection strategy that uses a Selector network to predict the standard deviation of rewards across the model's rollouts for each candidate question. The Selector prioritizes questions with high predicted reward variability, corresponding to questions that are neither too easy nor too hard for the model's current capabilities (Goldilocks principle), while training the model with GRPO. By leveraging the model's performance on seen samples, the Selector continuously adapts to the model's evolving abilities. Across the OpenMathReasoning and Polaris datasets, Goldilocks consistently improves over standard GRPO, requiring up to 78% fewer optimization steps to reach the corresponding GRPO performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
