arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.20532cs.LGcs.AIcs.CL
收藏

Actor-Curator: 通过策略改进带状机为RL后训练实现联合适应课程学习

Actor-Curator: Co-adaptive Curriculum Learning via Policy-Improvement Bandits for RL Post-Training

University of Illinois Chicago(伊利诺伊大学香槟分校);Caltech(加州理工学院);RPI(罗切斯特理工学院);MBZUAI(澳门大学人工智能研究院);University of Chicago(芝加哥大学);NEC Laboratories America(NEC美国实验室)

Zhengyao Gu, Jonathan Light, Raul Astudillo, Ziyu Ye, Langzhou He, Henry Peng Zou, Wei Cheng, Santiago Paternain, Philip S. Yu, Yisong Yue

AI总结:

ACTOR-CURATOR通过策略改进带状机实现RL后训练的联合适应课程学习,有效提升训练稳定性和效率。

摘要:

在后训练中使用强化学习对大规模基础模型进行训练通常依赖于大量且异质的数据集,使有效的课程学习变得至关重要且具有挑战性。在本文中,我们提出了ACTOR-CURATOR,这是一种可扩展且完全自动化的课程学习框架,用于大型语言模型(LLMs)的后训练强化学习。ACTOR-CURATOR学习一个神经 curator,通过直接优化预期策略性能提升,动态地从大型问题库中选择训练问题。我们将问题选择建模为一个非平稳随机带状问题,推导出基于在线随机镜像下降的原理性损失函数,并在部分反馈下建立后悔保证。实验证明,ACTOR-CURATOR在一系列具有挑战性的推理基准上,相对于均匀采样和强大的课程基线方法,表现优异,展示了改进的训练稳定性和效率。值得注意的是,它在AIME2024上实现了28.6%的相对增益,在ARC-1D上实现了30.5%的增益,相对于最强基线,达到80%的加速。这些结果表明,ACTOR-CURATOR是一种强大且实用的可扩展LLM后训练方法。

英文摘要:

Post-training large foundation models with reinforcement learning typically relies on massive and heterogeneous datasets, making effective curriculum learning both critical and challenging. In this work, we propose ACTOR-CURATOR, a scalable and fully automated curriculum learning framework for reinforcement learning post-training of large language models (LLMs). ACTOR-CURATOR learns a neural curator that dynamically selects training problems from large problem banks by directly optimizing for expected policy performance improvement. We formulate problem selection as a non-stationary stochastic bandit problem, derive a principled loss function based on online stochastic mirror descent, and establish regret guarantees under partial feedback. Empirically, ACTOR-CURATOR consistently outperforms uniform sampling and strong curriculum baselines across a wide range of challenging reasoning benchmarks, demonstrating improved training stability and efficiency. Notably, it achieves relative gains of 28.6% on AIME2024 and 30.5% on ARC-1D over the strongest baseline and up to 80% speedup. These results suggest that ACTOR-CURATOR is a powerful and practical approach for scalable LLM post-training.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑