用于智能强化学习的单步异步优化
Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning
AI总结：
研究针对异步强化学习中训练稳定性和离策略挑战，提出单步异步优化（SAO）。核心方法是用单步采样取代分组采样，改进单步策略并引入裁剪策略。主要贡献是SAO训练稳定，性能优于GRPO及其变体，在模拟在线学习中有效且成功部署于模型训练。
摘要：
强化学习对训练后的大语言模型愈发重要。以往大语言模型的强化学习管道多为同步和批交织，对长期智能任务效率低。异步强化学习通过随步更新模型更高效，但现有异步强化学习系统常重吞吐量，忽视训练稳定性和任务有效性。本文提出单步异步优化（SAO）解决异步强化学习中的稳定性和离策略挑战。用单步采样取代分组采样，通过实用价值模型训练设计改进单步策略，引入严格双边令牌级裁剪策略提高优化稳定性。SAO能稳定训练一千步，在智能编码和推理基准测试中优于GRPO及其变体，在模拟在线学习设置中也特别有效，还成功部署于训练开源GLM - 5.2模型的智能强化学习管道。
英文摘要：
Reinforcement learning (RL) is becoming increasingly important for post-training large language models (LLMs). Previous RL pipelines for LLMs were mostly synchronous and batch-interleaved, which is inefficient for long-horizon agentic tasks. Recently, asynchronous RL has emerged as a more efficient alternative by updating the model as rollouts arrive. However, existing asynchronous RL systems often emphasize throughput, while leaving training stability and task effectiveness largely underexplored. For example, a key challenge is that group-wise sampling in the widely-used GRPO framework does not naturally fit asynchronous agentic training. In this paper, we present Single-rollout Asynchronous Optimization (SAO) to address the stability and off-policy challenges in asynchronous RL. To reduce off-policy effects and improve generalization, we replace group-wise sampling with single-rollout sampling, that is, using one rollout per prompt. We further improve this single-rollout strategy with practical value-model training designs. To improve optimization stability, we introduce a strict double-side token-level clipping strategy. SAO is able to train stably for one thousand steps and consistently outperform GRPO and its variants on agentic coding and reasoning benchmarks, such as SWE-Bench Verified, BeyondAIME, and IMOAnswerBench. We also demonstrate that single-rollout RL is particularly effective in a simulated online learning setting, where the model must adapt to changing evolving environments. To this end, SAO is successfully deployed in the agentic RL pipeline for training the open GLM-5.2 model (750B-A40B).
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
