arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新，休息一下吧，周末愉快~~
arXiv 2602.19362cs.LG
收藏

大语言模型可通过非策略强化学习进行推理

LLMs Can Learn to Reason Via Off-Policy RL

Cornell University（康奈尔大学）；Harvard University（哈佛大学）

Daniel Ritter, Owen Oertell, Bradley Guo, Jonathan Chang, Kianté Brantley, Wen Sun

阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出OAPL算法，通过接受非策略性解决LLM强化学习中的策略滞后问题，在竞赛数学和编码任务中表现优异，训练效率更高。

摘要：

针对大语言模型（LLM）中强化学习（RL）方法通常使用策略内算法（如PPO或GRPO）的问题，本文指出分布式训练架构导致的策略滞后以及训练与推理策略差异使数据本质上成为非策略数据。为解决此问题，先前研究多试图使非策略数据显得更像策略数据，要么通过重要采样（IS），要么通过显式修改推理引擎以更紧密地对齐训练与推理策略。本文则接受非策略性，并提出一种新的非策略RL算法：最优优势基于策略优化与滞后推理策略（OAPL）。我们证明OAPL在竞赛数学基准上优于使用重要采样的GRPO，并在LiveCodeBench上可与公开可用的编码模型DeepCoder匹配，同时训练时生成次数减少3倍。我们进一步实证表明，通过OAPL训练的模型在Pass@k指标下具有改进的测试时间扩展性。OAPL即使在训练与推理策略之间存在超过400次梯度步的滞后（比先前方法多100倍的非策略性）时仍能实现高效且有效的训练。

英文摘要：

Reinforcement learning (RL) approaches for Large Language Models (LLMs) frequently use on-policy algorithms, such as PPO or GRPO. However, policy lag from distributed training architectures and differences between the training and inference policies break this assumption, making the data off-policy by design. To rectify this, prior work has focused on making this off-policy data appear more on-policy, either via importance sampling (IS), or by more closely aligning the training and inference policies by explicitly modifying the inference engine. In this work, we embrace off-policyness and propose a novel off-policy RL algorithm that does not require these modifications: Optimal Advantage-based Policy Optimization with Lagged Inference policy (OAPL). We show that OAPL outperforms GRPO with importance sampling on competition math benchmarks, and can match the performance of a publicly available coding model, DeepCoder, on LiveCodeBench, while using 3x fewer generations during training. We further empirically demonstrate that models trained via OAPL have improved test time scaling under the Pass@k metric. OAPL allows for efficient, effective post-training even with lags of more than 400 gradient steps between the training and inference policies, 100x more off-policy than prior approaches.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑