arXiv 2603.213832026-03-22cs.AI

PivotRL: 高精度代理后训练的低计算成本

PivotRL: High Accuracy Agentic Post-Training at Low Compute Cost

Department of XXX, University of YYY, Location, Country（XXX系，YYY大学，地点，国家）；School of ZZZ, Institute of WWW, Location, Country（ZZZ学院，WWW研究所，地点，国家）

Junkeun Yi, Damon Mosk-Aoyama, Baihe Huang, Ritu Gala, Charles Wang, Sugam Dipak Devare, Khushi Bhardwaj, Abhibha Gupta, Oleksii Kuchaiev, Jiantao Jiao, Jian Zh… 展开作者

Junkeun Yi, Damon Mosk-Aoyama, Baihe Huang, Ritu Gala, Charles Wang, Sugam Dipak Devare, Khushi Bhardwaj, Abhibha Gupta, Oleksii Kuchaiev, Jiantao Jiao, Jian Zhang, Venkat Srinivasan

备注：: 22 pages, 5 figures, 6 tables

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

PivotRL结合监督微调的高效计算与端到端强化学习的泛化能力，通过局部策略滚动和奖励机制提升代理后训练的准确性和效率。

摘要：

长期代理任务的后训练面临计算效率与泛化能力的矛盾。尽管监督微调（SFT）计算高效，但常出现领域外（OOD）退化。相反，端到端强化学习（E2E RL）保留OOD能力，但因多次策略滚动导致计算成本高。我们引入PivotRL框架，利用现有SFT轨迹结合SFT的计算效率与E2E RL的OOD准确性。PivotRL通过局部策略滚动筛选信息性中间步骤，以及利用奖励机制匹配功能等价动作而非严格匹配SFT数据。理论证明这些机制促进强学习信号，同时保留与训练任务无关的动作概率顺序。与标准SFT相比，PivotRL在四个代理领域平均提升4.17%的领域内准确率，在非代理任务中提升10.04%的OOD准确率。值得注意的是，在代理编码任务中，PivotRL以4倍更少的滚动步骤实现与E2E RL相当的准确率。PivotRL被NVIDIA的Nemotron-3-Super-120B-A12B采用，作为大规模代理后训练的核心工具。

英文摘要：

Post-training for long-horizon agentic tasks has a tension between compute efficiency and generalization. While supervised fine-tuning (SFT) is compute efficient, it often suffers from out-of-domain (OOD) degradation. Conversely, end-to-end reinforcement learning (E2E RL) preserves OOD capabilities, but incurs high compute costs due to many turns of on-policy rollout. We introduce PivotRL, a novel framework that operates on existing SFT trajectories to combine the compute efficiency of SFT with the OOD accuracy of E2E RL. PivotRL relies on two key mechanisms: first, it executes local, on-policy rollouts and filters for pivots: informative intermediate turns where sampled actions exhibit high variance in outcomes; second, it utilizes rewards for functional-equivalent actions rather than demanding strict string matching with the SFT data demonstration. We theoretically show that these mechanisms incentivize strong learning signals with high natural gradient norm, while maximally preserving policy probability ordering on actions unrelated to training tasks. In comparison to standard SFT on identical data, we demonstrate that PivotRL achieves +4.17% higher in-domain accuracy on average across four agentic domains, and +10.04% higher OOD accuracy in non-agentic tasks. Notably, on agentic coding tasks, PivotRL achieves competitive accuracy with E2E RL with 4x fewer rollout turns. PivotRL is adopted by NVIDIA's Nemotron-3-Super-120B-A12B, acting as the workhorse in production-scale agentic post-training.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。