arXivDaily arXiv每日学术速递 周一至周五更新
​通知：联邦节假日调休：哥伦布日 / 原住民日（Columbus Day / Indigenous Peoples' Day），今日（13日）无更新。
arXiv 2610.13438cs.AI

推理时间干预：解锁超越策略极限的智能体强化学习

Rollout-Time Intervention: Unlocking Agentic Reinforcement Learning Beyond Policy Limits

Alibaba Group（阿里巴巴集团）；Alibaba Token Hub（阿里巴巴令牌中心）；HKUST (Guangzhou)（香港科技大学（广州））；Amap（高德地图）

Chen Xi, Ding Ruixue, Wang Zhigaoyuan, Zhang Fanrui, Zhang Qiang, Chen Boli, Wang Shihang, Wang Tianfu, Ren Ailin, Wang Pengyong, Xie Pengjun, Xiong Hui

阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

该研究针对智能体强化学习的同策略推理瓶颈，提出Rollout-Time Intervention技术，经五基准实验验证，其性能优于RL基线，可使策略超越原始能力边界。

摘要：

强化学习（RL）已成为改进大语言模型（LLM）智能体的关键范式，可在交互环境中实现更强的推理、规划和工具使用能力。然而，现有的智能体RL方法仍受限于根本的同策略推理瓶颈：当任务超出模型当前能力边界时，有效轨迹会变得极其稀疏，训练常出现崩溃。我们提出推理时间干预（Rollout-Time Intervention，RTI），一种旨在引出超出策略固有探索能力的成功轨迹的技术。RTI通过在推理过程中利用奖励侧证据（即评估标准），将错误轨迹转化为成功轨迹，为策略极少自行解决的任务提供有效学习信号。尽管干预轨迹本质上是异策略的，但我们发现当过滤掉分布偏移过大的轨迹时，它们能有效引导策略优化。在五个基准上的大量实验表明，RTI始终优于RL基线，使后训练策略的性能接近更强的当代LLM，更重要的是，能超越其原始能力边界。代码可在该https URL获取。

英文摘要：

Reinforcement learning (RL) has emerged as a key paradigm for improving large language model (LLM) agents, enabling stronger reasoning, planning, and tool-use in interactive environments. However, existing agent RL methods remain fundamentally constrained by an on-policy reasoning bottleneck: when tasks exceed the model's current capability boundary, effective trajectories become extremely sparse and training often collapses. We introduce Rollout-Time Intervention (RTI), a technique designed to elicit successful trajectories beyond the policy's intrinsic exploration capacity. RTI operates by leveraging reward-side evidence (i.e., evaluation rubrics) during rollout to convert erroneous trajectories into successful ones, providing effective learning signals for tasks that the policy rarely solves on its own. Although intervention trajectories are inherently off-policy, we find that they can effectively guide policy optimization when trajectories with excessive distributional shifts are filtered out. Extensive experiments across five benchmarks demonstrate that RTI consistently outperforms RL baselines, enabling the post-trained policy to approach the performance of substantially stronger contemporary LLMs and, more importantly, transcend its original capability boundary. The code is available at https://github.com/Alibaba-NLP/qqr.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑