RAPO：通过检索增强策略优化扩展LLM代理的探索
RAPO: Expanding Exploration for LLM Agents via Retrieval-Augmented Policy Optimization
AI总结：
RAPO通过引入检索增强策略优化，扩展LLM代理的探索能力，提升训练效率和探索效果。
摘要：
代理强化学习（代理RL）在基于大语言模型（LLM）的代理中展现出了显著的潜力。这些工作能够通过多步骤、工具集成的推理使LLM代理解决复杂任务。然而，现有代理RL方法的一个固有局限性在于其对纯策略范式的依赖，限制了探索仅限于代理自动生成的输出，并阻止了发现新的推理视角以进一步改进。尽管最近的努力将辅助的非策略信号纳入其中以增强探索，但它们通常使用完整的非策略轨迹进行轨迹级策略估计，忽略了代理滚动中对细粒度、步骤级探索动态的必要性。在本文中，我们重新审视代理RL中的探索，并提出了检索增强策略优化（RAPO），一种新的RL框架，通过引入检索来在训练过程中显式扩展探索。为此，我们将代理RL的训练过程分解为两个阶段：（i）混合策略代理滚动，和（ii）检索感知的策略优化。具体而言，我们提出了一种混合策略代理滚动策略，使代理能够持续推理检索到的非策略步骤级轨迹。它动态扩展代理的推理接收场，使探索能够基于外部行为进行扩展。随后，我们引入了检索感知的策略优化机制，该机制通过检索奖励和重要性塑造校准策略梯度估计，稳定训练并优先考虑检索照明的探索。广泛的实验表明，RAPO在三个代理推理任务的十四个数据集上实现了+5.0%的平均增益，同时提供了1.2倍更快的训练效率。
英文摘要：
Agentic Reinforcement Learning (Agentic RL) has shown remarkable potential in large language model-based (LLM) agents. These works can empower LLM agents to tackle complex tasks via multi-step, tool-integrated reasoning. However, an inherent limitation of existing Agentic RL methods is their reliance on a pure on-policy paradigm for exploration, restricting exploration to the agent's self-generated outputs and preventing the discovery of new reasoning perspectives for further improvement. While recent efforts incorporate auxiliary off-policy signals to enhance exploration, they typically utilize full off-policy trajectories for trajectory-level policy estimation, overlooking the necessity for the fine-grained, step-level exploratory dynamics within agentic rollout. In this paper, we revisit exploration in Agentic RL and propose Retrieval-Augmented Policy Optimization (RAPO), a novel RL framework that introduces retrieval to explicitly expand exploration during training. To achieve this, we decompose the Agentic RL training process into two phases: (i) Hybrid-policy Agentic Rollout, and (ii) Retrieval-aware Policy Optimization. Specifically, we propose a Hybrid-policy Agentic Rollout strategy, which allows the agents to continuously reason over the retrieved off-policy step-level traces. It dynamically extends the reasoning receptive field of agents, enabling broader exploration conditioned on external behaviors. Subsequently, we introduce the Retrieval-aware Policy Optimization mechanism, which calibrates the policy gradient estimation with retrieval reward and importance shaping, stabilizing training and prioritizing retrieval-illuminating exploration. Extensive experiments show that RAPO achieves an +5.0% average gain on fourteen datasets across three agentic reasoning tasks, while delivering 1.2x faster training efficiency.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
