长周期代理任务的分层组策略优化
Hierarchy-of-Groups Policy Optimization for Long-Horizon Agentic Tasks
AI总结：
HGPO通过分层组策略优化解决长周期代理任务中分步优势估计的上下文不一致问题，提升策略优化效果。
摘要：
基于群体的强化学习（RL），例如GRPO，已经提高了大型语言模型在长周期代理任务上的能力。为了实现更细致的策略更新，近期研究越来越多地转向分步基于群体的策略优化，该方法将轨迹滚动中的每一步独立处理，同时使用记忆模块保留历史上下文。然而，我们发现估计分步相对优势时存在一个关键问题，即上下文不一致，同一组内的步骤可能在历史上下文中有所不同。实证研究表明，这个问题可能导致严重的偏差优势估计，从而显著降低策略优化效果。为了解决这个问题，本文提出了适用于长周期代理任务的分层组策略优化（HGPO）。具体而言，在轨迹组内，HGPO根据历史上下文的一致性将每个步骤分配到多个层级组中。然后，对于每个步骤，HGPO在每个组内计算不同的优势，并使用自适应加权方案进行聚合。这样，HGPO可以在分步优势估计中实现有利的偏差-方差权衡，而无需额外模型或滚动。在两个具有挑战性的代理任务ALFWorld和WebShop上，使用Qwen2.5-1.5B-Instruct和Qwen2.5-7B-Instruct进行评估，结果显示HGPO在相同计算约束下显著优于现有的代理RL方法。代码可在https://github.com/langfengQ/verl-agent/tree/master/recipe/hgpo上获得。
英文摘要：
Group-based reinforcement learning (RL), such as GRPO, has advanced the capabilities of large language models on long-horizon agentic tasks. To enable more fine-grained policy updates, recent research has increasingly shifted toward stepwise group-based policy optimization, which treats each step in a rollout trajectory independently while using a memory module to retain historical context. However, we find a key issue in estimating stepwise relative advantages, namely context inconsistency, where steps within the same group may differ in their historical contexts. Empirically, we reveal that this issue can lead to severely biased advantage estimation, thereby degrading policy optimization significantly. To address the issue, in this paper, we propose Hierarchy-of-Groups Policy Optimization (HGPO) for long-horizon agentic tasks. Specifically, within a group of rollout trajectories, HGPO assigns each step to multiple hierarchical groups according to the consistency of historical contexts. Then, for each step, HGPO computes distinct advantages within each group and aggregates them with an adaptive weighting scheme. In this way, HGPO can achieve a favorable bias-variance trade-off in stepwise advantage estimation, without extra models or rollouts. Evaluations on two challenging agentic tasks, ALFWorld and WebShop with Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct, show that HGPO significantly outperforms existing agentic RL methods under the same computational constraints. Code is available at https://github.com/langfengQ/verl-agent/tree/master/recipe/hgpo.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
