超越轨迹级归因：基于图的智能体强化学习信用分配
Beyond Trajectory-Level Attribution: Graph-Based Credit Assignment for Agentic Reinforcement Learning
AI总结：
提出GraphGPO方法，通过构建状态转移图并利用全局信息估计各状态到任务目标的距离，实现步骤级信用分配，提升训练效率和性能。
摘要：
基于组的强化学习方法在提升大型语言模型性能方面取得了显著成功，并已迅速扩展到智能体任务。然而，其信用分配严重依赖于根据最终结果进行的粗粒度轨迹级归因，难以捕捉单个步骤的贡献，例如失败轨迹中被掩盖的有价值步骤。为了揭示潜在信息并实现更忠实的步骤级信用分配，我们提出基于图的组策略优化（GraphGPO），该方法首先将所有 rollout 轨迹聚合为一个统一的状态转移图，然后利用图中编码的全局信息估计每个状态到任务目标的距离。最后，GraphGPO 通过估计基于图的优势函数，根据转移减少到任务目标距离的程度，为每条边分配信用。通过这种方式，GraphGPO 显著提高了训练效率，并在多个具有挑战性的基准测试中取得了最先进的性能。
英文摘要：
Group-based reinforcement learning (RL) methods have achieved remarkable success in improving the performance of large language models (LLMs) and have been rapidly extended to agentic tasks. However, their credit assignment relies heavily on coarse-grained trajectory-level attribution according to final outcomes, making it difficult to capture the contribution of individual steps, such as valuable steps obscured within failed trajectories. To uncover latent information and enable more faithful step-level credit assignment, we propose Graph-based Group Policy Optimization (GraphGPO), which first aggregates all rollout trajectories into a unified state-transition graph and then estimates the distance from each state to the task goal using the global information encoded in the graph. Finally, GraphGPO assigns credit to each edge by estimating a graph-based advantage, based on how much the transition reduces the distance to the task goal. In this way, GraphGPO significantly improves training efficiency and achieves state-of-the-art performance across a range of challenging benchmarks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
