arXiv 2605.066422026-09-28cs.CLcs.AI

StraTA: 通过战略轨迹抽象激励代理强化学习

StraTA: Incentivizing Agentic Reinforcement Learning with Strategic Trajectory Abstraction

The Chinese University of Hong Kong（香港中文大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；University of Georgia（佐治亚大学）；University of Oxford（牛津大学）；Shenzhen Loop Area Institute（深圳河套学院）

Xiangyuan Xue, Yifan Zhou, Zidong Wang, Shengji Tang, Philip Torr, Wanli Ouyang, Lei Bai, Zhenfei Yin

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出StraTA框架，通过引入显式的轨迹级策略提升代理强化学习的样本效率和最终性能，实验显示其在ALFWorld、WebShop和SciWorld上均优于基线模型。

摘要：

大型语言模型（LLMs）越来越多地被用作交互代理，但优化其长期决策仍困难，因为现有方法大多纯反应式，削弱了探索和长期轨迹的信用分配。本文提出战略轨迹抽象（StraTA），一种简单的框架，通过在代理强化学习（RL）中引入显式的轨迹级策略。StraTA从初始任务状态采样紧凑策略，将后续动作基于该策略进行条件化，并通过分层GRPO式滚出设计联合训练策略生成和动作执行，进一步通过多样化策略滚出和关键自我判断增强。在ALFWorld、WebShop和SciWorld上的实验表明，StraTA在样本效率和最终性能上均优于强基线。在ALFWorld上，StraTA达到93.1%的成功率，在WebShop上达到84.2%。在SciWorld上，StraTA获得63.5%的总体分数，优于前沿闭源模型。

英文摘要：

Large language models (LLMs) are increasingly used as interactive agents, but optimizing them for long-horizon decision making remains difficult because current methods are largely purely reactive, which weakens both exploration and credit assignment over extended trajectories. In this work, we present Strategic Trajectory Abstraction (StraTA), a simple framework that introduces an explicit trajectory-level strategy into agentic reinforcement learning (RL). StraTA samples a compact strategy from the initial task state, conditions subsequent actions on that strategy, and trains strategy generation and action execution jointly with a hierarchical GRPO-style rollout design, further enhanced by diverse strategy rollout and critical self-judgment. Experiments on ALFWorld, WebShop, and SciWorld show that StraTA consistently improves both sample efficiency and final performance over strong baselines. StraTA reaches success rates of 93.1% on ALFWorld and 84.2% on WebShop. On SciWorld, StraTA attains a 63.5% overall score, outperforming frontier closed-source models.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。