arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.16165cs.LGcs.AI
收藏

HiPER: 具有显式信用分配的分层强化学习用于大型语言模型智能体

HiPER: Hierarchical Reinforcement Learning with Explicit Credit Assignment for Large Language Model Agents

University of Minnesota;Northwestern University;Amazon AGI;Texas A\&M University;Cisco Research

Jiangweizhi Peng, Yuanxin Liu, Ruida Zhou, Charles Fleming, Zhaoran Wang, Alfredo Garcia, Mingyi Hong

AI总结:

针对稀疏奖励长程任务中LLM智能体信用分配困难的问题,提出HiPER分层规划-执行框架,通过分层优势估计(HAE)在规划和执行层面显式分配信用,在ALFWorld和WebShop上达到97.4%和83.3%的成功率。

摘要:

将LLM训练为用于多轮决策的交互式智能体仍然具有挑战性,特别是在具有稀疏和延迟奖励的长程任务中,智能体必须在获得有意义的反馈之前执行一系列扩展的动作。大多数现有的强化学习方法将LLM智能体建模为在单一时间尺度上运行的扁平策略,每轮选择一个动作。在稀疏奖励设置中,这种扁平策略必须跨整个轨迹传播信用,而没有显式的时间抽象,这常常导致不稳定的优化和低效的信用分配。我们提出HiPER,一种新颖的分层规划-执行强化学习框架,明确地将高层规划与低层执行分开。HiPER将策略分解为一个提出子目标的高层规划器和一个在多个动作步骤中执行这些子目标的低层执行器。为了将优化与此结构对齐,我们引入了一种称为分层优势估计(HAE)的关键技术,该技术在规划和执行层面仔细分配信用。通过聚合每个子目标执行过程中的回报并协调两个层面的更新,HAE提供了无偏的梯度估计器,并且与扁平广义优势估计相比,可证明地减少了方差。实验上,HiPER在具有挑战性的交互式基准测试中达到了最先进的性能,在ALFWorld上达到97.4%的成功率,在WebShop上达到83.3%的成功率(使用Qwen2.5-7B-Instruct,分别比先前最佳方法高出6.6%和8.3%),在需要多个依赖子任务的长程任务上尤其取得了巨大收益。这些结果突显了显式层次分解对于多轮LLM智能体的可扩展RL训练的重要性。

英文摘要:

Training LLMs as interactive agents for multi-turn decision-making remains challenging, particularly in long-horizon tasks with sparse and delayed rewards, where agents must execute extended sequences of actions before receiving meaningful feedback. Most existing reinforcement learning (RL) approaches model LLM agents as flat policies operating at a single time scale, selecting one action at each turn. In sparse-reward settings, such flat policies must propagate credit across the entire trajectory without explicit temporal abstraction, which often leads to unstable optimization and inefficient credit assignment. We propose HiPER, a novel Hierarchical Plan-Execute RL framework that explicitly separates high-level planning from low-level execution. HiPER factorizes the policy into a high-level planner that proposes subgoals and a low-level executor that carries them out over multiple action steps. To align optimization with this structure, we introduce a key technique called hierarchical advantage estimation (HAE), which carefully assigns credit at both the planning and execution levels. By aggregating returns over the execution of each subgoal and coordinating updates across the two levels, HAE provides an unbiased gradient estimator and provably reduces variance compared to flat generalized advantage estimation. Empirically, HiPER achieves state-of-the-art performance on challenging interactive benchmarks, reaching 97.4\% success on ALFWorld and 83.3\% on WebShop with Qwen2.5-7B-Instruct (+6.6\% and +8.3\% over the best prior method), with especially large gains on long-horizon tasks requiring multiple dependent subtasks. These results highlight the importance of explicit hierarchical decomposition for scalable RL training of multi-turn LLM agents.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑