Training LLMs as interactive agents for multi-turn decision-making remains challenging, particularly in long-horizon tasks with sparse and delayed rewards, where agents must execute extended sequences of actions before receiving meaningful feedback. Most existing reinforcement learning (RL) approaches model LLM agents as flat policies operating at a single time scale, selecting one action at each turn. In sparse-reward settings, such flat policies must propagate credit across the entire trajectory without explicit temporal abstraction, which often leads to unstable optimization and inefficient credit assignment.
We propose HiPER, a novel Hierarchical Plan-Execute RL framework that explicitly separates high-level planning from low-level execution. HiPER factorizes the policy into a high-level planner that proposes subgoals and a low-level executor that carries them out over multiple action steps. To align optimization with this structure, we introduce a key technique called hierarchical advantage estimation (HAE), which carefully assigns credit at both the planning and execution levels. By aggregating returns over the execution of each subgoal and coordinating updates across the two levels, HAE provides an unbiased gradient estimator and provably reduces variance compared to flat generalized advantage estimation.
Empirically, HiPER achieves state-of-the-art performance on challenging interactive benchmarks, reaching 97.4\% success on ALFWorld and 83.3\% on WebShop with Qwen2.5-7B-Instruct (+6.6\% and +8.3\% over the best prior method), with especially large gains on long-horizon tasks requiring multiple dependent subtasks. These results highlight the importance of explicit hierarchical decomposition for scalable RL training of multi-turn LLM agents.
Towards Practical Credit Assignment for Deep Reinforcement Learning
Yandex(俄罗斯搜索引擎公司(或:雅虎迪));Samsung AI Center NYC(三星人工智能中心纽约分部);Artificial Intelligence Research Institute (AIRI)(人工智能研究所(AIRI));National Research University Higher School of Economics(高等经济学院)
RACA: Relation-Aware Credit Assignment for Ad-Hoc Cooperation in Multi-Agent Deep Reinforcement Learning
University of Chinese Academy of Sciences(中国科学院大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);CAS Center for Excellence in Brain Science and Intelligence Technology(中国科学院脑科学与智能技术卓越创新中心)
Sequence Compression Speeds Up Credit Assignment in Reinforcement Learning
The Swiss AI Lab IDSIA, USI & SUPSI(瑞士IDSIA人工智能实验室、瑞士意大利语区大学、瑞士南部应用科学大学);University of Alberta(阿尔伯塔大学);Alberta Machine Intelligence Institute(阿尔伯塔机器智能研究所);AI Initiative, King Abdullah University of Science and Technology(阿卜杜拉国王科技大学人工智能计划)
Stop Summation: Min-Form Credit Assignment Is All Process Reward Model Needs for Reasoning
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences(多模态人工智能系统国家重点实验室,自动化研究所,中国科学院);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Tencent(腾讯);Faculty of Innovation Engineering, Macau University of Science and Technology(澳门科技大学创新工程学院);DeSci Center of Parallel Intelligence, Obuda University(并行智能德Sci中心,奥布达大学);Research Center for Chinese Economics and Social Security, University of Chinese Academy of Sciences(中国经济社会安全研究中心,中国科学院大学);Institute of Automation, Chinese Academy of Sciences(自动化研究所,中国科学院)
Would I have gotten that reward? Long-term credit assignment by counterfactual contribution analysis
Google DeepMind(谷歌DeepMind);Department of Computer Science, ETH Zürich(苏黎世联邦理工学院计算机科学系);Princeton Neuroscience Institute, Princeton University(普林斯顿大学普林斯顿神经科学研究所);Department of Psychology, Princeton University(普林斯顿大学心理学系)
Cooperative Game-Theoretic Credit Assignment for Multi-Agent Policy Gradients via the Core
Key Laboratory of Intelligence, Games and Information Processing in Shaanxi Province(陕西省智能信息处理重点实验室);Unmanned System Research Institute(无人系统研究院);Northwestern Polytechnical University(西北工业大学);School of Mathematics and Statistics(数学与统计学院);Information Science Academy(信息科学学院);China Electronics Technology Group Corporation(中国电子科技集团有限公司)
MACCA: Offline Multi-agent Reinforcement Learning with Causal Credit Assignment
King’s College London(伦敦国王学院);Eindhoven University of Technology(埃因霍温理工大学);University of Liverpool(利物浦大学);University of California San Diego(加州大学圣地亚哥分校)