分层实体导向强化学习与分因子子目标扩散
Hierarchical Entity-centric Reinforcement Learning with Factored Subgoal Diffusion
AI总结：
本文提出分层实体导向强化学习方法，通过分因子子目标生成模型提升多实体领域中长周期任务的性能。
摘要：
我们提出了一种分层实体导向框架，用于离线目标导向强化学习（GCRL），该框架结合了子目标分解与分因子结构，以解决包含多个实体的领域中的长周期任务。在复杂环境中实现长周期目标仍然是强化学习（RL）的核心挑战。包含多个实体的领域尤其困难，因为它们具有组合复杂性。GCRL促进了跨目标的泛化和子目标结构的利用，但在高维观察和组合状态空间方面存在困难，尤其是在稀疏奖励的情况下。我们采用两级层次结构，由基于价值的GCRL智能体和分因子子目标生成条件扩散模型组成。RL智能体和子目标生成器独立训练，并通过基于价值函数的选择性子目标生成进行后组合，使该方法模块化，并与现有GCRL算法兼容。我们引入了新的变体来基准任务，突显了多实体领域的挑战，并展示了我们的方法在稀疏奖励的图像基长周期任务中，持续提升了底层RL智能体的性能，实现了在我们套件中最难任务上的成功率超过150%，并且能够泛化到增加的周期和实体数量。Rollout视频提供在：https://sites.google.com/view/hecrl
英文摘要：
We propose a hierarchical entity-centric framework for offline Goal-Conditioned Reinforcement Learning (GCRL) that combines subgoal decomposition with factored structure to solve long-horizon tasks in domains with multiple entities. Achieving long-horizon goals in complex environments remains a core challenge in Reinforcement Learning (RL). Domains with multiple entities are particularly difficult due to their combinatorial complexity. GCRL facilitates generalization across goals and the use of subgoal structure, but struggles with high-dimensional observations and combinatorial state-spaces, especially under sparse reward. We employ a two-level hierarchy composed of a value-based GCRL agent and a factored subgoal-generating conditional diffusion model. The RL agent and subgoal generator are trained independently and composed post hoc through selective subgoal generation based on the value function, making the approach modular and compatible with existing GCRL algorithms. We introduce new variations to benchmark tasks that highlight the challenges of multi-entity domains, and show that our method consistently boosts performance of the underlying RL agent on image-based long-horizon tasks with sparse rewards, achieving over 150% higher success rates on the hardest task in our suite and generalizing to increasing horizons and numbers of entities. Rollout videos are provided at: https://sites.google.com/view/hecrl
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
