离线目标条件强化学习(Offline GCRL)是强化学习中的一个重要问题,专注于仅从预先收集的行为数据集中获取多样化的目标导向技能。在这种设定下,除了目标达成时,通常没有奖励反馈,这使得从有限的最优次行为数据集中学习策略变得困难。此外,现实场景涉及长时程规划,需要在子轨迹中提取有用的技能。最近,条件扩散模型已被证明是生成强化学习高质量长时程规划的有前景的方法。然而,由于这些方法做出的许多技术假设,它们在目标条件设定下的实用性仍然有限。在本文中,我们提出了SSD(Sub-trajectory Stitching with Diffusion,基于扩散的子轨迹缝合),这是一种基于模型的离线GCRL方法,利用条件扩散模型来解决这些局限性。总之,我们使用扩散模型生成以目标目标和价值为条件的未来规划,其中目标价值从目标重标记的离线数据集中估计。我们报告了在GCRL任务的标准基准集上的最先进性能,并展示了成功缝合离线数据中次优轨迹的片段以生成高质量规划的能力。
英文摘要:
Offline Goal-Conditioned Reinforcement Learning (Offline GCRL) is an important problem in RL that focuses on acquiring diverse goal-oriented skills solely from pre-collected behavior datasets. In this setting, the reward feedback is typically absent except when the goal is achieved, which makes it difficult to learn policies especially from a finite dataset of suboptimal behaviors. In addition, realistic scenarios involve long-horizon planning, which necessitates the extraction of useful skills within sub-trajectories. Recently, the conditional diffusion model has been shown to be a promising approach to generate high-quality long-horizon plans for RL. However, their practicality for the goal-conditioned setting is still limited due to a number of technical assumptions made by the methods. In this paper, we propose SSD (Sub-trajectory Stitching with Diffusion), a model-based offline GCRL method that leverages the conditional diffusion model to address these limitations. In summary, we use the diffusion model that generates future plans conditioned on the target goal and value, with the target value estimated from the goal-relabeled offline dataset. We report state-of-the-art performance in the standard benchmark set of GCRL tasks, and demonstrate the capability to successfully stitch the segments of suboptimal trajectories in the offline data to generate high-quality plans.
Graph-Assisted Stitching for Offline Hierarchical Reinforcement Learning
Department of Computer Science and Engineering, Sungkyunkwan University, Suwon, Republic of Korea(计算机科学与工程系,成均馆大学,苏旺,韩国);Department of Artificial Intelligence, Sungkyunkwan University, Suwon, Republic of Korea(人工智能系,成均馆大学,苏旺,韩国)
Free from Bellman Completeness: Trajectory Stitching via Model-based Return-conditioned Supervised Learning
Institute for Interdisciplinary Information Sciences, Tsinghua University(清华大学交叉信息研究院);Paul G. Allen School of Computer Science and Engineering, University of Washington(华盛顿大学保罗·G·艾伦计算机科学与工程学院)
GOPlan: Goal-conditioned Offline Reinforcement Learning by Planning with Learned Models
University of Warwick(华威大学);Hong Kong University of Science and Technology(香港科技大学);Tsinghua University(清华大学);University of Cambridge(剑桥大学);University of Liverpool(利物浦大学)
Chain-of-Goals Hierarchical Policy for Long-Horizon Offline Goal-Conditioned RL
Department of Electrical and Computer Engineering, Seoul National University, Seoul, South Korea(首尔国立大学电气与计算机工程系);Department of Automotive Engineering, Ajou University, Gyeonggi-do, South Korea(全州大学汽车工程系)
Option-aware Temporally Abstracted Value for Offline Goal-Conditioned Reinforcement Learning
Department of Electrical and Computer Engineering (ECE), Seoul National University(电子与计算机工程系,首尔国立大学);Interdisciplinary Program in Artificial Intelligence (IPAI), Seoul National University(人工智能跨学科项目,首尔国立大学);ASRI / INMC, Seoul National University(ASRI/INMC,首尔国立大学)
Robust Offline Imitation Learning Through State-level Trajectory Stitching
National Key Lab of Autonomous Intelligent Unmanned Systems, Beijing Institute of Technology, Beijing 100081, China(自主智能无人系统国家重点实验室,北京理工大学,北京100081,中国);Department of Control Science and Engineering, Harbin Institute of Technology, Harbin 150001, China(控制科学与工程学院,哈尔滨工业大学,哈尔滨150001,中国)
Robo-ValueRL: Reliable Value Estimation for Offline-to-Online Reinforcement Learning
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);Beijing Innovation Center of Humanoid Robotics(北京人形机器人创新中心);Beijing Forestry University(北京林业大学);Peking University(北京大学);Microsoft Research(微软研究院)
Learning Multiple Timescales for Goal-Conditioned Reinforcement Learning
Universidade Federal de Minas Gerais(米纳斯吉拉斯联邦大学);University of Alberta(阿尔伯塔大学);Alberta Machine Intelligence Institute (Amii)(阿尔伯塔机器智能研究所);Canada CIFAR AI Chair(加拿大CIFAR AI主席)
ACT: Empowering Decision Transformer with Dynamic Programming via Advantage Conditioning
National Key Laboratory for Novel Software Technology, Nanjing University(南京大学计算机软件新技术全国重点实验室);School of Artificial Intelligence, Nanjing University(南京大学人工智能学院)
DAWM: Diffusion Action World Models for Offline Reinforcement Learning via Action-Inferred Transitions
Department of Computer Science, University of Munich, Munich, Germany(慕尼黑大学计算机科学系);Munich Center for Machine Learning (MCML), Munich, Germany(慕尼黑机器学习中心(MCML))