arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.07226cs.AI
收藏

基于条件扩散模型缝合子轨迹的目标条件离线强化学习

Stitching Sub-Trajectories with Conditional Diffusion Model for Goal-Conditioned Offline RL

KAIST(韩国科学技术院)

Sungyoon Kim, Yunseon Choi, Daiki E. Matsunaga, Kee-Eung Kim

AI总结:

提出SSD方法,利用条件扩散模型生成基于目标价值和目标的规划,通过缝合次优子轨迹解决离线目标条件强化学习中的长时程规划问题,实现最先进性能。

摘要:

离线目标条件强化学习(Offline GCRL)是强化学习中的一个重要问题,专注于仅从预先收集的行为数据集中获取多样化的目标导向技能。在这种设定下,除了目标达成时,通常没有奖励反馈,这使得从有限的最优次行为数据集中学习策略变得困难。此外,现实场景涉及长时程规划,需要在子轨迹中提取有用的技能。最近,条件扩散模型已被证明是生成强化学习高质量长时程规划的有前景的方法。然而,由于这些方法做出的许多技术假设,它们在目标条件设定下的实用性仍然有限。在本文中,我们提出了SSD(Sub-trajectory Stitching with Diffusion,基于扩散的子轨迹缝合),这是一种基于模型的离线GCRL方法,利用条件扩散模型来解决这些局限性。总之,我们使用扩散模型生成以目标目标和价值为条件的未来规划,其中目标价值从目标重标记的离线数据集中估计。我们报告了在GCRL任务的标准基准集上的最先进性能,并展示了成功缝合离线数据中次优轨迹的片段以生成高质量规划的能力。

英文摘要:

Offline Goal-Conditioned Reinforcement Learning (Offline GCRL) is an important problem in RL that focuses on acquiring diverse goal-oriented skills solely from pre-collected behavior datasets. In this setting, the reward feedback is typically absent except when the goal is achieved, which makes it difficult to learn policies especially from a finite dataset of suboptimal behaviors. In addition, realistic scenarios involve long-horizon planning, which necessitates the extraction of useful skills within sub-trajectories. Recently, the conditional diffusion model has been shown to be a promising approach to generate high-quality long-horizon plans for RL. However, their practicality for the goal-conditioned setting is still limited due to a number of technical assumptions made by the methods. In this paper, we propose SSD (Sub-trajectory Stitching with Diffusion), a model-based offline GCRL method that leverages the conditional diffusion model to address these limitations. In summary, we use the diffusion model that generates future plans conditioned on the target goal and value, with the target value estimated from the goal-relabeled offline dataset. We report state-of-the-art performance in the standard benchmark set of GCRL tasks, and demonstrate the capability to successfully stitch the segments of suboptimal trajectories in the offline data to generate high-quality plans.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑