arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2307.04726cs.LGcs.AIcs.NEcs.RO
收藏

离线强化学习中用于分布外泛化的扩散策略

Diffusion Policies for Out-of-Distribution Generalization in Offline Reinforcement Learning

Department of Computer Engineering, Bogazici University(计算机工程系,博雅大学);SISReC, OTRI, Osaka University(SISReC、OTRI、大阪大学);Department of Computer Science, Ozyegin University(计算机科学系,奥兹格伦大学)

Suzan Ece Ada, Erhan Oztop, Emre Ugur

AI总结:

本文提出SRDP方法,在扩散策略中引入状态重建特征学习以缓解离线强化学习中的分布外泛化问题,并在机器人、仿真及D4RL基准上取得最先进结果。

摘要:

离线强化学习(RL)方法利用以往经验来学习比用于数据收集的行为策略更优的策略。然而,由于训练期间缺乏在线交互,它们在处理分布偏移时面临挑战。为此,我们提出了一种名为“面向扩散策略的状态重建”(State Reconstruction for Diffusion Policies,SRDP)的新方法,该方法在近期一类扩散策略中引入状态重建特征学习,以解决分布外(OOD)泛化问题。我们的方法促进可泛化状态表示的学习,从而缓解由OOD状态引起的分布偏移。为了说明SRDP的OOD泛化能力和更快收敛性,我们设计了一个新颖的二维多模态上下文赌博机环境,并在一个6自由度真实UR10机器人上以及仿真中实现该环境,并将其性能与先前算法进行比较。特别地,我们通过消融研究展示了所提出的状态重建的重要性。此外,我们在标准连续控制基准(D4RL)上评估了我们模型的性能,即8自由度蚂蚁的导航以及half-cheetah、hopper和walker2d的前向运动,取得了最先进的结果。最后,我们证明,在稀疏连续控制导航任务中,当状态空间的各个区域(包括包含目标的区域)从离线RL数据集中被移除时,我们的方法相比竞争基线能够实现167%的提升。

英文摘要:

Offline Reinforcement Learning (RL) methods leverage previous experiences to learn better policies than the behavior policy used for data collection. However, they face challenges handling distribution shifts due to the lack of online interaction during training. To this end, we propose a novel method named State Reconstruction for Diffusion Policies (SRDP) that incorporates state reconstruction feature learning in the recent class of diffusion policies to address the problem of out-of-distribution (OOD) generalization. Our method promotes learning of generalizable state representation to alleviate the distribution shift caused by OOD states. To illustrate the OOD generalization and faster convergence of SRDP, we design a novel 2D Multimodal Contextual Bandit environment and realize it on a 6-DoF real-world UR10 robot, as well as in simulation, and compare its performance with prior algorithms. In particular, we show the importance of the proposed state reconstruction via ablation studies. In addition, we assess the performance of our model on standard continuous control benchmarks (D4RL), namely the navigation of an 8-DoF ant and forward locomotion of half-cheetah, hopper, and walker2d, achieving state-of-the-art results. Finally, we demonstrate that our method can achieve 167% improvement over the competing baseline on a sparse continuous control navigation task where various regions of the state space are removed from the offline RL dataset, including the region encapsulating the goal.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑