arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2302.01877cs.LG
收藏

AdaptDiffuser:扩散模型作为自适应自进化规划器

AdaptDiffuser: Diffusion Models as Adaptive Self-evolving Planners

The University of Hong Kong(香港大学);University of California, Berkeley(加州大学伯克利分校);Tianjin University(天津大学);Shanghai AI Laboratory(上海人工智能实验室)

Zhixuan Liang, Yao Mu, Mingyu Ding, Fei Ni, Masayoshi Tomizuka, Ping Luo

AI总结:

AdaptDiffuser是一种自适应自进化扩散规划方法,通过奖励梯度生成合成数据并用判别器筛选微调模型,提升对未见任务的泛化能力,在多个基准上超越现有方法。

摘要:

扩散模型在许多任务中展现了强大的生成能力,并有望成为离线强化学习的一种范式。然而,扩散模型的质量受到训练数据多样性不足的限制,这阻碍了规划的性能和对新任务的泛化能力。本文介绍了AdaptDiffuser,一种基于扩散的进化规划方法,它可以自我进化以改进扩散模型,从而成为更好的规划器,不仅适用于已见任务,还能适应未见任务。AdaptDiffuser利用奖励梯度的引导,能够为目标条件任务生成丰富的合成专家数据。然后,它通过判别器选择高质量数据来微调扩散模型,从而提高对未见任务的泛化能力。在两个基准环境和KUKA工业机械臂及Maze2D环境中精心设计的两个未见任务上的实证实验证明了AdaptDiffuser的有效性。例如,AdaptDiffuser不仅在Maze2D上比先前的最先进方法Diffuser高出20.8%,在MuJoCo运动上高出7.5%,而且能更好地适应新任务,例如KUKA拾取和放置任务,无需额外专家数据即可提高27.9%。更多可视化结果和演示视频可在我们的项目页面上找到。

英文摘要:

Diffusion models have demonstrated their powerful generative capability in many tasks, with great potential to serve as a paradigm for offline reinforcement learning. However, the quality of the diffusion model is limited by the insufficient diversity of training data, which hinders the performance of planning and the generalizability to new tasks. This paper introduces AdaptDiffuser, an evolutionary planning method with diffusion that can self-evolve to improve the diffusion model hence a better planner, not only for seen tasks but can also adapt to unseen tasks. AdaptDiffuser enables the generation of rich synthetic expert data for goal-conditioned tasks using guidance from reward gradients. It then selects high-quality data via a discriminator to finetune the diffusion model, which improves the generalization ability to unseen tasks. Empirical experiments on two benchmark environments and two carefully designed unseen tasks in KUKA industrial robot arm and Maze2D environments demonstrate the effectiveness of AdaptDiffuser. For example, AdaptDiffuser not only outperforms the previous art Diffuser by 20.8% on Maze2D and 7.5% on MuJoCo locomotion, but also adapts better to new tasks, e.g., KUKA pick-and-place, by 27.9% without requiring additional expert data. More visualization results and demo videos could be found on our project page.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑