arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2305.18459cs.LGcs.AI
收藏

扩散模型是多任务强化学习的有效规划器和数据合成器

Diffusion Model is an Effective Planner and Data Synthesizer for Multi-Task Reinforcement Learning

Shanghai Jiao Tong University(上海交通大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Fudan University(复旦大学);Yale University(耶鲁大学);Northwestern Polytechnical University(西北工业大学)

Haoran He, Chenjia Bai, Kang Xu, Zhuoran Yang, Weinan Zhang, Dong Wang, Bin Zhao, Xuelong Li

AI总结:

本研究提出多任务扩散模型(MTDiff),结合Transformer和提示学习,用于多任务离线强化学习的生成式规划与数据合成,在Meta-World和Maze2D上超越现有方法,并能基于单演示生成高质量数据以增强低质量数据集。

摘要:

扩散模型在视觉和自然语言处理领域展现了高度表达性的生成能力。最近在强化学习(RL)中的研究表明,扩散模型在建模离线数据集中的复杂策略或轨迹方面也很强大。然而,这些工作仅限于单任务设置,缺乏能够应对多任务困境的通用智能体。在本文中,我们旨在研究单个扩散模型在建模大规模多任务离线数据方面的有效性,这由于多样且多模态的数据分布而可能具有挑战性。具体来说,我们提出了多任务扩散模型(MTDiff),一种基于扩散的方法,结合了Transformer骨干和提示学习,用于多任务离线设置中的生成式规划与数据合成。MTDiff利用了多任务数据中可获取的大量知识,并在任务间进行隐式知识共享。对于生成式规划,我们发现MTDiff在Meta-World的50个任务和Maze2D的8张地图上优于最先进的算法。对于数据合成,MTDiff以单个演示作为提示,为测试任务生成高质量数据,这增强了即使是未见过的任务的低质量数据集。

英文摘要:

Diffusion models have demonstrated highly-expressive generative capabilities in vision and NLP. Recent studies in reinforcement learning (RL) have shown that diffusion models are also powerful in modeling complex policies or trajectories in offline datasets. However, these works have been limited to single-task settings where a generalist agent capable of addressing multi-task predicaments is absent. In this paper, we aim to investigate the effectiveness of a single diffusion model in modeling large-scale multi-task offline data, which can be challenging due to diverse and multimodal data distribution. Specifically, we propose Multi-Task Diffusion Model (\textsc{MTDiff}), a diffusion-based method that incorporates Transformer backbones and prompt learning for generative planning and data synthesis in multi-task offline settings. \textsc{MTDiff} leverages vast amounts of knowledge available in multi-task data and performs implicit knowledge sharing among tasks. For generative planning, we find \textsc{MTDiff} outperforms state-of-the-art algorithms across 50 tasks on Meta-World and 8 maps on Maze2D. For data synthesis, \textsc{MTDiff} generates high-quality data for testing tasks given a single demonstration as a prompt, which enhances the low-quality datasets for even unseen tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑