面向通用扩散规划器的任务无关预训练与任务引导微调
Task-Agnostic Pre-training and Task-Guided Fine-tuning for Versatile Diffusion Planner
AI总结:
提出两阶段框架SODP,通过任务无关次优轨迹预训练通用扩散规划器,并利用RL微调快速适应下游任务,在多任务基准上以少量数据超越现有方法。
摘要:
扩散模型已展现出在多任务轨迹建模方面的能力。然而,现有的多任务规划器或策略通常依赖通过多任务模仿学习的特定任务演示,或需要特定任务的奖励标签以通过强化学习(RL)促进策略优化。由于收集专家数据或设计奖励函数需要大量人工,这些方法成本高昂。为应对这些挑战,我们旨在开发一种通用扩散规划器,能够利用包含任务无关次优轨迹的大规模劣质数据,并具备快速适应特定任务的能力。本文提出SODP,一个利用次优数据学习扩散规划器的两阶段框架,可泛化至各种下游任务。具体而言,在预训练阶段,我们训练一个基础扩散规划器,通过建模多任务轨迹的通用分布来提取通用规划能力,这些轨迹可以是次优的且具有广泛的数据覆盖。随后针对下游任务,我们采用基于RL的微调,利用特定任务奖励快速优化扩散规划器,旨在生成具有更高特定任务回报的动作序列。来自Meta-World和Adroit等多任务领域的实验结果表明,SODP仅需少量数据用于奖励引导微调,即可超越现有最先进方法。
英文摘要:
Diffusion models have demonstrated their capabilities in modeling trajectories of multi-tasks. However, existing multi-task planners or policies typically rely on task-specific demonstrations via multi-task imitation, or require task-specific reward labels to facilitate policy optimization via Reinforcement Learning (RL). They are costly due to the substantial human efforts required to collect expert data or design reward functions. To address these challenges, we aim to develop a versatile diffusion planner capable of leveraging large-scale inferior data that contains task-agnostic sub-optimal trajectories, with the ability to fast adapt to specific tasks. In this paper, we propose SODP, a two-stage framework that leverages Sub-Optimal data to learn a Diffusion Planner, which is generalizable for various downstream tasks. Specifically, in the pre-training stage, we train a foundation diffusion planner that extracts general planning capabilities by modeling the versatile distribution of multi-task trajectories, which can be sub-optimal and has wide data coverage. Then for downstream tasks, we adopt RL-based fine-tuning with task-specific rewards to quickly refine the diffusion planner, which aims to generate action sequences with higher task-specific returns. Experimental results from multi-task domains including Meta-World and Adroit demonstrate that SODP outperforms state-of-the-art methods with only a small amount of data for reward-guided fine-tuning.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。