MetaDiffuser：作为离线元强化学习条件规划器的扩散模型
MetaDiffuser: Diffusion Model as Conditional Planner for Offline Meta-RL
AI总结：
本文提出MetaDiffuser，将离线元强化学习泛化问题建模为条件轨迹生成任务，通过上下文条件扩散模型和双重引导采样模块，在MuJoCo基准上超越现有离线元强化学习方法。
摘要：
近年来，扩散模型作为离线强化学习（RL）中序列建模范式的一种有前景的主干网络而崭露头角。然而，这些工作大多缺乏在奖励或动力学变化的任务之间的泛化能力。为应对这一挑战，本文提出了一种面向任务的、以条件化扩散规划器用于离线元强化学习的方法（MetaDiffuser），该方法将泛化问题视为带有上下文表示的条件轨迹生成任务。其关键在于学习一个以上下文为条件的扩散模型，该模型能够生成面向任务的轨迹，用于跨不同任务进行规划。为了增强生成轨迹的动力学一致性，同时鼓励轨迹获得高回报，我们进一步在扩散模型的采样过程中设计了一个双重引导模块。所提出的框架对来自测试任务的收集到的热启动数据质量具有鲁棒性，并且能够灵活地与不同的任务表示方法相结合。在MuJoCo基准上的实验结果表明，MetaDiffuser优于其他强离线元强化学习基线方法，展示了扩散架构出色的条件生成能力。
英文摘要：
Recently, diffusion model shines as a promising backbone for the sequence modeling paradigm in offline reinforcement learning(RL). However, these works mostly lack the generalization ability across tasks with reward or dynamics change. To tackle this challenge, in this paper we propose a task-oriented conditioned diffusion planner for offline meta-RL(MetaDiffuser), which considers the generalization problem as conditional trajectory generation task with contextual representation. The key is to learn a context conditioned diffusion model which can generate task-oriented trajectories for planning across diverse tasks. To enhance the dynamics consistency of the generated trajectories while encouraging trajectories to achieve high returns, we further design a dual-guided module in the sampling process of the diffusion model. The proposed framework enjoys the robustness to the quality of collected warm-start data from the testing task and the flexibility to incorporate with different task representation method. The experiment results on MuJoCo benchmarks show that MetaDiffuser outperforms other strong offline meta-RL baselines, demonstrating the outstanding conditional generation ability of diffusion architecture.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
