先验引导的扩散规划用于离线强化学习
Prior-Guided Diffusion Planning for Offline Reinforcement Learning
AI总结：
针对离线强化学习中扩散规划器引导采样存在的次优、分布漂移和推理成本高问题，提出先验引导框架，用可学习分布替换高斯先验，直接生成高价值轨迹，在长视界基准上超越现有方法。
摘要：
扩散模型最近在离线强化学习中因其能够从静态数据集有效学习高性能、可泛化的策略而受到关注。基于扩散的规划器通过迭代去噪，在回报最大化目标的引导下生成高质量轨迹，从而促进长视界决策。然而，现有的引导采样策略，如分类器引导、无分类器引导和蒙特卡洛样本选择，要么产生次优的多模态动作，要么面临分布漂移问题，要么产生高昂的推理时间成本。为解决这些挑战，我们提出先验引导（PG），一种新颖的引导采样框架，它用通过行为正则化目标优化的可学习分布替换行为克隆扩散模型的标准高斯先验。PG直接生成高价值轨迹，无需对扩散模型本身进行昂贵的回报优化，并消除了推理时采样多个候选以进行样本选择的需要。我们提出了一种在潜在空间中应用行为正则化的高效训练策略，并实证表明PG在多种长视界离线RL基准上优于最先进的扩散策略和规划器。我们的代码可在 https://github.com/ku-dmlab/PG 获取。
英文摘要：
Diffusion models have recently gained prominence in offline reinforcement learning due to their ability to effectively learn high-performing, generalizable policies from static datasets. Diffusion-based planners facilitate long-horizon decision-making by generating high-quality trajectories through iterative denoising, guided by return-maximizing objectives. However, existing guided sampling strategies such as Classifier Guidance, Classifier-Free Guidance, and Monte Carlo Sample Selection either produce suboptimal multi-modal actions, struggle with distributional drift, or incur prohibitive inference-time costs. To address these challenges, we propose Prior Guidance (PG), a novel guided sampling framework that replaces the standard Gaussian prior of a behavior-cloned diffusion model with a learnable distribution, optimized via a behavior-regularized objective. PG directly generates high-value trajectories without costly reward optimization of the diffusion model itself, and eliminates the need to sample multiple candidates at inference for sample selection. We present an efficient training strategy that applies behavior regularization in latent space, and empirically demonstrate that PG outperforms state-of-the-art diffusion policies and planners across diverse long-horizon offline RL benchmarks.Our code is available at https://github.com/ku-dmlab/PG.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
