策略引导扩散
Policy-Guided Diffusion
AI总结：
本文提出策略引导扩散，用扩散模型生成行为分布下的完整轨迹并由目标策略引导，以更低动力学误差产生高目标概率的合成经验，显著提升多种离线强化学习算法性能。
摘要：
在许多现实场景中，智能体必须从由某种先前行为策略收集的离线数据集中学习。这种设置自然会导致行为策略与正在训练的目标策略之间出现分布偏移，因此需要策略保守性以避免不稳定性和高估偏差。自回归世界模型通过生成合成的在策略经验，为该问题提供了一种不同的解决方案。然而在实践中，模型推演必须被严重截断，以避免误差累积。作为替代方案，我们提出了策略引导扩散。我们的方法使用扩散模型在行为分布下生成完整轨迹，并应用来自目标策略的引导，使合成经验进一步趋向在策略。我们表明，策略引导扩散对目标分布的一种正则化形式进行建模，该形式平衡目标策略和行为策略下的动作似然，从而产生具有高目标策略概率的合理轨迹，同时保持比离线世界模型基线更低的动力学误差。通过将策略引导扩散产生的合成经验作为真实数据的即插即用替代物，我们证明了在一系列标准离线强化学习算法和环境中的显著性能提升。我们的方法为自回归离线世界模型提供了一种有效的替代方案，为可控生成合成训练数据打开了大门。
英文摘要：
In many real-world settings, agents must learn from an offline dataset gathered by some prior behavior policy. Such a setting naturally leads to distribution shift between the behavior policy and the target policy being trained - requiring policy conservatism to avoid instability and overestimation bias. Autoregressive world models offer a different solution to this by generating synthetic, on-policy experience. However, in practice, model rollouts must be severely truncated to avoid compounding error. As an alternative, we propose policy-guided diffusion. Our method uses diffusion models to generate entire trajectories under the behavior distribution, applying guidance from the target policy to move synthetic experience further on-policy. We show that policy-guided diffusion models a regularized form of the target distribution that balances action likelihood under both the target and behavior policies, leading to plausible trajectories with high target policy probability, while retaining a lower dynamics error than an offline world model baseline. Using synthetic experience from policy-guided diffusion as a drop-in substitute for real data, we demonstrate significant improvements in performance across a range of standard offline reinforcement learning algorithms and environments. Our approach provides an effective alternative to autoregressive offline world models, opening the door to the controllable generation of synthetic training data.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
