基于生成轨迹策略的离线强化学习
Offline Reinforcement Learning with Generative Trajectory Policies
AI总结：
本文提出生成轨迹策略（GTP），通过统一扩散、流匹配和一致性模型为常微分方程驱动的连续时间生成轨迹，并引入两种理论自适应方法，在D4RL基准上达到最先进性能。
摘要：
生成模型因其捕获复杂多模态行为的能力，已成为离线强化学习中一类强大的策略。然而，现有方法面临明显的权衡：扩散策略等慢速迭代模型计算成本高，而一致性策略等快速单步模型性能往往下降。在本文中，我们证明弥合这一差距是可能的。我们认为，超越个体方法局限的关键在于一个统一视角，该视角将现代生成模型（包括扩散、流匹配和一致性模型）视为学习由常微分方程驱动的连续时间生成轨迹的具体实例。这一原则性基础为强化学习中的生成策略提供了更清晰的设计空间，并使我们能够提出生成轨迹策略（GTP），一种新的、更通用的策略范式，学习底层ODE的完整解映射。为使该范式适用于离线强化学习，我们进一步引入了两种理论上原则性的自适应方法。实验结果表明，GTP在D4RL基准上达到了最先进的性能——它显著优于先前的生成策略，在多个以困难著称的AntMaze任务上取得了完美分数。
英文摘要：
Generative models have emerged as a powerful class of policies for offline reinforcement learning (RL) due to their ability to capture complex, multi-modal behaviors. However, existing methods face a stark trade-off: slow, iterative models like diffusion policies are computationally expensive, while fast, single-step models like consistency policies often suffer from degraded performance. In this paper, we demonstrate that it is possible to bridge this gap. The key to moving beyond the limitations of individual methods, we argue, lies in a unifying perspective that views modern generative models, including diffusion, flow matching, and consistency models, as specific instances of learning a continuous-time generative trajectory governed by an Ordinary Differential Equation (ODE). This principled foundation provides a clearer design space for generative policies in RL and allows us to propose Generative Trajectory Policies (GTPs), a new and more general policy paradigm that learns the entire solution map of the underlying ODE. To make this paradigm practical for offline RL, we further introduce two key theoretically principled adaptations. Empirical results demonstrate that GTP achieves state-of-the-art performance on D4RL benchmarks - it significantly outperforms prior generative policies, achieving perfect scores on several notoriously hard AntMaze tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
