Diffusion Models作为离线RL中高效规划的优化器
Diffusion Models as Optimizers for Efficient Planning in Offline RL
AI总结：
针对离线强化学习中Diffusion models推理过程漫长的问题，提出Trajectory Diffuser，将采样分解为可行轨迹生成与优化，利用自回归模型加速生成并保留扩散优化，在D4RL上实现3-10倍提速且性能更优。
摘要：
Diffusion models通过将决策制定表述为序列生成，在离线强化学习任务中展现出强大的竞争力。然而，由于需要漫长的推理过程，这些方法的实用性受到限制。在本文中，我们通过将Diffusion models的采样过程分解为两个解耦的子过程来解决这一问题：1）生成可行轨迹，这是一个耗时的过程；2）优化轨迹。通过这种分解方法，我们能够部分分离效率和质量因素，使我们同时获得效率优势并确保质量保证。我们提出了Trajectory Diffuser，它利用更快的自回归模型来处理可行轨迹的生成，同时保留Diffusion models的轨迹优化过程。这使我们能够在不牺牲能力的情况下实现更高效的规划。为了评估Trajectory Diffuser的有效性和效率，我们在D4RL基准上进行了实验。结果表明，与以前的序列建模方法相比，我们的方法实现了$\it 3$-$\it 10 \times$更快的推理速度，同时在整体性能上也优于它们。
英文摘要：
Diffusion models have shown strong competitiveness in offline reinforcement learning tasks by formulating decision-making as sequential generation. However, the practicality of these methods is limited due to the lengthy inference processes they require. In this paper, we address this problem by decomposing the sampling process of diffusion models into two decoupled subprocesses: 1) generating a feasible trajectory, which is a time-consuming process, and 2) optimizing the trajectory. With this decomposition approach, we are able to partially separate efficiency and quality factors, enabling us to simultaneously gain efficiency advantages and ensure quality assurance. We propose the Trajectory Diffuser, which utilizes a faster autoregressive model to handle the generation of feasible trajectories while retaining the trajectory optimization process of diffusion models. This allows us to achieve more efficient planning without sacrificing capability. To evaluate the effectiveness and efficiency of the Trajectory Diffuser, we conduct experiments on the D4RL benchmarks. The results demonstrate that our method achieves $\it 3$-$\it 10 \times$ faster inference speed compared to previous sequence modeling methods, while also outperforming them in terms of overall performance. https://github.com/RenMing-Huang/TrajectoryDiffuser Keywords: Reinforcement Learning and Efficient Planning and Diffusion Model
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
