扩散策略策略优化
Diffusion Policy Policy Optimization
AI总结：
本文提出 DPPO 框架，用强化学习策略梯度微调基于扩散的机器人控制策略，实现稳定高效探索，并在仿真与真实机械臂零样本操作中验证优势。
摘要：
我们提出 Diffusion Policy Policy Optimization（DPPO），这是一个包含最佳实践的算法框架，用于借助强化学习（RL）中的策略梯度（PG）方法，在连续控制和机器人学习任务中微调基于扩散的策略（例如 Diffusion Policy）。PG 方法在训练采用其他策略参数化形式的 RL 策略时十分普遍；然而，人们此前推测它们对于基于扩散的策略效率较低。令人惊讶的是，我们表明，在常用基准上，无论是与面向基于扩散策略的其他 RL 方法相比，还是与其他策略参数化形式的 PG 微调相比，DPPO 在微调中都取得了最强的综合性能与效率。通过实验研究，我们发现 DPPO 利用了 RL 微调与扩散参数化之间独特的协同作用，从而带来结构化且位于流形上的探索、稳定训练以及强大的策略鲁棒性。我们进一步在一系列现实设置中展示 DPPO 的优势，包括具有像素观测的仿真机器人任务，以及将仿真训练得到的策略零样本部署到机器人硬件上，执行长时序、多阶段操作任务。包含代码的网站：diffusion-ppo.github.io。
英文摘要：
We introduce Diffusion Policy Policy Optimization, DPPO, an algorithmic framework including best practices for fine-tuning diffusion-based policies (e.g. Diffusion Policy) in continuous control and robot learning tasks using the policy gradient (PG) method from reinforcement learning (RL). PG methods are ubiquitous in training RL policies with other policy parameterizations; nevertheless, they had been conjectured to be less efficient for diffusion-based policies. Surprisingly, we show that DPPO achieves the strongest overall performance and efficiency for fine-tuning in common benchmarks compared to other RL methods for diffusion-based policies and also compared to PG fine-tuning of other policy parameterizations. Through experimental investigation, we find that DPPO takes advantage of unique synergies between RL fine-tuning and the diffusion parameterization, leading to structured and on-manifold exploration, stable training, and strong policy robustness. We further demonstrate the strengths of DPPO in a range of realistic settings, including simulated robotic tasks with pixel observations, and via zero-shot deployment of simulation-trained policies on robot hardware in a long-horizon, multi-stage manipulation task. Website with code: diffusion-ppo.github.io
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
