基于漂移的策略优化:面向在线机器人控制的原生一步生成策略
Drift-Based Policy Optimization: Native One-Step Policy Learning for Online Robot Control
AI总结:
本文提出一种原生一步生成策略框架,通过将迭代细化移至训练阶段,提升在线机器人控制的效率与稳定性,实验显示其在推理速度和性能上优于多步扩散策略。
摘要:
尽管多步生成策略在机器人操作中表现出色,但其在推理时需要多步迭代去噪,导致计算成本高。为此,本文提出一种两阶段框架,通过引入基于漂移的策略(DBP)和基于漂移的策略优化(DBPO),将迭代细化内化到模型参数中,实现原生一步生成策略,同时保持多模态动作建模能力。DBPO框架通过预训练的骨干网络和兼容的随机接口,实现在线强化学习中的稳定更新。实验表明,该框架在离线模仿学习、在线微调和现实世界控制中均表现出色,推理速度提升达100倍,且在挑战性操作基准上优于现有一步基线。
英文摘要:
Diffusion policies effectively model multimodal action distributions for robotic manipulation, but their iterative denoising requires tens to hundreds of network function evaluations (NFEs) for each control prediction, limiting their applicability to high-frequency closed-loop control and online reinforcement learning (RL). We present a two-stage framework for native one-step generative policies that transfers iterative refinement from inference to training. First, Drift-Based Policy (DBP) uses a fixed-point drifting objective to internalize corrective dynamics into the model parameters, producing multimodal action chunks with a single network evaluation by design. Second, Drift-Based Policy Optimization (DBPO) augments the pretrained backbone with a stochastic interface that provides exact conditional rollout likelihoods for PPO-style on-policy updates while preserving 1-NFE deployment. On the 12-task Diffusion Policy suite, DBP improves the average success rate from 0.79 to 0.83 while reducing inference from 100 NFEs to 1. Across 37 point-cloud manipulation tasks, DBP achieves an average success rate of 88.4%, surpassing the leading 1-NFE baseline OMP at 82.3%. DBPO further improves pretrained one-step policies through stable online fine-tuning on RoboMimic and D4RL. On a physical dual-arm UR5 platform, DBP achieves 123/150 successes (82%) with an average end-to-end latency of 9.5 ms, compared with MP1's 89/150 successes (59%) under the same setup. Code is available at https://github.com/YuxuanGao0822/DBPO.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。