DR-APO：解耦表示优势引导的策略优化——从连续控制到具身导航
DR-APO: Decoupled-Representation Advantage-Guided Policy Optimization from Continuous Control to Embodied Navigation
AI总结：
DR-APO为高斯、扩散、流匹配策略提供统一优化原则，在MuJoCo任务中具竞争力，NavDP后训练显著提升仿真与真实导航的成功率。
摘要：
强化学习中的策略表示已从高斯策略扩展到基于扩散和流匹配的生成策略，但现有强化学习方法仍与特定表示绑定，缺乏针对新兴生成策略的统一优化接口。特别是，将基于似然的目标扩展到多步生成策略时，优化会与其内部生成过程耦合，累积的似然比会削弱更新信号。同时，生成策略专用方法常依赖多候选生成和基于评论者的选择，虽能提升整体回报，但对策略本身的改进有限。本文提出DR-APO（Decoupled-Representation Advantage-Guided Policy Optimization，解耦表示优势引导的策略优化），通过优势引导的目标改进和与表示兼容的直接动作拟合，将策略改进与策略表示解耦，为高斯策略、扩散策略和流匹配策略提供统一优化原则。在5个MuJoCo-v5任务中，DR-APO对于高斯策略与SAC相比具有竞争力，且在扩散策略和流匹配策略上均实现仅策略的最佳综合性能。NavDP的后训练在仿真中将成功率（SR）从70.33%提升至89.55%，在真实世界导航中从25%提升至65%。这些结果证明了从连续控制学习到具身策略后训练，高斯、扩散和流匹配策略间存在统一的策略改进原则。
英文摘要：
Policy representations in reinforcement learning have expanded beyond Gaussian policies to diffusion and flow-matching based generative policies. However, existing RL methods remain representation-specific, lacking a unified optimization interface for emerging generative policies. In particular, extending likelihood-based objectives to multi-step generative policies couples optimization to their internal generation process, where accumulated likelihood ratios can weaken the update signal. Meanwhile, generative-policy-specific methods often rely on multi-candidate generation and critic-based selection, improving overall return while providing limited improvement to the policy itself. We introduce DR-APO (Decoupled-Representation Advantage-Guided Policy Optimization), which decouples policy improvement from policy representation through advantage-guided target improvement and representation-compatible direct action fitting, providing a shared optimization principle for Gaussian, diffusion, and flow policies. Across five MuJoCo-v5 tasks, DR-APO remains competitive with SAC for Gaussian policies and achieves the best aggregate policy-only performance for both diffusion and flow-matching policies. NavDP post-training improves the Success Rate (SR) from 70.33% to 89.55% in simulation and from 25% to 65% in real-world navigation. These results demonstrate a unified policy-improvement principle across Gaussian, diffusion, and flow policies, from continuous-control learning to embodied-policy post-training.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
