基于伴随匹配的可扩展最大熵强化学习扩散策略
Scalable Maximum Entropy Reinforcement Learning for Diffusion Policies via Adjoint Matching
AI总结：
提出伴随匹配方法，实现无需模拟训练的扩散策略优化，降低计算开销，使扩散策略适用于在线强化学习。
摘要：
扩散策略最近成为强化学习（RL）中表示复杂动作分布的有力范式。然而，在缺乏真实数据的情况下，标准优化技术（如分数匹配）不直接适用，扩散策略在在线RL中的应用仍受限于可扩展训练的挑战。在这项工作中，我们利用随机最优控制的最新进展，引入了一种高效优化扩散策略的算法。我们的方法基于伴随匹配，能够实现无模拟训练，避免了对显式似然估计或通过扩散过程进行昂贵反向传播的需求。此外，我们提出了几种扩展，以提高该方法在实际设置中的鲁棒性和稳定性。实验结果表明，我们的方法在显著降低计算开销的同时实现了有竞争力的性能，使扩散策略在在线RL场景中更加可行。
英文摘要：
Diffusion policies have recently emerged as a powerful paradigm for representing complex action distributions in reinforcement learning (RL). However, their application to online RL remains limited by the challenge of scalable training in the absence of ground-truth data, where standard optimization techniques such as score matching are not directly applicable. In this work, we introduce a highly efficient algorithm for optimizing diffusion policies by leveraging recent advances in stochastic optimal control. Our approach is based on adjoint matching, which enables simulation-free training and circumvents the need for explicit likelihood estimation or costly backpropagation through the diffusion process. Furthermore, we propose several extensions that improve the robustness and stability of the method in practical settings. Empirical results demonstrate that our approach achieves competitive performance while significantly reducing computational overhead, making diffusion policies more viable for online RL scenarios.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
