用于离线强化学习的行为正则化扩散策略优化
Behavior-Regularized Diffusion Policy Optimization for Offline Reinforcement Learning
AI总结：
本文提出BDPO框架，将行为正则化扩展至扩散策略，通过解析计算KL正则化作为逆向时间转移核的累积差异，开发双时间尺度actor-critic算法，在D4RL基准上验证了有效性与优越性能。
摘要：
行为正则化通过约束策略使其接近某个行为策略，在离线强化学习（RL）中被广泛用于管理对未见动作进行危险利用的风险。然而，现有的行为正则化RL文献主要关注显式策略参数化，如高斯策略。因此，目前尚不清楚如何将此框架扩展到更高级的策略参数化，如扩散模型。在本文中，我们引入了BDPO，这是一种为基于扩散的策略量身定制的原则性行为正则化RL框架，从而结合了扩散策略的表达能力和正则化提供的鲁棒性。我们方法的关键在于，将Kullback-Leibler（KL）正则化解析地计算为扩散轨迹上逆向时间转移核的累积差异。通过整合该正则化，我们开发了一种高效的双时间尺度actor-critic RL算法，该算法在尊重行为约束的同时产生最优策略。在合成2D任务和D4RL基准的连续控制任务上进行的综合评估验证了其有效性和优越性能。
英文摘要：
Behavior regularization, which constrains the policy to stay close to some behavior policy, is widely used in offline reinforcement learning (RL) to manage the risk of hazardous exploitation of unseen actions. Nevertheless, existing literature on behavior-regularized RL primarily focuses on explicit policy parameterizations, such as Gaussian policies. Consequently, it remains unclear how to extend this framework to more advanced policy parameterizations, such as diffusion models. In this paper, we introduce BDPO, a principled behavior-regularized RL framework tailored for diffusion-based policies, thereby combining the expressive power of diffusion policies and the robustness provided by regularization. The key ingredient of our method is to calculate the Kullback-Leibler (KL) regularization analytically as the accumulated discrepancies in reverse-time transition kernels along the diffusion trajectory. By integrating the regularization, we develop an efficient two-time-scale actor-critic RL algorithm that produces the optimal policy while respecting the behavior constraint. Comprehensive evaluations conducted on synthetic 2D tasks and continuous control tasks from the D4RL benchmark validate its effectiveness and superior performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
