用于离线强化学习的带Q-Ensembles的熵正则化扩散策略
Entropy-regularized Diffusion Policy with Q-Ensembles for Offline Reinforcement Learning
AI总结：
针对离线强化学习，提出基于均值回复SDE的熵正则化扩散策略，并利用Q-Ensembles下置信界抑制OOD价值误差，在D4RL多数任务上达到SOTA。
摘要：
本文提出了训练用于离线强化学习（RL）的扩散策略的先进技术。其核心是一个均值回复随机微分方程（SDE），该方程将复杂的动作分布转换为标准高斯分布，然后像典型的扩散策略一样，使用相应的逆时间SDE对以环境状态为条件的动作进行采样。我们证明了这种SDE具有一个可用于计算策略对数概率的解，从而产生一个熵正则化器，可改善对离线数据集的探索。为缓解来自分布外数据点的不准确价值函数的影响，我们进一步提出学习Q-Ensembles的下置信界，以实现更稳健的策略改进。通过在离线RL中将熵正则化扩散策略与Q-Ensembles相结合，我们的方法在D4RL基准测试的大多数任务上取得了最先进的性能。代码可在https://github.com/ruoqizzz/Entropy-Regularized-Diffusion-Policy-with-QEnsemble获取。
英文摘要：
This paper presents advanced techniques of training diffusion policies for offline reinforcement learning (RL). At the core is a mean-reverting stochastic differential equation (SDE) that transfers a complex action distribution into a standard Gaussian and then samples actions conditioned on the environment state with a corresponding reverse-time SDE, like a typical diffusion policy. We show that such an SDE has a solution that we can use to calculate the log probability of the policy, yielding an entropy regularizer that improves the exploration of offline datasets. To mitigate the impact of inaccurate value functions from out-of-distribution data points, we further propose to learn the lower confidence bound of Q-ensembles for more robust policy improvement. By combining the entropy-regularized diffusion policy with Q-ensembles in offline RL, our method achieves state-of-the-art performance on most tasks in D4RL benchmarks. Code is available at https://github.com/ruoqizzz/Entropy-Regularized-Diffusion-Policy-with-QEnsemble.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
