arXiv 2405.187292024-05-29cs.LGcs.AI

面向离线强化学习的偏好动作优化扩散策略

Preferred-Action-Optimized Diffusion Policies for Offline Reinforcement Learning

JD Explore Academy（京东探索研究院）；Tongji University（同济大学）；Beijing Institute of Technology（北京理工大学）

Tianle Zhang, Jiayi Guan, Lin Zhao, Yihang Li, Dongjiang Li, Zecui Zeng, Lei Sun, Yue Chen, Xuelong Wei, Lusong Li, Xiaodong He

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对离线RL中扩散策略受限于加权回归的问题，提出偏好动作优化扩散策略，利用critic函数生成偏好动作并结合抗噪偏好优化实现策略改进，在稀疏奖励任务上表现优异。

摘要：

离线强化学习（RL）旨在从先前收集的数据集中学习最优策略。最近，由于其强大的表示能力，扩散模型作为离线RL问题的策略模型展现出了巨大的潜力。然而，以往基于扩散策略的离线RL算法普遍采用加权回归来改进策略。这种方法仅使用收集到的动作来优化策略，并且对Q值敏感，这限制了进一步提升性能的潜力。为此，我们提出了一种用于离线RL的新型偏好动作优化扩散策略。具体而言，利用一个富有表现力的条件扩散模型来表示行为策略的多样分布。同时，基于该扩散模型，通过critic函数自动生成同一行为分布内的偏好动作。此外，设计了一种抗噪偏好优化，通过使用偏好动作实现策略改进，该优化能够适应带噪声的偏好动作以实现稳定训练。大量实验表明，与以往最先进的离线RL方法相比，所提出的方法提供了具有竞争力或更优的性能，特别是在Kitchen和AntMaze等稀疏奖励任务中。另外，我们通过实验证明了抗噪偏好优化的有效性。

英文摘要：

Offline reinforcement learning (RL) aims to learn optimal policies from previously collected datasets. Recently, due to their powerful representational capabilities, diffusion models have shown significant potential as policy models for offline RL issues. However, previous offline RL algorithms based on diffusion policies generally adopt weighted regression to improve the policy. This approach optimizes the policy only using the collected actions and is sensitive to Q-values, which limits the potential for further performance enhancement. To this end, we propose a novel preferred-action-optimized diffusion policy for offline RL. In particular, an expressive conditional diffusion model is utilized to represent the diverse distribution of a behavior policy. Meanwhile, based on the diffusion model, preferred actions within the same behavior distribution are automatically generated through the critic function. Moreover, an anti-noise preference optimization is designed to achieve policy improvement by using the preferred actions, which can adapt to noise-preferred actions for stable training. Extensive experiments demonstrate that the proposed method provides competitive or superior performance compared to previous state-of-the-art offline RL methods, particularly in sparse reward tasks such as Kitchen and AntMaze. Additionally, we empirically prove the effectiveness of anti-noise preference optimization.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。