CDSA：用于离线强化学习的保守去噪评分算法
CDSA: Conservative Denoising Score-based Algorithm for Offline Reinforcement Learning
AI总结：
本文提出保守去噪评分算法CDSA，利用去噪评分模型建模数据集密度梯度以调整预训练策略生成的动作，在D4RL数据集上显著提升离线强化学习基线性能并具备即插即用能力。
摘要：
分布偏移是离线强化学习中的一个主要障碍，这要求最小化学习策略与行为策略之间的差异，以避免对稀有或未见动作的高估。以往的保守离线强化学习算法尽管在学习良好的分布内策略方面取得了成功，但难以泛化到未见动作。相比之下，我们提出利用由预训练离线强化学习算法生成的数据集密度的梯度场来调整原始动作。我们将保守性约束与策略解耦，从而能够惠及广泛的离线强化学习算法。因此，我们提出了保守去噪评分算法（CDSA），该算法利用去噪评分模型来建模数据集密度的梯度，而不是数据集密度本身，并在确定性和连续马尔可夫决策过程（MDP）环境中提供了一种更准确、更高效的方法来调整由预训练策略生成的动作。在实验中，我们表明我们的方法在D4RL数据集上显著提升了基线算法的性能，并证明了我们的模型在不同任务中、跨不同预训练离线强化学习策略的泛化能力和即插即用能力。我们还验证了，在使用我们的方法后，智能体表现出更强的风险规避能力，同时展示了其在多样化任务中有效泛化的能力。
英文摘要：
Distribution shift is a major obstacle in offline reinforcement learning, which necessitates minimizing the discrepancy between the learned policy and the behavior policy to avoid overestimating rare or unseen actions. Previous conservative offline RL algorithms struggle to generalize to unseen actions, despite their success in learning good in-distribution policy. In contrast, we propose to use the gradient fields of the dataset density generated from a pre-trained offline RL algorithm to adjust the original actions. We decouple the conservatism constraints from the policy, thus can benefit wide offline RL algorithms. As a consequence, we propose the Conservative Denoising Score-based Algorithm (CDSA) which utilizes the denoising score-based model to model the gradient of the dataset density, rather than the dataset density itself, and facilitates a more accurate and efficient method to adjust the action generated by the pre-trained policy in a deterministic and continuous MDP environment. In experiments, we show that our approach significantly improves the performance of baseline algorithms in D4RL datasets, and demonstrate the generalizability and plug-and-play capability of our model across different pre-trained offline RL policy in different tasks. We also validate that the agent exhibits greater risk aversion after employing our method while showcasing its ability to generalize effectively across diverse tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
