离线强化学习中仅通过支持约束实现的策略约束
Policy Constraint by Only Support Constraint for Offline Reinforcement Learning
AI总结：
针对离线强化学习中策略约束过度保守的问题，本文提出仅支持约束OSC，通过最大化学习策略在行为策略支持集内的总概率并利用扩散模型刻画支持集，在多个基准上显著提升性能。
摘要：
离线强化学习（RL）旨在利用预先收集的数据集优化策略，以最大化累积奖励。然而，由于学习策略与行为策略之间的分布偏移，离线强化学习面临挑战，导致在计算分布外（OOD）动作的Q值时产生误差。为缓解这一问题，策略约束方法旨在将学习策略的分布约束在行为策略的分布范围内，或将动作选择限制在行为策略的支持集内。然而，当前的策略约束方法往往表现出过度的保守性，阻碍策略进一步超越行为策略的性能。在本工作中，我们提出仅支持约束（Only Support Constraint，OSC），该方法源于最大化学习策略在行为策略支持集内的总概率，以解决策略约束的保守性问题。OSC引入了一个正则化项，仅将策略限制在支持集内，而不对支持集内的动作施加额外约束。此外，为充分发挥新策略约束的性能，OSC利用扩散模型有效刻画行为策略的支持集。在多种离线RL基准上的实验评估表明，OSC显著提升了性能，缓解了与分布偏移相关的挑战，并减轻了策略约束的保守性。代码可在 https://github.com/MoreanP/OSC 获取。
英文摘要：
Offline reinforcement learning (RL) aims to optimize a policy by using pre-collected datasets, to maximize cumulative rewards. However, offline reinforcement learning suffers challenges due to the distributional shift between the learned and behavior policies, leading to errors when computing Q-values for out-of-distribution (OOD) actions. To mitigate this issue, policy constraint methods aim to constrain the learned policy's distribution with the distribution of the behavior policy or confine action selection within the support of the behavior policy. However, current policy constraint methods tend to exhibit excessive conservatism, hindering the policy from further surpassing the behavior policy's performance. In this work, we present Only Support Constraint (OSC) which is derived from maximizing the total probability of learned policy in the support of behavior policy, to address the conservatism of policy constraint. OSC presents a regularization term that only restricts policies to the support without imposing extra constraints on actions within the support. Additionally, to fully harness the performance of the new policy constraints, OSC utilizes a diffusion model to effectively characterize the support of behavior policies. Experimental evaluations across a variety of offline RL benchmarks demonstrate that OSC significantly enhances performance, alleviating the challenges associated with distributional shifts and mitigating conservatism of policy constraints. Code is available at https://github.com/MoreanP/OSC.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
