基于数据集约束的离线强化学习策略正则化
Policy Regularization with Dataset Constraint for Offline Reinforcement Learning
AI总结：
针对离线强化学习中传统分布或支撑集约束过于保守的问题，本文提出PRDC，通过最近状态-动作样本的动作正则化策略，在保持保守性的同时缓解价值高估并取得最先进性能。
摘要：
我们研究从固定数据集中学习尽可能最优策略的问题，即离线强化学习（Reinforcement Learning，RL）。现有离线RL工作的一个常见类别是策略正则化，这类方法通常通过行为策略的分布或支撑集来约束所学策略。然而，分布约束和支撑集约束都过于保守，因为在考虑特定状态时，它们都会迫使策略选择与行为策略相似的动作。这将限制所学策略的性能，尤其是在行为策略次优时。本文发现，将策略向最近的状态-动作对正则化可以更加有效，因此提出了基于数据集约束的策略正则化（Policy Regularization with Dataset Constraint，PRDC）。在给定状态下更新策略时，PRDC会在整个数据集中搜索最近的状态-动作样本，随后用该样本的动作约束策略。与以往工作不同，PRDC能够利用数据集中恰当的行为引导策略，同时允许其选择在给定状态下未出现在数据集中的动作。这是一种更柔和的约束，但仍对分布外动作保持足够的保守性。实证证据和理论分析表明，PRDC能够缓解离线RL中根本性且具有挑战性的价值高估问题，并具有有界的性能差距。此外，在一系列运动和导航任务上，与现有方法相比，PRDC取得了最先进的性能。代码可在 https://github.com/LAMDA-RL/PRDC 获取。
英文摘要：
We consider the problem of learning the best possible policy from a fixed dataset, known as offline Reinforcement Learning (RL). A common taxonomy of existing offline RL works is policy regularization, which typically constrains the learned policy by distribution or support of the behavior policy. However, distribution and support constraints are overly conservative since they both force the policy to choose similar actions as the behavior policy when considering particular states. It will limit the learned policy's performance, especially when the behavior policy is sub-optimal. In this paper, we find that regularizing the policy towards the nearest state-action pair can be more effective and thus propose Policy Regularization with Dataset Constraint (PRDC). When updating the policy in a given state, PRDC searches the entire dataset for the nearest state-action sample and then restricts the policy with the action of this sample. Unlike previous works, PRDC can guide the policy with proper behaviors from the dataset, allowing it to choose actions that do not appear in the dataset along with the given state. It is a softer constraint but still keeps enough conservatism from out-of-distribution actions. Empirical evidence and theoretical analysis show that PRDC can alleviate offline RL's fundamentally challenging value overestimation issue with a bounded performance gap. Moreover, on a set of locomotion and navigation tasks, PRDC achieves state-of-the-art performance compared with existing methods. Code is available at https://github.com/LAMDA-RL/PRDC
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
