基于约束潜在动作策略的模型驱动离线强化学习
Constrained Latent Action Policies for Model-Based Offline Reinforcement Learning
AI总结：
本文提出C-LAP，通过约束潜在动作分布生成策略，提高模型驱动离线强化学习在视觉观测数据集上的性能。
摘要：
在离线强化学习中，政策是在缺乏环境反馈的情况下利用静态数据集进行学习的。与在线设置不同，仅使用静态数据集会带来额外的挑战，例如策略生成分布外样本。模型驱动的离线强化学习方法试图通过学习环境底层动力学的模型并用其指导策略搜索来克服这些挑战。虽然这有益，但在数据有限的情况下，模型误差和分布外状态的价值过估计问题可能会恶化性能。当前的模型驱动方法在贝尔曼更新中应用了一些保守性概念，通常通过从模型集合中推导出的不确定性估计来实现。在本文中，我们提出了约束潜在动作策略（C-LAP），它学习了观测和动作联合分布的生成模型。我们将策略学习视为一个受约束的目标，以始终留在潜在动作分布的支持内，并利用模型的生成能力对生成的动作施加隐含的约束。从而消除了在贝尔曼更新中使用额外不确定性惩罚的需要，并显著减少了学习策略所需的梯度步骤数量。我们通过D4RL和V-D4RL基准对C-LAP进行了实证评估，并证明C-LAP在与最新方法相竞争，特别是在具有视觉观测的数据集上表现优于其他方法。
英文摘要：
In offline reinforcement learning, a policy is learned using a static dataset in the absence of costly feedback from the environment. In contrast to the online setting, only using static datasets poses additional challenges, such as policies generating out-of-distribution samples. Model-based offline reinforcement learning methods try to overcome these by learning a model of the underlying dynamics of the environment and using it to guide policy search. It is beneficial but, with limited datasets, errors in the model and the issue of value overestimation among out-of-distribution states can worsen performance. Current model-based methods apply some notion of conservatism to the Bellman update, often implemented using uncertainty estimation derived from model ensembles. In this paper, we propose Constrained Latent Action Policies (C-LAP) which learns a generative model of the joint distribution of observations and actions. We cast policy learning as a constrained objective to always stay within the support of the latent action distribution, and use the generative capabilities of the model to impose an implicit constraint on the generated actions. Thereby eliminating the need to use additional uncertainty penalties on the Bellman update and significantly decreasing the number of gradient steps required to learn a policy. We empirically evaluate C-LAP on the D4RL and V-D4RL benchmark, and show that C-LAP is competitive to state-of-the-art methods, especially outperforming on datasets with visual observations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
