组合保守性：离线强化学习中的一种直推式方法
Compositional Conservatism: A Transductive Approach in Offline Reinforcement Learning
AI总结：
针对离线RL的分布偏移问题，本文提出COCOA，通过逆向动力学模型在直推重参数化的组合输入空间中实现与行为保守性正交的保守性，并在D4RL上验证其可稳定提升四种SOTA算法。
摘要：
离线强化学习（RL）是一个极具吸引力的框架，可从过往经验中学习最优策略，而无需与环境进行额外交互。然而，离线RL不可避免地面临分布偏移问题，即在策略执行期间遇到的状态和动作可能不在训练数据集分布内。一种常见解决方案是在策略或价值函数中引入保守性，以防范不确定性和未知因素。在本工作中，我们专注于实现相同的保守性目标，但采用了不同视角。我们提出用于离线RL的带锚点寻找的组合保守性（COmpositional COnservatism with Anchor-seeking，COCOA），该方法在直推重参数化（Netanyahu等人，2023）之上以组合方式追求保守性；该重参数化将输入变量（在我们的情形中为状态）分解为一个锚点及其与原始输入的差异。我们的COCOA利用学习到的逆向动力学模型同时寻找分布内锚点和差异，从而在策略或价值函数的组合输入空间中促进保守性。这种组合保守性独立于离线RL中普遍存在的行为保守性，并且对其不可知。我们将COCOA应用于四种最先进的离线RL算法，并在D4RL基准上进行评估，COCOA总体上提升了每种算法的性能。代码可在https://github.com/runamu/compositional-conservatism获取。
英文摘要：
Offline reinforcement learning (RL) is a compelling framework for learning optimal policies from past experiences without additional interaction with the environment. Nevertheless, offline RL inevitably faces the problem of distributional shifts, where the states and actions encountered during policy execution may not be in the training dataset distribution. A common solution involves incorporating conservatism into the policy or the value function to safeguard against uncertainties and unknowns. In this work, we focus on achieving the same objectives of conservatism but from a different perspective. We propose COmpositional COnservatism with Anchor-seeking (COCOA) for offline RL, an approach that pursues conservatism in a compositional manner on top of the transductive reparameterization (Netanyahu et al., 2023), which decomposes the input variable (the state in our case) into an anchor and its difference from the original input. Our COCOA seeks both in-distribution anchors and differences by utilizing the learned reverse dynamics model, encouraging conservatism in the compositional input space for the policy or value function. Such compositional conservatism is independent of and agnostic to the prevalent behavioral conservatism in offline RL. We apply COCOA to four state-of-the-art offline RL algorithms and evaluate them on the D4RL benchmark, where COCOA generally improves the performance of each algorithm. The code is available at https://github.com/runamu/compositional-conservatism.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
