arXiv 2302.085602024-01-26cs.LGcs.AIcs.RO

Dual RL：强化学习与模仿学习的统一及新方法

Dual RL: Unification and New Methods for Reinforcement and Imitation Learning

The University of Texas at Austin（德克萨斯大学奥斯汀分校）；FAIR, Meta AI（Meta AI 基础人工智能研究）；University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）

Harshit Sikchi, Qinqing Zheng, Amy Zhang, Scott Niekum

备注：: Published as a conference paper (spotlight) at ICLR 2024. 48 pages

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本研究提出dual RL统一框架，将现有离线强化学习与模仿学习算法纳入其中，分析出原有方法缺陷，并分别提出ReCOIL和f-DVL新方法解决对应问题，在模拟机器人任务上验证了性能提升。

摘要：

强化学习（RL）的目标是找到一个能最大化期望累积回报的策略。已有研究表明，该目标可表示为带线性约束的状态-动作访问分布优化问题。我们将该公式的对偶问题称为dual RL（对偶强化学习），它是无约束的，更易于优化。在本研究中，我们首先将多种最先进的离线RL和离线模仿学习（IL）算法归为具有共享结构的dual RL方法实例。这种统一框架让我们能够找出先前方法缺陷的根本原因。对于离线IL，我们的分析显示，先前方法基于严格的覆盖假设，这极大限制了它们在实际中的性能。为解决这一局限，我们提出了一种新的无判别器方法ReCOIL，该方法能从任意离策略数据中学习模仿，以达到接近专家的性能。对于离线RL，我们的分析将最新的离线RL方法XQL纳入对偶框架，并进一步提出新方法f-DVL，它为Gumbel回归损失提供了替代选择，解决了XQL已知的训练不稳定问题。我们提出的ReCOIL和f-DVL两种方法分别在IL和RL任务上的性能提升，已在大量模拟机器人运动和操作任务套件上得到验证。项目代码和细节可查看https://hari-sikchi.github.io/dual-rl。

英文摘要：

The goal of reinforcement learning (RL) is to find a policy that maximizes the expected cumulative return. It has been shown that this objective can be represented as an optimization problem of state-action visitation distribution under linear constraints. The dual problem of this formulation, which we refer to as dual RL, is unconstrained and easier to optimize. In this work, we first cast several state-of-the-art offline RL and offline imitation learning (IL) algorithms as instances of dual RL approaches with shared structures. Such unification allows us to identify the root cause of the shortcomings of prior methods. For offline IL, our analysis shows that prior methods are based on a restrictive coverage assumption that greatly limits their performance in practice. To fix this limitation, we propose a new discriminator-free method ReCOIL that learns to imitate from arbitrary off-policy data to obtain near-expert performance. For offline RL, our analysis frames a recent offline RL method XQL in the dual framework, and we further propose a new method f-DVL that provides alternative choices to the Gumbel regression loss that fixes the known training instability issue of XQL. The performance improvements by both of our proposed methods, ReCOIL and f-DVL, in IL and RL are validated on an extensive suite of simulated robot locomotion and manipulation tasks. Project code and details can be found at this https://hari-sikchi.github.io/dual-rl.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。