arXiv 2210.083232023-04-05cs.LGcs.AI

一种用于离线强化学习的策略引导模仿方法

A Policy-Guided Imitation Approach for Offline Reinforcement Learning

JD Technology（京东科技）；Tsinghua University（清华大学）；Shanghai AI Laboratory（上海人工智能实验室）

Haoran Xu, Li Jiang, Jianxiong Li, Xianyuan Zhan

备注：: Oral @ NeurIPS 2022; An extended version with more experiments & correct some experiment details in previous version

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出策略引导离线RL方法POR，将策略分解为引导策略与执行策略，以监督解耦方式训练，实现状态组合性与分布外泛化，在D4RL基准上达到SOTA。

摘要：

离线强化学习（RL）方法通常可分为两类：基于RL的方法和基于模仿的方法。基于RL的方法原则上能享受分布外泛化，但会受到错误的离策略评估影响。基于模仿的方法避免了离策略评估，但过于保守而无法超越数据集。在本研究中，我们提出了一种替代方法，继承了模仿式方法的训练稳定性，同时仍允许逻辑上的分布外泛化。我们将离线RL中传统的奖励最大化策略分解为引导策略和执行策略。在训练期间，引导策略和执行策略仅使用数据集中的数据，以监督和解耦的方式学习。在评估期间，引导策略通过指示执行策略应前往何处来引导它以最大化奖励，充当先知。通过这种方式，我们的算法允许来自数据集的状态组合性，而非先前模仿式方法中进行的动作组合性。我们将这种新方法称为策略引导离线RL（POR）。POR在D4RL（离线RL的标准基准）上展示了最先进的性能。我们还强调了POR在利用补充的次优数据进行改进以及仅通过更改引导策略即可轻松适应新任务方面的优势。

英文摘要：

Offline reinforcement learning (RL) methods can generally be categorized into two types: RL-based and Imitation-based. RL-based methods could in principle enjoy out-of-distribution generalization but suffer from erroneous off-policy evaluation. Imitation-based methods avoid off-policy evaluation but are too conservative to surpass the dataset. In this study, we propose an alternative approach, inheriting the training stability of imitation-style methods while still allowing logical out-of-distribution generalization. We decompose the conventional reward-maximizing policy in offline RL into a guide-policy and an execute-policy. During training, the guide-poicy and execute-policy are learned using only data from the dataset, in a supervised and decoupled manner. During evaluation, the guide-policy guides the execute-policy by telling where it should go so that the reward can be maximized, serving as the \textit{Prophet}. By doing so, our algorithm allows \textit{state-compositionality} from the dataset, rather than \textit{action-compositionality} conducted in prior imitation-style methods. We dumb this new approach Policy-guided Offline RL (\texttt{POR}). \texttt{POR} demonstrates the state-of-the-art performance on D4RL, a standard benchmark for offline RL. We also highlight the benefits of \texttt{POR} in terms of improving with supplementary suboptimal data and easily adapting to new tasks by only changing the guide-poicy.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。