MAHALO：统一离线强化学习与基于观测的模仿学习
MAHALO: Unifying Offline Reinforcement Learning and Imitation Learning from Observations
AI总结：
本文提出MAHALO，一种基于悲观性能下界和对抗训练的通用离线PLfO方法，统一处理离线IL、ILfO和RL，并在多类任务上优于或匹配专门算法。
摘要：
我们研究了一种新的序列决策范式，称为基于观测的离线策略学习（PLfO）。离线PLfO旨在利用具有次优质量的数据集来学习策略：1）只有一部分轨迹带有奖励标签，2）带标签的轨迹可能不包含动作，3）带标签的轨迹可能质量不高，4）数据可能不具备完全覆盖性。这种不完善性在现实世界学习场景中很常见，离线PLfO涵盖了许多现有的离线学习设置，包括离线模仿学习（IL）、基于观测的离线模仿学习（ILfO）和离线强化学习（RL）。在这项工作中，我们提出了一种通用的离线PLfO方法，称为模态无关的对抗性假设自适应观测学习（MAHALO）。MAHALO建立在离线RL中的悲观主义概念之上，利用一个性能下界来优化策略，该下界考虑了由于数据集覆盖不足而产生的不确定性。我们通过对抗性地训练与数据一致的评论家函数和奖励函数来实现这一思想，从而迫使学习到的策略对数据缺陷具有鲁棒性。我们证明，在理论和实验中，MAHALO在各种离线PLfO任务上始终优于或匹配专门算法。我们的代码可在https://github.com/AnqiLi/mahalo获取。
英文摘要：
We study a new paradigm for sequential decision making, called offline policy learning from observations (PLfO). Offline PLfO aims to learn policies using datasets with substandard qualities: 1) only a subset of trajectories is labeled with rewards, 2) labeled trajectories may not contain actions, 3) labeled trajectories may not be of high quality, and 4) the data may not have full coverage. Such imperfection is common in real-world learning scenarios, and offline PLfO encompasses many existing offline learning setups, including offline imitation learning (IL), offline IL from observations (ILfO), and offline reinforcement learning (RL). In this work, we present a generic approach to offline PLfO, called $\textbf{M}$odality-agnostic $\textbf{A}$dversarial $\textbf{H}$ypothesis $\textbf{A}$daptation for $\textbf{L}$earning from $\textbf{O}$bservations (MAHALO). Built upon the pessimism concept in offline RL, MAHALO optimizes the policy using a performance lower bound that accounts for uncertainty due to the dataset's insufficient coverage. We implement this idea by adversarially training data-consistent critic and reward functions, which forces the learned policy to be robust to data deficiency. We show that MAHALO consistently outperforms or matches specialized algorithms across a variety of offline PLfO tasks in theory and experiments. Our code is available at https://github.com/AnqiLi/mahalo.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
