桥接离线强化学习与模仿学习：一个关于悲观主义的故事
Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism
AI总结：
提出一种统一模仿学习与离线RL的新框架，基于悲观主义的LCB算法能适应未知数据组成，在近专家数据集上实现更快的1/N收敛速率，并在上下文老虎机和MDPs中展现自适应最优性。
摘要：
离线（或批量）强化学习（RL）算法旨在从固定数据集中学习最优策略，而无需主动收集数据。根据离线数据集的组成，主要有两类方法：适用于专家数据集的模仿学习和通常需要均匀覆盖数据集的普通离线 RL。从实际角度来看，数据集往往偏离这两个极端，确切的数据组成通常是先验未知的。为了弥合这一差距，我们提出了一种新的离线 RL 框架，该框架在数据组成的两个极端之间平滑插值，从而统一了模仿学习和普通离线 RL。新框架以集中系数的弱版本为中心，该系数仅衡量从行为策略到专家策略的偏差。在这个新框架下，我们进一步研究算法设计问题：能否开发出一种算法，既能达到极小极大最优速率，又能适应未知的数据组成？为了解决这个问题，我们考虑了一种基于离线 RL 中面对不确定性时的悲观主义而开发的置信下界（LCB）算法。我们研究了 LCB 的有限样本特性以及多臂老虎机、上下文老虎机和马尔可夫决策过程（MDPs）中的信息论极限。我们的分析揭示了关于最优速率的惊人事实。特别是，在所有三种设置中，对于近专家数据集，LCB 实现了更快的 1/N 速率，而离线 RL 中的通常速率为 1/√N，其中 N 是批量数据集中的样本数量。在具有至少两个上下文的上下文老虎机情况下，我们证明 LCB 对于整个数据组成范围是自适应最优的，实现了从模仿学习到离线 RL 的平滑过渡。我们进一步表明 LCB 在 MDPs 中几乎自适应最优。
英文摘要：
Offline (or batch) reinforcement learning (RL) algorithms seek to learn an optimal policy from a fixed dataset without active data collection. Based on the composition of the offline dataset, two main categories of methods are used: imitation learning which is suitable for expert datasets and vanilla offline RL which often requires uniform coverage datasets. From a practical standpoint, datasets often deviate from these two extremes and the exact data composition is usually unknown a priori. To bridge this gap, we present a new offline RL framework that smoothly interpolates between the two extremes of data composition, hence unifying imitation learning and vanilla offline RL. The new framework is centered around a weak version of the concentrability coefficient that measures the deviation from the behavior policy to the expert policy alone. Under this new framework, we further investigate the question on algorithm design: can one develop an algorithm that achieves a minimax optimal rate and also adapts to unknown data composition? To address this question, we consider a lower confidence bound (LCB) algorithm developed based on pessimism in the face of uncertainty in offline RL. We study finite-sample properties of LCB as well as information-theoretic limits in multi-armed bandits, contextual bandits, and Markov decision processes (MDPs). Our analysis reveals surprising facts about optimality rates. In particular, in all three settings, LCB achieves a faster rate of $1/N$ for nearly-expert datasets compared to the usual rate of $1/\sqrt{N}$ in offline RL, where $N$ is the number of samples in the batch dataset. In the case of contextual bandits with at least two contexts, we prove that LCB is adaptively optimal for the entire data composition range, achieving a smooth transition from imitation learning to offline RL. We further show that LCB is almost adaptively optimal in MDPs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
