具有通用函数逼近和单策略集中性的可证明高效的离线目标条件强化学习
Provably Efficient Offline Goal-Conditioned Reinforcement Learning with General Function Approximation and Single-Policy Concentrability
AI总结：
本文针对离线目标条件强化学习，提出首个在通用函数逼近和单策略集中性假设下可证明高效且无需极小极大优化的算法，通过利用目标函数强凸性实现样本复杂度保证，并在真实环境中验证了优越性。
摘要：
目标条件强化学习（GCRL）是指学习旨在达到多样目标的通用技能。特别是，离线GCRL仅需要纯预收集的数据集来执行训练任务，而无需与环境进行额外交互。尽管离线GCRL已日益普及，且许多先前工作已展示其实证成功，但对高效离线GCRL算法的理论理解尚未充分建立，尤其是在状态空间巨大且离线数据集仅覆盖我们旨在学习的策略的情况下。在本文中，我们对一种现有实证成功的离线GCRL算法进行了严格的理论分析。我们证明，在轻微修改下，该算法凭借目标函数的（半）强凸性，在通用函数逼近下享有$\widetilde{O}(\text{poly}(1/ε))$的样本复杂度（其中$ε$是学习策略的期望次优性）。我们仅需对数据集（单策略集中性）和函数类（可实现性）的近乎最小假设。此外，该算法由两个不交错的优化步骤组成，我们称之为$V$-学习和策略学习，并且由于不涉及极小极大优化，它在计算上是稳定的。我们还通过展示修改后的算法在各种真实世界环境中优于先前算法来实证验证我们的理论。据我们所知，这是第一个在通用函数逼近和单策略集中性下既可证明高效，又无需解决极小极大优化问题而实证成功的算法。
英文摘要：
Goal-conditioned reinforcement learning (GCRL) refers to learning general-purpose skills that aim to reach diverse goals. In particular, offline GCRL only requires purely pre-collected datasets to perform training tasks without additional interactions with the environment. Although offline GCRL has become increasingly prevalent and many previous works have demonstrated its empirical success, the theoretical understanding of efficient offline GCRL algorithms is not well established, especially when the state space is huge and the offline dataset only covers the policy we aim to learn. In this paper, we provide a rigorous theoretical analysis of an existing empirically successful offline GCRL algorithm. We prove that under slight modification, this algorithm enjoys an $\widetilde{O}(\text{poly}(1/ε))$ sample complexity (where $ε$ is the desired suboptimality of the learned policy) with general function approximation thanks to the property of (semi-)strong convexity of the objective functions. We only require nearly minimal assumptions on the dataset (single-policy concentrability) and the function class (realizability). Moreover, this algorithm consists of two uninterleaved optimization steps, which we refer to as $V$-learning and policy learning, and is computationally stable since it does not involve minimax optimization. We also empirically validate our theory by showing that the modified algorithm outperforms the previous algorithm in various real-world environments. To the best of our knowledge, this is the first algorithm that is both provably efficient with general function approximation and single-policy concentrability, and empirically successful without requiring solving minimax optimization problems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
