超越状态级镜像下降：带有参数策略的离线策略优化
Beyond State-Wise Mirror Descent: Offline Policy Optimization with Parametric Policies
AI总结：
本文研究了在一般函数逼近下离线强化学习的理论方面，扩展了理论保证到大或连续动作空间上的参数策略类，并提出了基于上下文耦合的新型分析和算法见解。
摘要：
我们研究了在一般函数逼近下离线强化学习（RL）的理论方面。尽管先前的工作（如Xie等人，2021）已经建立了通过悲观主义从离线数据中学习良好策略的理论基础，但现有的计算上可行的算法（通常在oracle高效意义上），如PSPI，仅适用于有限和小的动作空间。此外，这些算法依赖于状态级镜像下降，并要求演员隐式地由批评者函数诱导，无法适应实践中普遍存在的独立策略参数化。在本文中，我们解决了这些限制，并将理论保证扩展到大或连续动作空间上的参数策略类。当将镜像下降扩展到参数策略时，我们识别出上下文耦合是核心困难，并展示了如何将镜像下降连接到自然策略梯度，从而导致新的分析、保证和算法见解，包括对离线RL和模仿学习之间惊人统一的揭示。
英文摘要：
We investigate the theoretical aspects of offline reinforcement learning (RL) under general function approximation. While prior works (e.g., Xie et al., 2021) have established the theoretical foundations of learning a good policy from offline data via pessimism, existing algorithms that are computationally tractable (often in an oracle-efficient sense), such as PSPI, only apply to finite and small action spaces. Moreover, these algorithms rely on state-wise mirror descent and require actors to be implicitly induced from the critic functions, failing to accommodate standalone policy parameterization which is ubiquitous in practice. In this work, we address these limitations and extend the theoretical guarantees to parameterized policy classes over large or continuous action spaces. When extending mirror descent to parameterized policies, we identify contextual coupling as the core difficulty, and show how connecting mirror descent to natural policy gradient leads to novel analyses, guarantees, and algorithmic insights, including a surprising unification between offline RL and imitation learning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
