可实现性与部分覆盖下的离线极小极大软Q学习
Offline Minimax Soft-Q-learning Under Realizability and Partial Coverage
AI总结：
本文提出在仅部分覆盖和软Q函数可实现性下，利用极小极大学习算法估计Q函数，为离线强化学习提供PAC保证，并推广至软间隔下的普通Q函数。
摘要：
在离线强化学习（RL）中，我们没有探索的机会，因此必须假设数据足以指导选择好的策略，具体形式为假设某种覆盖性、可实现性、贝尔曼完备性和/或硬间隔（间隙）。在这项工作中，我们提出了在仅部分覆盖（具体而言，仅覆盖单个比较策略）以及单个策略的软（熵正则化）Q函数和定义为某个极小极大优化问题鞍点的相关函数的可实现性下，具有PAC保证的基于值的离线RL算法。这为离线RL提供了更精细且通常更宽松的条件。我们进一步展示了在软间隔条件下对普通Q函数的类似结果。为获得这些保证，我们利用新颖的极小极大学习算法，以$L^2$收敛保证精确估计软或普通Q函数。我们的算法的损失函数源于将估计问题转化为非线性凸优化问题并进行拉格朗日化。
英文摘要：
In offline reinforcement learning (RL) we have no opportunity to explore so we must make assumptions that the data is sufficient to guide picking a good policy, taking the form of assuming some coverage, realizability, Bellman completeness, and/or hard margin (gap). In this work we propose value-based algorithms for offline RL with PAC guarantees under just partial coverage, specifically, coverage of just a single comparator policy, and realizability of soft (entropy-regularized) Q-function of the single policy and a related function defined as a saddle point of certain minimax optimization problem. This offers refined and generally more lax conditions for offline RL. We further show an analogous result for vanilla Q-functions under a soft margin condition. To attain these guarantees, we leverage novel minimax learning algorithms to accurately estimate soft or vanilla Q-functions with $L^2$-convergence guarantees. Our algorithms' loss functions arise from casting the estimation problems as nonlinear convex optimization problems and Lagrangifying.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
