在线上下文老虎机中基于前向KL正则化的快速收敛率
Fast Rates for Offline Contextual Bandits with Forward-KL Regularization under Single-Policy Concentrability
AI总结：
本文研究了在线上下文老虎机中前向KL正则化下的快速收敛率问题，通过简化分析提出首个$\tilde{O}(ε^{-1})$上界，统一了表格和一般函数逼近设置，并展示了下界证明了上界的统计紧性。
摘要：
Kullback-Leibler（KL）正则化在强化学习算法中普遍存在，以反KL或前向KL形式出现。最近的研究显示，在反KL正则化下，决策的$ε^{-1}$型快速收敛率优于标准$ε^{-2}$型样本复杂度。然而，对于前向KL正则化的目标，现有统计分析要么不适用，要么导致$\tilde{O}(ε^{-2})$的慢收敛率。本文通过简化分析前向KL正则化下的在线上下文老虎机问题，首次在表格和一般函数逼近设置中给出$\tilde{O}(ε^{-1})$的上界，均基于单策略集中性概念。特别地，我们的凸分析流程通过新颖地利用悲观原理统一了这些设置，并完全绕过了基于中值定理的先前证明流程。此外，我们提供了最优的下界，证明了我们的上界在统计速率上的紧性。我们的下界还表明，前向KL正则化的样本复杂度在低正则化区域恢复了无正则化慢速率，与反KL正则化相似。
英文摘要：
\emph{Kullback-Leibler} (KL) regularization is ubiquitous in reinforcement learning algorithms in the form of \emph{reverse} or \emph{forward} KL. Recent studies have demonstrated $ε^{-1}$-type fast rates for decision making under reverse KL regularization, in contrast to the standard $ε^{-2}$-type sample complexity. However, for forward-KL-regularized objectives, existing statistical analyses are either not applicable or result in $\tilde{O}(ε^{-2})$ slow rates. We take the first step towards addressing this problem via a streamlined analysis of forward-KL-regularized offline CBs. We give the first $\tilde{O}(ε^{-1})$ upper bounds in tabular and general function approximation settings, both under notions of \emph{single-policy concentrability}. In particular, our convex-analytical pipeline unifies these settings by exploiting the pessimism principle in a novel way and completely bypasses the proof routines in previous works based on the mean value theorem, which might be of independent interest. Moreover, we provide rate-optimal lower bounds, manifesting the tightness of our upper bounds in terms of statistical rates. Our lower bounds also demonstrate that the forward-KL-regularized sample complexity recovers the unregularized slow rate in the low-regularization regime, similarly to the reverse-KL regularization.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
