面向$f$-散度正则化上下文带机的精确分析
Towards a Sharp Analysis of Offline Policy Learning for $f$-Divergence-Regularized Contextual Bandits
AI总结：
本文研究了$f$-散度正则化上下文带机的样本复杂度分析，首次在单一策略可集中性下实现了$\tilde{O}(ε^{-1})$的样本复杂度，并提出了近似匹配的下界。
摘要：
许多离线强化学习算法都基于$f$-散度正则化，但其样本复杂度*以正则化目标定义*仍然缺乏紧致分析，尤其是在具体数据覆盖条件方面。本文研究了实现离线$f$-散度正则化上下文带机$\tilde{Θ}(ε^{-1})$样本复杂度的精确可集中性要求。对于逆Kullback-Leibler（KL）散度，即最常用的之一，我们首次通过一种新的悲观性分析方法，在单一策略可集中性下实现了$\tilde{O}(ε^{-1})$的样本复杂度，超越了现有在所有策略可集中性下的$\tilde{O}(ε^{-1})$界和在单一策略可集中性下的$\tilde{O}(ε^{-2})$界。我们还提出了一个近似匹配的下界，证明了乘以单一策略可集中性的依赖性是必要条件，以最大地利用逆KL的曲率属性。此外，对于具有强凸$f$的$f$-散度，其中逆KL不包含在内，我们显示即使没有悲观估计或单一策略可集中性，$\tilde{Θ}(ε^{-1})$的精确样本复杂度也是可以实现的。我们进一步用数值实验验证了我们的理论见解，并将我们的分析扩展到上下文对决带机。我们相信这些结果在全面理解带有$f$-散度正则化的目标方面迈出了重要一步。
英文摘要：
Many offline reinforcement learning algorithms are underpinned by $f$-divergence regularization, but their sample complexity *defined with respect to regularized objectives* still lacks tight analyses, especially in terms of concrete data coverage conditions. In this paper, we study the exact concentrability requirements to achieve the $\tildeΘ(ε^{-1})$ sample complexity for offline $f$-divergence-regularized contextual bandits. For reverse Kullback-Leibler (KL) divergence, arguably the most commonly used one, we achieve an $\tilde{O}(ε^{-1})$ sample complexity under single-policy concentrability for the first time via a novel pessimism-based analysis, surpassing existing $\tilde{O}(ε^{-1})$ bound under all-policy concentrability and $\tilde{O}(ε^{-2})$ bound under single-policy concentrability. We also propose a near-matching lower bound, demonstrating that a multiplicative dependency on single-policy concentrability is necessary to maximally exploit the curvature property of reverse KL. Moreover, for $f$-divergences with strongly convex $f$, to which reverse KL *does not* belong, we show that the sharp sample complexity $\tildeΘ(ε^{-1})$ is achievable even without pessimistic estimation or single-policy concentrability. We further corroborate our theoretical insights with numerical experiments and extend our analysis to contextual dueling bandits. We believe these results take a significant step towards a comprehensive understanding of objectives with $f$-divergence regularization.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
