带有KL正则化的离线Q学习的快速收敛速率
Fast Rates for Offline Q-Learning with KL Regularization
AI总结：
针对有限时间马尔可夫决策过程的离线Q学习，提出KL正则化Q学习（KLQ）和带悲观性的KL正则化Q学习（KLPQ），证明KLPQ在保留快速收敛速率的同时消除了指数依赖，为离线Q学习提供了首个快速收敛保证。
摘要：
本文研究带有Kullback-Leibler（KL）正则化的有限时间马尔可夫决策过程的离线Q学习。近期研究表明，KL正则化可在离线决策问题中实现快速收敛速率。然而，采用增量随机近似更新的Q学习能否在离线场景中达到此类快速速率仍不明确。我们首先提出经典Q学习的简单变体——KL正则化Q学习（KLQ），通过利用KL正则化，证明KLQ无需显式悲观性即可达到$\widetilde{O}(1/\epsilon)$的样本复杂度。不过，该保证对KL正则化参数和时间跨度存在指数依赖。为解决此局限，我们进一步提出将悲观性融入KLQ的KL正则化悲观Q学习（KLPQ），证明KLPQ在保留相同的\\\\(\widetilde O(1/\epsilon)\\\\
英文摘要：
This paper investigates offline Q-learning for finite-horizon Markov decision processes with Kullback-Leibler (KL) regularization. Recent studies have shown that KL regularization can yield fast rates in offline decision-making problems. However, it remains unclear whether Q-learning, which adopts incremental stochastic approximation updates, can achieve such fast rates in the offline setting. We first propose KL-regularized Q-Learning (KLQ), a simple variant of classical Q-learning. By exploiting KL regularization, we prove that KLQ achieves a sample complexity of $\widetilde{O}(1/ε)$ without explicit pessimism. This guarantee, however, has an exponential dependence on the KL regularization parameter and the horizon. To address this limitation, we further propose KL-regularized Pessimistic Q-Learning (KLPQ), which incorporates pessimism into KLQ. We prove that KLPQ retains the same \(\widetilde O(1/ε)\) rate while removing the explicit exponential dependence, thereby further improving the sample complexity. To the best of our knowledge, these results establish the first fast-rate guarantees for offline Q-learning in tabular finite-horizon KL-regularized MDPs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
