锥压缩算子的随机逼近：$Q$ -学习的精确$\ell_\infty$ -界
Stochastic approximation with cone-contractive operators: Sharp $\ell_\infty$-bounds for $Q$-learning
摘要：
受强化学习中的$Q$学习算法研究的启发，本文研究了一类基于算子的随机逼近过程，该算子满足单调性和拟压缩性条件.我们证明了一个一般的三明治关系在每一个时间的误差，并使用它来获得锥诱导规范范数的误差的非渐近界。这些结果是在确定性框架内得出的，不需要对噪声进行假设。我们说明了这些一般的界限应用到同步$Q$ -学习折扣马尔可夫决策过程的离散状态-动作空间，特别是通过推导非渐近界的$\ell_\infty$ -规范的步长范围。这些结果是迄今为止已知的最尖锐的，我们通过模拟表明，在最坏的情况下，我们的边界的依赖性不能得到改善。这些结果表明，相对于基于模型的$Q$迭代，$Q$学习的基于$\ell_\infty$的样本复杂度在折扣因子$γ$方面是次优的。
英文摘要：
Motivated by the study of $Q$-learning algorithms in reinforcement learning, we study a class of stochastic approximation procedures based on operators that satisfy monotonicity and quasi-contractivity conditions with respect to an underlying cone. We prove a general sandwich relation on the iterate error at each time, and use it to derive non-asymptotic bounds on the error in terms of a cone-induced gauge norm. These results are derived within a deterministic framework, requiring no assumptions on the noise. We illustrate these general bounds in application to synchronous $Q$-learning for discounted Markov decision processes with discrete state-action spaces, in particular by deriving non-asymptotic bounds on the $\ell_\infty$-norm for a range of stepsizes. These results are the sharpest known to date, and we show via simulation that the dependence of our bounds cannot be improved in a worst-case sense. These results show that relative to a model-based $Q$-iteration, the $\ell_\infty$-based sample complexity of $Q$-learning is suboptimal in terms of the discount factor $γ$.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
