异步分类分布时序差分学习的有限迭代理论
A Sharp Finite-Iteration Theory for Asynchronous Categorical Distributional Temporal-Difference Learning
AI总结：
本文提出两种分类策略的有限迭代理论，通过等距嵌入将算法转化为异步单状态随机近似递归，从而在折扣和非折扣问题中提供保证。
摘要：
近期的非渐近分析显著推进了分布策略评估的理论，但主要关注同步全状态更新、生成模型、基于模型的估计器、加速变体或不同近似架构。标准分类时序差分学习通常在不同模式下使用。它在每次迭代中异步执行单状态更新，并在在线设置中由马尔可夫轨迹驱动。这在现有有限迭代理论与最贴近实际分布时序差分实现的分类递归之间留下重要空白。本文为两种分类策略方法填补了这一空白：在Cramér几何中使用标量分类时序差分学习，在最大均值偏差几何中使用多变量带符号分类时序差分学习。经过适当的等距嵌入后，两种算法都以异步单状态随机近似递归形式出现，其在状态wise supremum范数中收缩。这允许在折扣问题中使用i.i.d.和马尔可夫态采样，在非折扣固定时间 horizon 问题中使用i.i.d.事件采样提供有限迭代保证。
英文摘要：
We study finite-iteration behavior of asynchronous categorical distributional temporal-difference methods, covering scalar categorical TD (CTD) in the Cramér geometry and multivariate signed-categorical TD (MTD) in the maximum mean discrepancy (MMD) geometry. We establish high-probability guarantees under i.i.d. sampling and under a continuing Markovian trajectory. For CTD, the Cramér sample complexity to the true return law has leading term $\tilde O(ρ_{\min}^{-1}(1-γ)^{-2}\varepsilon^{-2})$ in the i.i.d. and $\tilde O(μ_{\min}^{-1}(1-γ)^{-2}\varepsilon^{-2}+μ_{\min}^{-1}t_{\mathrm{mix}})$ in the Markovian setting, without using variance reduction or data dropping. For MTD, the MMD sample complexity has leading terms $\tilde O(ρ_{\min}^{-1}(1-γ)^{-1-c}\varepsilon^{-2})$ and $\tilde O(μ_{\min}^{-1}(1-γ)^{-1-c}\varepsilon^{-2}+μ_{\min}^{-1}t_{\mathrm{mix}})$, where $c\in(0,2)$ is the homogeneity exponent of the MMD kernel, and they reduce to the CTD rates when $c=1$. These are, to our knowledge, the first finite-iteration rates for MTD. For undiscounted fixed-horizon policy evaluation, the same analysis applies to fixed-horizon versions of CTD and MTD under i.i.d. and episodic sampling, and the rates match the discounted ones with the effective horizon $(1-γ)^{-1}$ replaced by the horizon $H$ in the leading term. Matching minimax lower bounds show that the leading terms, and the implied $1$-Wasserstein rates, are optimal under i.i.d., Markovian, and episodic sampling. Together, these results provide a unified non-asymptotic analysis of asynchronous categorical distributional TD across scalar, multivariate, discounted, and fixed-horizon settings, with rates that are minimax optimal in their leading terms.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
