发表机构
Indian Institute of Science, Bengaluru; Laboratoire de Recherche de l’EPITA（印度科学理工学院（班加罗尔）; EPITA研究实验室）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本文研究非线性双时间尺度随机逼近的平稳偏差，推导一阶偏差展开式，发现混合项 $\varepsilon^2/\eta$ 影响偏差减少，并验证了 Richardson--Romberg 外推需匹配路径权重，通过非线性马尔可夫例子和有限运行分析验证了结果。
AI 中文摘要
常步长随机逼近通常具有非零的平稳均值误差，该误差在时间平均下持续存在。本文针对由外生有限状态马尔可夫链驱动的非线性双时间尺度递推研究该误差。在给定的光滑性假设和关于平稳分布的条件下，我们推导了一阶偏差展开式，其误差界在慢步长远小于快步长时保持一致。快流形坐标使得相关的协方差方程在该极限下保持正则。对于快步长 $\eta$ 和慢步长 $\varepsilon$，展开式揭示了混合贡献 $\varepsilon^2/\eta$ 以及分别与每个步长线性相关的项。这种依赖关系对偏差减少至关重要：沿幂律步长路径，偏差指数不必为整数，因此 Richardson--Romberg 外推需要与路径匹配的权重。一个可精确求解的非线性马尔可夫例子验证了系数。我们验证了时间差分学习的局部化，并在相同更新预算下比较了有限运行的外推。对于有限运行，我们在额外的耦合假设下，界定了两个时间尺度上尾部平均的初始化误差。在加性独立噪声的特殊情况下，有符号三阶矩抵消产生了更尖锐的余项。
英文摘要
Constant-step stochastic approximation generally has a nonzero stationary mean error that persists under time averaging. This paper studies that error for nonlinear two-timescale recursions driven by an exogenous finite-state Markov chain. Under stated smoothness assumptions and conditions on the stationary distribution, we derive a first-order bias expansion whose error bound remains uniform as the slow step size becomes much smaller than the fast step size. Fast-manifold coordinates keep the associated covariance equation regular in this limit. For fast step $η$ and slow step $\varepsilon$, the expansion reveals a mixed contribution $\varepsilon^2/η$ alongside terms linear in each step size. This dependence matters for bias reduction: along power-law step-size paths, the bias exponents need not be integers, so Richardson--Romberg extrapolation requires weights matched to the path. An exactly solvable nonlinear Markov example verifies the coefficients. We verify localization for temporal-difference learning and compare finite-run extrapolation at equal update budgets. For finite runs, we bound the initialization error of tail averages on both timescales under an additional coupling assumption. In the special case of additive independent noise, signed third-moment cancellation yields a sharper remainder.
