发表机构
Univ. Grenoble Alpes; CNRS; Inria; Grenoble INP; LIG（格勒诺布尔阿尔卑斯大学; 法国国家科学研究中心; 法国国家信息与自动化研究所; 格勒诺布尔国立理工学院; 格勒诺布尔信息学实验室）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本文研究恒定步长下非线性双时间尺度随机逼近的有限时间性质，给出了均方误差和偏差的紧上界，并揭示了非线性动力学引入的额外有限时间效应。
AI 中文摘要
双时间尺度随机逼近（TTSA）是分析强化学习、优化和随机控制中耦合迭代算法的基本工具。然而，非线性双时间尺度方案的有限时间保证仍然难以获得，尤其是在恒定步长下。本文研究了步长为$\alpha\gg\beta$的非线性TTSA。在标准稳定性、正则性和马尔可夫噪声假设下，我们给出了两个迭代序列围绕其极限平衡点的均方误差和偏差的上界。我们的界以$O(\alpha+\beta^2/\alpha^2)$为尺度，并证明当$\beta\le\alpha^{3/2}$时该界是紧的。分析将初始条件、快时间尺度跟踪误差、马尔可夫依赖和时间尺度耦合的贡献分离开来，从而阐明了$\beta^2/\alpha^2$项的来源。我们的结果揭示了与先前研究的线性TTSA设置的定性差异，表明非线性动力学引入了线性情形中不存在的额外有限时间效应。
英文摘要
Two-timescale stochastic approximation (TTSA) is a fundamental tool for analyzing coupled iterative algorithms in reinforcement learning, optimization, and stochastic control. However, finite-time guarantees for nonlinear two-timescale schemes remain difficult to obtain, especially under constant step-sizes. In this paper, we study nonlinear TTSA with step-sizes $α\ggβ$. Under standard stability, regularity, and Markovian noise assumptions, we upper bound the mean-squared error and the bias of both iterates around their limiting equilibria. Our bounds scale as $O(α+β^2/α^2)$, which we prove to be tight when $β\leα^{3/2}$. The analysis separates the contributions of initial conditions, fast-timescale tracking error, Markovian dependence, and timescale coupling, thereby clarifying the origin of the $β^2/α^2$ term. Our results reveal qualitative differences from the linear TTSA setting previously studied, showing that nonlinear dynamics introduce additional finite-time effects that are absent in the linear case.
