arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2401.12764math.OCcs.LG
收藏

快速非线性双时间尺度随机逼近:实现 $O(1/k)$ 的有限样本复杂度

Fast Nonlinear Two-Time-Scale Stochastic Approximation: Achieving $O(1/k)$ Finite-Sample Complexity

Virginia Tech(弗吉尼亚理工大学)

Thinh T. Doan

AI总结:

本文提出一种双时间尺度随机逼近新变体,利用 Ruppert-Polyak 平均技术估计耦合非线性算子,在强单调条件下实现最优 $O(1/k)$ 收敛速率,显著改进现有 $O(1/k^{2/3})$ 速率并应用于强化学习。

摘要:

本文提出了一种双时间尺度随机逼近的新变体,用于寻找两个耦合非线性算子的根,假设只能观察到这些算子的含噪样本。我们的核心思想是利用经典的 Ruppert-Polyak 平均技术,通过样本动态估计算子。这些平均步骤的估计值随后将用于双时间尺度随机逼近的更新中,以找到所需的解。我们的主要理论结果是证明在底层非线性算子的强单调条件下,所提方法生成的迭代序列的均方误差以最优速率 $O(1/k)$ 收敛于零,其中 $k$ 为迭代次数。我们的结果显著改进了双时间尺度随机逼近的现有结果,后者已知的最佳有限时间收敛速率为 $O(1/k^{2/3})$。我们通过将所提方法应用于开发具有改进性能的新强化学习算法来阐明这一结果。

英文摘要:

This paper proposes to develop a new variant of the two-time-scale stochastic approximation to find the roots of two coupled nonlinear operators, assuming only noisy samples of these operators can be observed. Our key idea is to leverage the classic Ruppert-Polyak averaging technique to dynamically estimate the operators through their samples. The estimated values of these averaging steps will then be used in the two-time-scale stochastic approximation updates to find the desired solution. Our main theoretical result is to show that under the strongly monotone condition of the underlying nonlinear operators the mean-squared errors of the iterates generated by the proposed method converge to zero at an optimal rate $O(1/k)$, where $k$ is the number of iterations. Our result significantly improves the existing result of two-time-scale stochastic approximation, where the best known finite-time convergence rate is $O(1/k^{2/3})$. We illustrate this result by applying the proposed method to develop new reinforcement learning algorithms with improved performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑