arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2601.17954cs.LGmath.PR
收藏

神经Actor-Critic算法中的标度效应与不确定性量化

Scaling Effects and Uncertainty Quantification in Neural Actor Critic Algorithms

Boston University(波士顿大学);University of Oxford(牛津大学)

Nikos Georgoudios, Konstantinos Spiliopoulos, Justin Sirignano

AI总结:

本研究探讨了神经Actor-Critic算法中不同标度方案对收敛性和不确定性量化的影响,提出了一种通用逆多项式标度方法,通过渐近展开分析网络输出的统计特性,并为超参数选择提供了理论指导。

摘要:

我们研究了使用浅层神经网络作为Actor和Critic模型的神经Actor-Critic算法。本研究的重点有两个方面:首先,比较在网络宽度和训练步数趋于无穷大时,不同标度方案下网络输出的收敛性质;其次,提供对每个标度 regime 的近似误差的精确控制。先前的工作显示,在网络宽度的逆平方根标度下,算法收敛到具有随机初始条件的常微分方程。在本工作中,我们从仅关注收敛速度转向对算法输出的更全面的统计特征刻画,旨在量化神经Actor-Critic方法的不确定性。具体来说,我们研究了网络宽度的通用逆多项式标度,其中指数作为可调超参数,取值严格在1/2和1之间。我们推导了网络输出的渐近展开,将其解释为统计估计量,以阐明其结构。在主导项上,我们显示方差随着网络宽度的幂次衰减,指数等于1/2减去标度参数,这表明随着标度参数接近1时,统计鲁棒性得到改善。数值实验支持了这种行为,并进一步表明这种标度选择具有更快的收敛速度。最后,我们的分析为选择算法超参数提供了具体的指导,包括学习率和探索率作为网络宽度和标度参数的函数,以确保可证明的有利统计行为。

英文摘要:

We investigate the neural Actor Critic algorithm using shallow neural networks for both the Actor and Critic models. The focus of this work is twofold: first, to compare the convergence properties of the network outputs under various scaling schemes as the network width and the number of training steps tend to infinity; and second, to provide precise control of the approximation error associated with each scaling regime. Previous work has shown convergence to ordinary differential equations with random initial conditions under inverse square root scaling in the network width. In this work, we shift the focus from convergence speed alone to a more comprehensive statistical characterization of the algorithm's output, with the goal of quantifying uncertainty in neural Actor Critic methods. Specifically, we study a general inverse polynomial scaling in the network width, with an exponent treated as a tunable hyperparameter taking values strictly between one half and one. We derive an asymptotic expansion of the network outputs, interpreted as statistical estimators, in order to clarify their structure. To leading order, we show that the variance decays as a power of the network width, with an exponent equal to one half minus the scaling parameter, implying improved statistical robustness as the scaling parameter approaches one. Numerical experiments support this behavior and further suggest faster convergence for this choice of scaling. Finally, our analysis yields concrete guidelines for selecting algorithmic hyperparameters, including learning rates and exploration rates, as functions of the network width and the scaling parameter, ensuring provably favorable statistical behavior.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑