关于随机牛顿算法及其加权平均版本的渐近收敛速率
On the asymptotic rate of convergence of Stochastic Newton algorithms and their Weighted Averaged versions
AI总结：
本文定义了一种通用的随机牛顿算法及其加权平均版本，通过直接更新逆Hessian估计来优化流式样本下的风险函数，并在温和假设下证明了其几乎必然收敛性和收敛速率，数值实验表明该方法在初始化较差时优于流行竞争方法。
摘要：
大多数机器学习方法可以被视为对一个不可用的风险函数进行最小化。为了优化后者，给定以流式方式提供的样本，我们定义了一种通用的随机牛顿算法及其加权平均版本。在几个用例中，将证明这两种实现都不需要在每次迭代时对 Hessian 估计求逆，而是倾向于直接更新逆 Hessian 的估计。这推广了文献[2]中针对逻辑回归特定情况引入的技巧，即直接更新逆 Hessian 的估计。在温和假设（如在最优点处局部强凸）下，我们建立了算法的几乎必然收敛性和收敛速率，以及所构建参数估计的中心极限定理。本文考虑的统一框架涵盖了线性回归、逻辑回归或 softmax 回归等情况。在模拟数据上的数值实验为所提方法的相关性提供了经验证据，这些方法优于流行的竞争方法，特别是在初始化较差的情况下。
英文摘要：
The majority of machine learning methods can be regarded as the minimization of an unavailable risk function. To optimize the latter, given samples provided in a streaming fashion, we define a general stochastic Newton algorithm and its weighted average version. In several use cases, both implementations will be shown not to require the inversion of a Hessian estimate at each iteration, but a direct update of the estimate of the inverse Hessian instead will be favored. This generalizes a trick introduced in [2] for the specific case of logistic regression, by directly updating the estimate of the inverse Hessian. Under mild assumptions such as local strong convexity at the optimum, we establish almost sure convergences and rates of convergence of the algorithms, as well as central limit theorems for the constructed parameter estimates. The unified framework considered in this paper covers the case of linear, logistic or softmax regressions to name a few. Numerical experiments on simulated data give the empirical evidence of the pertinence of the proposed methods, which outperform popular competitors particularly in case of bad initializa-tions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
