IQN：一种具有局部超线性收敛速率的增量拟牛顿法
IQN: An Incremental Quasi-Newton Method with Local Superlinear Convergence Rate
摘要：
考虑了极小化一个目标函数的问题，该目标函数可表示为一组$n$光滑强凸函数之和。本文提出的增量拟牛顿（IQN）方法属于随机增量方法族，其每次迭代的成本与$n$无关。IQN迭代是BFGS迭代的随机版本，它使用内存来减少随机逼近的方差。IQN的收敛性质弥补了确定性和随机拟牛顿法之间的差距。确定性拟牛顿法利用了使用目标梯度差来近似牛顿步骤的可能性。他们是有吸引力的，因为他们有一个较小的计算成本，每次迭代相对于牛顿的方法，并实现了超线性收敛速率下，通常的规律性假设。随机拟牛顿法利用随机梯度差代替实际梯度差。这使得它们每次迭代的计算成本与目标函数$n$的数量无关。然而，现有的随机拟牛顿法至多具有次线性或线性收敛。IQN是第一个随机拟牛顿法证明超线性收敛的最优解的局部邻域。IQN在三个方面不同于最先进的增量拟牛顿法：（i）使用变量、梯度和拟牛顿海森近似矩阵的聚合信息来减少梯度和海森近似的噪声。(ii)用泰勒展开式对每个函数进行的近似，其中线性项和二次项都是相对于同一根线计算的。(iii)使用循环方案来更新函数以代替随机选择例程。我们利用IQN的这些基本性质来建立它的局部超线性收敛速率。
英文摘要：
The problem of minimizing an objective that can be written as the sum of a set of $n$ smooth and strongly convex functions is considered. The Incremental Quasi-Newton (IQN) method proposed here belongs to the family of stochastic and incremental methods that have a cost per iteration independent of $n$. IQN iterations are a stochastic version of BFGS iterations that use memory to reduce the variance of stochastic approximations. The convergence properties of IQN bridge a gap between deterministic and stochastic quasi-Newton methods. Deterministic quasi-Newton methods exploit the possibility of approximating the Newton step using objective gradient differences. They are appealing because they have a smaller computational cost per iteration relative to Newton's method and achieve a superlinear convergence rate under customary regularity assumptions. Stochastic quasi-Newton methods utilize stochastic gradient differences in lieu of actual gradient differences. This makes their computational cost per iteration independent of the number of objective functions $n$. However, existing stochastic quasi-Newton methods have sublinear or linear convergence at best. IQN is the first stochastic quasi-Newton method proven to converge superlinearly in a local neighborhood of the optimal solution. IQN differs from state-of-the-art incremental quasi-Newton methods in three aspects: (i) The use of aggregated information of variables, gradients, and quasi-Newton Hessian approximation matrices to reduce the noise of gradient and Hessian approximations. (ii) The approximation of each individual function by its Taylor's expansion in which the linear and quadratic terms are evaluated with respect to the same iterate. (iii) The use of a cyclic scheme to update the functions in lieu of a random selection routine. We use these fundamental properties of IQN to establish its local superlinear convergence rate.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
