跳过Hessian，保持速度：全局化半光滑牛顿法与惰性Hessian更新
Skip the Hessian, Keep the Rates: Globalized Semismooth Newton with Lazy Hessian Updates
AI总结：
本文提出了一种无需每次迭代计算Hessian的半光滑牛顿方法，通过惰性更新策略实现高效的全局和超线性收敛，适用于具有非光滑导数的优化问题。
摘要：
二阶方法在理论上比一阶方法更快，且其在大规模优化问题中的高效实现引起了广泛关注。然而，机器学习中的优化问题往往具有非光滑导数，这使得现有二阶方法的收敛速率理论不再适用。在本文中，我们提出了一种新的半光滑牛顿方法（SSN），该方法在不要求二阶可微的情况下，既具有全局收敛速率又具有渐近超线性收敛速率。关键在于，我们的方法不需在每次迭代中计算（广义）Hessian，而是仅在周期性地计算，并在其他情况下重用过时的Hessian（即执行惰性Hessian更新），从而节省计算成本，通常在时间上带来显著加速，同时仍保持强的全局和局部收敛速率保证。我们是在无限维环境下发展了我们的理论，并通过矩阵分解和具有Lipschitz约束的神经网络的数值实验来展示。
英文摘要：
Second-order methods are provably faster than first-order methods, and their efficient implementations for large-scale optimization problems have attracted significant attention. Yet, optimization problems in ML often have nonsmooth derivatives, which makes the existing convergence rate theory of second-order methods inapplicable. In this paper, we propose a new semismooth Newton method (SSN) that enjoys both global convergence rates and asymptotic superlinear convergence without requiring second-order differentiability. Crucially, our method does not require (generalized) Hessians to be evaluated at each iteration but only periodically, and it reuses stale Hessians otherwise (i.e., it performs lazy Hessian updates), saving compute cost and often leading to significant speedups in time, whilst still maintaining strong global and local convergence rate guarantees. We develop our theory in an infinite-dimensional setting and illustrate it with numerical experiments on matrix factorization and neural networks with Lipschitz constraints.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
