更现实的非线性模型的Hessian本征谱
Hessian Eigenspectra of More Realistic Nonlinear Models
摘要:
给定一个优化问题,Hessian矩阵及其特征谱可以以多种方式使用,从设计更有效的二阶算法到执行模型分析和回归诊断。当考虑非线性模型和非凸问题时,为了使Hessian谱分析更易于处理,通常会做强简化假设。这就引出了一个问题,即这些分析的结论对于更现实的非线性模型有多大的相关性。在本文中,我们利用随机矩阵理论的确定性等价技术,使一个广泛的非线性模型,包括模型,推广经典的广义线性模型的海森特征谱的\emph{precise}表征,而不依赖于强简化假设先前使用。我们表明,根据数据属性、非线性响应模型和损失函数,Hessian可以具有\emph{qualitatively}不同的谱行为:有界或无界支持、单块或多块以及孤立的特征值在左侧或右侧的块。通过关注这样一个简单但不平凡的非线性模型,我们的分析向前迈出了一步,揭示了在更复杂的机器学习模型中观察到的许多视觉上引人注目的特征的理论起源。
英文摘要:
Given an optimization problem, the Hessian matrix and its eigenspectrum can be used in many ways, ranging from designing more efficient second-order algorithms to performing model analysis and regression diagnostics. When nonlinear models and non-convex problems are considered, strong simplifying assumptions are often made to make Hessian spectral analysis more tractable. This leads to the question of how relevant the conclusions of such analyses are for more realistic nonlinear models. In this paper, we exploit deterministic equivalent techniques from random matrix theory to make a \emph{precise} characterization of the Hessian eigenspectra for a broad family of nonlinear models, including models that generalize the classical generalized linear models, without relying on strong simplifying assumptions used previously. We show that, depending on the data properties, the nonlinear response model, and the loss function, the Hessian can have \emph{qualitatively} different spectral behaviors: of bounded or unbounded support, with single- or multi-bulk, and with isolated eigenvalues on the left- or right-hand side of the bulk. By focusing on such a simple but nontrivial nonlinear model, our analysis takes a step forward to unveil the theoretical origin of many visually striking features observed in more complex machine learning models.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。