理解Hessian矩阵与梯度协方差矩阵的子空间稳定性
Understanding the Subspace Stabilization of the Hessian and Gradient Covariance Matrix
浏览论文内容
中文总结 AI 辅助
本文提出新的不稳定性度量，发现Hessian与梯度协方差矩阵顶部子空间稳定化独立于参数变化幅度，并通过类间类内分解及特征值分离解释该现象。
中文摘要 AI 辅助
Hessian矩阵顶部子空间的稳定化现象是研究神经网络训练二阶信息中一个令人惊讶且关键的方面。先前的工作通过测量逐步Hessian矩阵顶部子空间之间的重叠，认为Hessian矩阵的顶部子空间会稳定，并将这种稳定化归因于训练后期参数变化的减少。在本文中，我们为子空间演化定义了一个新的不稳定性度量，并用它来检测独立于参数变化幅度的子空间稳定化。同时，我们观察到梯度协方差矩阵的顶部子空间与Hessian矩阵具有相似的性质。通过使用梯度协方差矩阵的类间和类内分解，我们确定了一个显式形式，该形式对Hessian矩阵和梯度协方差矩阵的顶部(C-1)子空间给出了近乎完美的近似。在梯度流设置中，我们表明所识别近似的缓慢演化是由于Hessian矩阵的离群特征值与体特征值之间的分离，从而为Hessian矩阵顶部子空间的稳定化现象提供了解释。
英文摘要
The phenomenon of the top subspace stabilization of the Hessian matrix is an surprising and critical aspect in study of the second-order information of neural network training. Prior work argues that the top subspace of the Hessian stabilizes by measuring the overlap between the top subspaces of the step-wise Hessian, and explains this stabilization with diminishing parameter change in the late phase of training. In this paper, we define a new instability metric for the subspace evolution, and use it to detect subspace stabilization that is independent of the magnitude of parameter change. In the meantime, we observe that the gradient covariance matrix has a similar property of its top subspace to the Hessian. By using a between-class and within-class decomposition of the gradient covariance matrix, we identify an explicit form that gives a near-perfect approximation of the top-$(C-1)$ subspace of the Hessian and the gradient covariance matrix. In the gradient flow set-up, we show that the slow evolution of the idenfied approximation is due to the separation between the outlier and the bulk eigenvalues of the Hessian matrix, thus providing an explanation to the phenomenon of the top subspace stabilization of the Hessian matrix.
