arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2504.17618cs.LGcs.CV
收藏

Hessian特征值谱密度类型对Hessian分析在神经网络泛化能力评估中适用性的影响

The effects of Hessian eigenvalue spectral density type on the applicability of Hessian analysis to generalization capability assessment of neural networks

Joint Stock “Research and production company “Kryptonite”(“Kryptonite”股份公司“科研生产公司”)

Nikita Gabdullin

AI总结:

本文研究Hessian特征值谱密度类型对神经网络泛化能力评估的影响,提出统一分析方法,并揭示准奇异谱密度的出现及其对传统假设的挑战。

摘要:

神经网络(NN)的Hessian矩阵包含关于NN损失景观曲率的关键信息,可用于估计NN的泛化能力。我们先前提出了基于观察的泛化准则,即对于广泛类别的NN,Hessian特征值谱密度(HESD)表现出相似行为。本文通过研究可能导致不同类型HESD的因素,进一步探讨了这些准则的适用性。我们进行了广泛的实验,表明在使用不同优化器、不同数据集以及不同预处理和增强流程进行NN训练和微调时,HESD主要具有正特征值(MP-HESD)。我们还表明,主要负HESD(MN-HESD)是外部梯度操纵的结果,表明先前提出的Hessian分析方法在此类情况下不适用。我们还提出了确定HESD类型和估计NN泛化潜力的准则及相应条件。这些HESD类型与先前提出的泛化准则被整合为一个统一的HESD分析方法。最后,我们讨论了训练过程中HESD的变化,并展示了准奇异(QS)HESD的出现及其对提出的方法以及对Hessian特征值与NN损失景观曲率之间关系的传统假设的影响。

英文摘要:

Hessians of neural network (NN) contain essential information about the curvature of NN loss landscapes which can be used to estimate NN generalization capabilities. We have previously proposed generalization criteria that rely on the observation that Hessian eigenvalue spectral density (HESD) behaves similarly for a wide class of NNs. This paper further studies their applicability by investigating factors that can result in different types of HESD. We conduct a wide range of experiments showing that HESD mainly has positive eigenvalues (MP-HESD) for NN training and fine-tuning with various optimizers on different datasets with different preprocessing and augmentation procedures. We also show that mainly negative HESD (MN-HESD) is a consequence of external gradient manipulation, indicating that the previously proposed Hessian analysis methodology cannot be applied in such cases. We also propose criteria and corresponding conditions to determine HESD type and estimate NN generalization potential. These HESD types and previously proposed generalization criteria are combined into a unified HESD analysis methodology. Finally, we discuss how HESD changes during training, and show the occurrence of quasi-singular (QS) HESD and its influence on the proposed methodology and on the conventional assumptions about the relation between Hessian eigenvalues and NN loss landscape curvature.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑