arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1901.08244cs.LGstat.ML
收藏

Deepnet Hessians谱中异常点的三层层次结构度量

Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessians

Department of Statistics, Stanford University(斯坦福大学统计系)

Vardan Papyan

摘要:

我们考虑对神经网络进行深度分类。我们暴露的logits的导数相对于模型的参数,这是用来解释的海森谱中的离群值的存在的结构。以前的工作将Hessian分解为两个分量,将离群值归因于其中一个,即所谓的梯度协方差。我们证明这一项不是协方差,而是二阶矩矩阵,即,它受到梯度的影响。这些平均值具有加性双向结构,这是谱中离群值的来源。这种结构可以用来近似的主要子空间的海森使用某些“平均”操作,避免了需要高维特征分析。我们在不同的数据集、架构和样本量上证实了这一说法。

英文摘要:

We consider deep classifying neural networks. We expose a structure in the derivative of the logits with respect to the parameters of the model, which is used to explain the existence of outliers in the spectrum of the Hessian. Previous works decomposed the Hessian into two components, attributing the outliers to one of them, the so-called Covariance of gradients. We show this term is not a Covariance but a second moment matrix, i.e., it is influenced by means of gradients. These means possess an additive two-way structure that is the source of the outliers in the spectrum. This structure can be used to approximate the principal subspace of the Hessian using certain "averaging" operations, avoiding the need for high-dimensional eigenanalysis. We corroborate this claim across different datasets, architectures and sample sizes.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑