arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2008.11865cs.LGcs.CVstat.ML
收藏

类/跨类结构的痕迹遍布深度学习光谱

Traces of Class/Cross-Class Structure Pervade Deep Learning Spectra

Stanford University(斯坦福大学)

Vardan Papyan

摘要:

最近,许多研究人员将经验谱分析应用于现代深度学习分类器的研究。我们确定并讨论了一个重要的正式类/跨类结构,并展示了它是如何成为在深网光谱中观察到的许多视觉上引人注目的特征的起源的,其中一些在最近的文章中报道过,另一些则是首次在这里公布。这些包括光谱异常值,“尖峰”,和小但不同的连续分布,“凸点”,经常看到超出边缘的“主块”。跨类结构的意义在三个方面说明:(i)我们证明了Fisher信息矩阵的频谱中的离群值与体积的比率在多项逻辑回归的背景下是错误分类的预测;(ii)我们展示了如何随着深度的增加,网络能够将类区别性信息与类变异性分开,同时将类区别性信息正交化;以及(iii)我们提出了对KFAC的修正,KFAC是一种用于训练深度网络的众所周知的二阶优化算法。

英文摘要:

Numerous researchers recently applied empirical spectral analysis to the study of modern deep learning classifiers. We identify and discuss an important formal class/cross-class structure and show how it lies at the origin of the many visually striking features observed in deepnet spectra, some of which were reported in recent articles, others are unveiled here for the first time. These include spectral outliers, "spikes", and small but distinct continuous distributions, "bumps", often seen beyond the edge of a "main bulk". The significance of the cross-class structure is illustrated in three ways: (i) we prove the ratio of outliers to bulk in the spectrum of the Fisher information matrix is predictive of misclassification, in the context of multinomial logistic regression; (ii) we demonstrate how, gradually with depth, a network is able to separate class-distinctive information from class variability, all while orthogonalizing the class-distinctive information; and (iii) we propose a correction to KFAC, a well-known second-order optimization algorithm for training deepnets.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑