arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2010.04261cs.LGcs.NEstat.ML
收藏

剖析Hessian:理解神经网络中Hessian的常见结构

Dissecting Hessian: Understanding Common Structure of Hessian in Neural Networks

Princeton University(普林斯顿大学);Duke University(杜克大学)

Yikai Wu, Xingyu Zhu, Chenwei Wu, Annie Wang, Rong Ge

摘要:

Hessian捕捉了深度神经网络损失景观的重要属性。以前的工作已经观察到低秩结构的神经网络的海森。在本文中,我们提出了一个解耦猜想,分解的两个较小的矩阵的克罗内克积的网络的逐层海森。我们可以分析这些小矩阵的性质,并证明顶部特征空间随机2层网络的结构。解耦猜想还有其他几个有趣的含义-不同模型的顶部特征空间具有惊人的高重叠,并且顶部特征向量形成低秩矩阵时,它们被重塑为与相应的权重矩阵相同的形状。所有这些都可以在更深层次的网络中得到经验验证。最后,我们利用逐层Hessian的结构得到了神经网络更好的显式泛化界。

英文摘要:

Hessian captures important properties of the deep neural network loss landscape. Previous works have observed low rank structure in the Hessians of neural networks. In this paper, we propose a decoupling conjecture that decomposes the layer-wise Hessians of a network as the Kronecker product of two smaller matrices. We can analyze the properties of these smaller matrices and prove the structure of top eigenspace random 2-layer networks. The decoupling conjecture has several other interesting implications - top eigenspaces for different models have surprisingly high overlap, and top eigenvectors form low rank matrices when they are reshaped into the same shape as the corresponding weight matrix. All of these can be verified empirically for deeper networks. Finally, we use the structure of layer-wise Hessian to get better explicit generalization bounds for neural networks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑