Loss landscape analysis is extremely useful for a deeper understanding of the generalization ability of deep neural network models. In this work, we propose a layerwise loss landscape analysis where the loss surface at every layer is studied independently and also on how each correlates to the overall loss surface. We study the layerwise loss landscape by studying the eigenspectra of the Hessian at each layer. In particular, our results show that the layerwise Hessian geometry is largely similar to the entire Hessian. We also report an interesting phenomenon where the Hessian eigenspectrum of middle layers of the deep neural network are observed to most similar to the overall Hessian eigenspectrum. We also show that the maximum eigenvalue and the trace of the Hessian (both full network and layerwise) reduce as training of the network progresses. We leverage on these observations to propose a new regularizer based on the trace of the layerwise Hessian. Penalizing the trace of the Hessian at every layer indirectly forces Stochastic Gradient Descent to converge to flatter minima, which are shown to have better generalization performance. In particular, we show that such a layerwise regularizer can be leveraged to penalize the middlemost layers alone, which yields promising results. Our empirical studies on well-known deep nets across datasets support the claims of this work
We show that the input correlation matrix of typical classification datasets has an eigenspectrum where, after a sharp initial drop, a large number of small eigenvalues are distributed uniformly over an exponentially large range. This structure is mirrored in a network trained on this data: we show that the Hessian and the Fisher Information Matrix (FIM) have eigenvalues that are spread uniformly over exponentially large ranges. We call such eigenspectra "sloppy" because sets of weights corresponding to small eigenvalues can be changed by large magnitudes without affecting the loss. Networks t…
Hessian captures important properties of the deep neural network loss landscape. Previous works have observed low rank structure in the Hessians of neural networks. In this paper, we propose a decoupling conjecture that decomposes the layer-wise Hessians of a network as the Kronecker product of two smaller matrices. We can analyze the properties of these smaller matrices and prove the structure of top eigenspace random 2-layer networks. The decoupling conjecture has several other interesting implications - top eigenspaces for different models have surprisingly high overlap, and top eigenvector…
Stochastic gradient descent (SGD) is widely used in deep learning due to its computational efficiency, but a complete understanding of why SGD performs so well remains a major challenge. It has been observed empirically that most eigenvalues of the Hessian of the loss functions on the loss landscape of over-parametrized deep neural networks are close to zero, while only a small number of eigenvalues are large. Zero eigenvalues indicate zero diffusion along the corresponding directions. This indicates that the process of minima selection mainly happens in the relatively low-dimensional subspace…
A Teacher-Student Perspective on the Dynamics of Learning Near the Optimal Point
CeFEMA-LaPMET & CAMGSD & IT（CeFEMA-LaPMET与CAMGSD及IT）；Instituto Superior Técnico University of Lisbon, Portugal（里斯本大学技术学院）；CAMGSD；Instituto de Telecomunicações（电信研究所）；CeFEMA-LaPMET
He et al.（2015）提出的残差网络（ResNet）利用快捷连接显著降低了训练难度，从而在训练和泛化误差方面都有很大的性能提升。He等人（2015）根据经验观察到，使用快捷方式2堆叠更多层的残差块会导致更小的训练误差，而对于长度为1或3的快捷方式则不成立。我们提供了一个理论解释的唯一性的捷径2。我们证明了，无论有无非线性，通过添加深度为2的快捷方式，损失函数在零初始点处的Hessian的条件数是深度不变的，这使得训练非常深的模型并不比浅的模型更困难。更高深度的捷径最初会导致非常平坦的（高阶）稳定点，优化算法很难逃脱。然而，快捷方式1基本上等同于没有快捷方式，其具有随着层数的增加而爆炸到无穷大的条件数。我们进一步认为，由于层数趋于无穷大，它足以只看在零初始点的损失函数。广泛的实验提供伴随我们的理论结果。我们表明，使用捷径2将网络初始化为小权重比随机高斯（Xavier）初始化，正交初始化和更深深度的捷径取得了更好的结果，从最终损失，学习动态和稳定性，到Hessian的行为等各个角度来看。
University of California, Berkeley（加州大学伯克利分校）；IBM T. J. Watson Research Center（IBM T. J. 沃森研究中心）
Adversarial training (AT) has become a popular choice for training robust networks. However, it tends to sacrifice clean accuracy heavily in favor of robustness and suffers from a large generalization error. To address these concerns, we propose Smooth Adversarial Training (SAT), guided by our analysis on the eigenspectrum of the loss Hessian. We find that curriculum learning, a scheme that emphasizes on starting "easy" and gradually ramping up on the "difficulty" of training, smooths the adversarial loss landscape for a suitably chosen difficulty metric. We present a general formulation for c…
University of Pennsylvania（宾夕法尼亚大学）；Universidad Politécnica de Madrid（马德里理工大学）；Google Inc.（谷歌公司）
We show that learning can be improved by using loss functions that evolve cyclically during training to emphasize one class at a time. In underparameterized networks, such dynamical loss functions can lead to successful training for networks that fail to find a deep minima of the standard cross-entropy loss. In overparameterized networks, dynamical loss functions can lead to better generalization. Improvement arises from the interplay of the changing loss landscape with the dynamics of the system as it evolves to minimize the loss. In particular, as the loss function oscillates, instabilities …
Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapes
Peking University（北京大学）；Beijing Institute of Big Data Research（北京大数据研究院）；Center for Data Science, Peking University（北京大学数据科学中心）；BICMR, Peking University（北京大学北京国际数学研究中心）；Princeton University（普林斯顿大学）
Influence functions approximate the effect of training samples in test-time predictions and have a wide variety of applications in machine learning interpretability and uncertainty estimation. A commonly-used (first-order) influence function can be implemented efficiently as a post-hoc method requiring access only to the gradients and Hessian of the model. For linear models, influence functions are well-defined due to the convexity of the underlying loss function and are generally accurate even across difficult settings where model changes are fairly large such as estimating group influences. …
