arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2102.00485cs.LGstat.ML
收藏

探索神经网络损失景观的几何和拓扑

Exploring the Geometry and Topology of Neural Network Loss Landscapes

Université de Montréal(蒙特利尔大学);Mila - Quebec Artificial Intelligence Institute(米拉-魁北克人工智能研究所);Yale University(耶鲁大学);Helmholtz Centre Munich(慕尼黑亥姆霍兹中心)

Stefan Horoi, Jessie Huang, Bastian Rieck, Guillaume Lajoie, Guy Wolf, Smita Krishnaswamy

摘要:

最近的工作已经建立了明确的联系之间的泛化性能的训练神经网络和几何形状的损失景观附近的局部极小值,他们收敛。这表明,定性和定量检查的损失景观的几何形状可以产生的见解神经网络泛化性能在训练过程中。为此,研究人员提出了通过使用简单的降维技术来可视化损失景观。然而,这种可视化方法受到其线性性质的限制,并且仅捕获一维或二维的特征,从而将损失景观的采样限制为线或平面。在这里,我们从三个方面扩展和改进这些。首先,我们提出了一个新的“跳跃和重新训练”的过程中采样的损失景观的相关部分。我们表明,由此产生的采样数据拥有更有意义的信息,网络的泛化能力。接下来,我们通过PHATE(一种轨迹和流形保持方法)对跳跃和再训练轨迹进行非线性降维,使我们能够可视化网络之间的差异,这些网络的泛化能力很好,而不是很差。最后,我们将PHATE轨迹与计算同源性表征相结合,以量化轨迹差异。

英文摘要:

Recent work has established clear links between the generalization performance of trained neural networks and the geometry of their loss landscape near the local minima to which they converge. This suggests that qualitative and quantitative examination of the loss landscape geometry could yield insights about neural network generalization performance during training. To this end, researchers have proposed visualizing the loss landscape through the use of simple dimensionality reduction techniques. However, such visualization methods have been limited by their linear nature and only capture features in one or two dimensions, thus restricting sampling of the loss landscape to lines or planes. Here, we expand and improve upon these in three ways. First, we present a novel "jump and retrain" procedure for sampling relevant portions of the loss landscape. We show that the resulting sampled data holds more meaningful information about the network's ability to generalize. Next, we show that non-linear dimensionality reduction of the jump and retrain trajectories via PHATE, a trajectory and manifold-preserving method, allows us to visualize differences between networks that are generalizing well vs poorly. Finally, we combine PHATE trajectories with a computational homology characterization to quantify trajectory differences.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑