用类热噪声绘制神经网络景观的地形图
Charting the Topography of the Neural Network Landscape with Thermal-Like Noise
AI总结:
本文利用Langevin动力学和涨落统计研究过参数化全连接网络在随机分类任务中的损失景观,发现低损失区域是维度由决策边界附近数据点数量控制的低维流形,并揭示二次近似失效的原因。
摘要:
神经网络的训练是一个复杂、高维、非凸且含噪的优化问题,其理论理解无论从应用角度还是基础原因来看都很有意义。一个核心挑战是理解引导优化的景观的几何与地形。在这项工作中,我们采用标准统计力学方法,即利用 Langevin 动力学进行相空间探索,来研究一个在随机数据上执行分类任务的过参数化全连接网络的这一景观。通过分析涨落统计,类比恒定温度下的热动力学,我们推断出低损失区域的清晰几何描述。我们发现它是一个低维流形,其维度可以很容易地从涨落中获得。此外,该维度由位于分类决策边界附近的数据点数量控制。重要的是,我们发现由于决策边界的指数性质和低损失区域的平坦性,在最小值附近对损失进行二次近似从根本上是不充分的。这导致动力学在较高温度下采样具有较高曲率的区域,同时在任意给定温度下产生类似二次的统计。我们通过一个简化损失模型来解释这一行为,该模型可解析求解,并能重现观测到的涨落统计。
英文摘要:
The training of neural networks is a complex, high-dimensional, non-convex and noisy optimization problem whose theoretical understanding is interesting both from an applicative perspective and for fundamental reasons. A core challenge is to understand the geometry and topography of the landscape that guides the optimization. In this work, we employ standard Statistical Mechanics methods, namely, phase-space exploration using Langevin dynamics, to study this landscape for an over-parameterized fully connected network performing a classification task on random data. Analyzing the fluctuation statistics, in analogy to thermal dynamics at a constant temperature, we infer a clear geometric description of the low-loss region. We find that it is a low-dimensional manifold whose dimension can be readily obtained from the fluctuations. Furthermore, this dimension is controlled by the number of data points that reside near the classification decision boundary. Importantly, we find that a quadratic approximation of the loss near the minimum is fundamentally inadequate due to the exponential nature of the decision boundary and the flatness of the low-loss region. This causes the dynamics to sample regions with higher curvature at higher temperatures, while producing quadratic-like statistics at any given temperature. We explain this behavior by a simplified loss model which is analytically tractable and reproduces the observed fluctuation statistics.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。