非凸神经网络中带噪解的鲁棒性研究
On the robustness of noisy solutions in non-convex neural networks
浏览论文内容
中文总结 AI 辅助
本文研究非凸神经网络带噪解的鲁棒性，将重叠间隙性质（OGP）扩展至有限温度，证明有限能量区域在α_OGP外仍存在且泛化良好，热噪声可在计算困难的约束密度区实现有效泛化。
中文摘要 AI 辅助
非凸神经网络模型的优化受解空间几何结构的强烈影响：稀疏、孤立、点状的簇通常是算法无法访问的，而宽且平坦的区域尽管相对稀少却能被高效找到。在零温度下，这一图像已通过重叠间隙性质（OGP）在二元感知机中得到形式化，该性质限制了算法对超过临界约束密度α_OGP时零训练误差配置的访问。本文将该描述扩展至有限温度，此时允许正训练误差并对其进行统计惩罚。我们首先证明，主导零温度平衡测度的冻结一步复制对称破缺解在任意有限温度下均存在。此外，我们基于单模式吉布斯权重在决策边界附近的平滑性推导了一个通用判据，用于确定损失的有限温度松弛何时会消除冻结。随后，我们将OGP构造扩展至有限温度，证明有限能量配置的密集、算法可访问区域在α_OGP之外仍存在，直至阈值α_OGP(ε)，该阈值随允许的训练误差ε增大而增长。最后，在教师-学生设置中，我们证明这些宽的有限能量区域仍保持良好的泛化性能。通过有限能量消息传递算法，我们数值证明热噪声在约束密度区域实现了有效泛化，而该区域中恢复教师模型和寻找零温度解均计算困难。
英文摘要
Optimization in non-convex neural network models is strongly influenced by the geometry of the solution space: sparse, isolated, point-like clusters are typically algorithmically inaccessible, whereas wide and flat regions can be found efficiently despite being relatively rare. At zero temperature this picture has been formalized in binary perceptrons through the overlap gap property (OGP), which limits algorithmic access to configurations with zero training error above a critical constraint density $α_{\rm OGP}$. Here we extend this description to finite temperature, where a positive training error is allowed and statistically penalized. We first show that the frozen one-step replica-symmetry-breaking solution, dominating the zero temperature equilibrium measure, survives at any finite temperature. We furthermore derive a general criterion, based on the smoothness of the single-pattern Gibbs weight near the decision boundary, that determines when a finite-temperature relaxation of the loss removes freezing. We then extend the OGP construction to finite temperature and show that dense, algorithmically accessible regions of finite-energy configurations persist beyond $α_{\rm OGP}$, up to a threshold $α_{\rm OGP}(ε)$ that grows with the allowed training error $ε$. Finally, in the teacher-student setting, we show that these wide, finite-energy regions still retain good generalization. Using a finite energy message-passing algorithm, we demonstrate numerically that thermal noise enables effective generalization in the regime of constraint densities where both recovering the teacher and finding a zero temperature solution are computationally hard.
发表机构
- Bocconi University（博科尼大学）
- Bocconi Institute for Data Science and Analytics (BIDSA)（博科尼数据科学与分析研究所）
机构由 AI 辅助整理，请以论文原文为准。
