具有离散和连续权重的非凸神经网络中的典型与非典型解
Typical and atypical solutions in non-convex neural networks with discrete and continuous weights
AI总结:
该研究分析二元与连续负边界感知机两种非凸神经网络的解景观,揭示次优平坦极小值与主导解的结构差异、约束密度阈值的影响,及平坦极小值对泛化性能的提升作用。
摘要:
我们将二元和连续负边界感知机(negative-margin perceptrons)作为学习随机规则与关联的简单非凸神经网络模型展开研究。我们分析了两种模型的解景观几何结构,发现了二者之间重要的相似性与差异。两种模型都呈现出极其平坦且宽阔的次优极小值。这些极小值与主导解背景共存:在二元模型(冻结1-RSB相)中,主导解由指数级数量的算法无法访问的小簇构成;在球形模型(全RSB相)中,主导解则是由不同尺寸的簇组成的层级结构。在两种情况下,当约束密度越过某一阈值时,宽阔平坦极小值的局部熵会变得非单调,表明鲁棒解空间会分裂为互不连通的分量。这对二元模型中的算法行为有强烈影响,算法无法访问剩余的孤立簇。球形模型的行为则有所不同,因为即使在宽阔平坦极小值消失之后,剩余的解在任意距离下(直至容量上限)始终被大量其他解环绕。实际上,我们提供了数值证据,表明算法似乎能在SAT/UNSAT相变之前找到解,我们在此使用1RSB近似计算了该相变点。对于两种模型,研究表明,即使在负边界极大的高度欠约束区域中训练,宽阔平坦极小值的存在也能极大提升其作为学习装置的泛化性能。
英文摘要:
We study the binary and continuous negative-margin perceptrons as simple non-convex neural network models learning random rules and associations. We analyze the geometry of the landscape of solutions in both models and find important similarities and differences. Both models exhibit subdominant minimizers which are extremely flat and wide. These minimizers coexist with a background of dominant solutions which are composed by an exponential number of algorithmically inaccessible small clusters for the binary case (the frozen 1-RSB phase) or a hierarchical structure of clusters of different sizes for the spherical case (the full RSB phase). In both cases, when a certain threshold in constraint density is crossed, the local entropy of the wide flat minima becomes non-monotonic, indicating a break-up of the space of robust solutions into disconnected components. This has a strong impact on the behavior of algorithms in binary models, which cannot access the remaining isolated clusters. For the spherical case the behaviour is different, since even beyond the disappearance of the wide flat minima the remaining solutions are shown to always be surrounded by a large number of other solutions at any distance, up to capacity. Indeed, we exhibit numerical evidence that algorithms seem to find solutions up to the SAT/UNSAT transition, that we compute here using an 1RSB approximation. For both models, the generalization performance as a learning device is shown to be greatly improved by the existence of wide flat minimizers even when trained in the highly underconstrained regime of very negative margins.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。