The geometrical features of the (non-convex) loss landscape of neural network models are crucial in ensuring successful optimization and, most importantly, the capability to generalize well. While minimizers' flatness consistently correlates with good generalization, there has been little rigorous work in exploring the condition of existence of such minimizers, even in toy models. Here we consider a simple neural network model, the symmetric perceptron, with binary weights. Phrasing the learning problem as a constraint satisfaction problem, the analogous of a flat minimizer becomes a large and dense cluster of solutions, while the narrowest minimizers are isolated solutions. We perform the first steps toward the rigorous proof of the existence of a dense cluster in certain regimes of the parameters, by computing the first and second moment upper bounds for the existence of pairs of arbitrarily close solutions. Moreover, we present a non rigorous derivation of the same bounds for sets of $y$ solutions at fixed pairwise distances.
Quantum Approximate Optimization Algorithm applied to the binary perceptron
SISSA(国际高等研究学院);Universität Innsbruck(因斯布鲁克大学);Bocconi University(博科尼大学);International Centre for Theoretical Physics (ICTP)(国际理论物理中心);CNR-IOM, Consiglio Nazionale delle Ricerche - Istituto Officina dei Materiali(意大利国家研究委员会-材料研究所)
The Symmetric Perceptron: a Teacher-Student Scenario
Institute for Cross-disciplinary Physics and Complex Systems IFISC (CSIC-UIB), Campus Universitat Illes Balears, 07122 Palma de Mallorca, Spain(跨学科物理与复杂系统研究所 IFISC(CSIC-UIB),巴利阿里大学校区,西班牙帕尔马);Université Paris-Saclay, CNRS, INRIA Tau team, LISN, 91190 Gif-sur-Yvette, France(巴黎-萨克雷大学,法国国家科学研究中心,INRIA Tau团队,LISN,法国吉夫-苏尔-伊维特);Indian Institute of Science Education and Research, Bhopal 462066, India(印度科学教育与研究院,博帕尔)