平坦性与泛化：使用齐次神经网络学习多指标模型
Flatness and Generalization: Learning Multi-Index Models with Homogeneous Neural Networks
AI总结：
本文研究两层齐次神经网络学习多指标模型时，平坦性与泛化之间的关系，证明最平坦插值器总能泛化，而某些非泛化插值器的平坦性无法接近最平坦值。
摘要：
用于解释一阶梯度方法在非凸神经网络上泛化能力的常见启发式方法是“平坦插值器泛化良好”（Hochreiter and Schmidhuber, 1994; Keskar et al., 2017），其中平坦性可通过经验损失Hessian矩阵的迹来衡量。然而，Dinh等人（2017）表明，利用网络的对称性（可在保持总体和经验损失不变的情况下改变平坦性），任何插值器都可以变得更尖锐或更平坦。这一结果使得之前的启发式陈述变得空洞。在本文中，我们表明，对于使用两层非凸齐次神经网络学习未知多指标模型，尽管存在对称性，平坦性与泛化之间仍存在联系。这种联系涉及“最平坦”插值器，即所有插值器中具有阶数最小平坦性的插值器。首先，我们证明存在一类自然的非泛化插值器，其平坦性即使利用对称性也无法接近最平坦可能值。其次，我们证明，对于由单指标模型之和生成的数据，如果近似误差和标签噪声较低，任何最平坦插值器都能实现较小的总体损失，即最平坦插值器总是泛化的。这建立了平坦性与泛化之间的直接联系，适用于一大类激活函数和现实数据分布。
英文摘要：
A common heuristic used to explain the generalization of first-order gradient methods on non-convex neural networks is that "flat interpolators generalize well" (Hochreiter and Schmidhuber, 1994; Keskar et al., 2017), where flatness can be measured by the trace of the Hessian of the empirical loss. However, Dinh et al. 2017) showed that, using symmetry of the network that can change flatness while keeping the population and empirical losses unchanged, any interpolator can be made sharper or flatter. This result makes the earlier heuristic statement vacuous. In this paper, we show that for learning an unknown multi-index model with $2$-layer non-convex homogeneous neural networks, there is a connection between flatness and generalization, despite the existence of symmetries. This connection pertains to the "flattest" interpolators, i.e., the interpolators that have orderwise minimum flatness among all interpolators. First, we show that there exists a natural class of non-generalizing interpolators whose flatness cannot be made closer to the flattest possible, even using symmetries. Second, we show that for data generated by a sum of single-index models, if the approximation error and label noise are low, any flattest interpolator achieves small population loss, i.e., the flattest interpolators always generalize. This establishes a direct link between flatness and generalization which applies to a large class of activations and realistic data distributions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
