教师-学生树委员会机器中的平坦性-泛化关系

中文摘要 AI 辅助

损失景观在极小值处的平坦性是一种广泛用于推理神经网络泛化能力的启发式方法，然而支持这一关系的证据大多是经验性的且存在争议。我们在教师-学生树委员会机器中研究这一关系，其中在比例高维极限下，经验风险最小化（ERM）估计量和Hessian谱均可解析处理。首先，我们使用零温Gibbs公式来获得经验损失典型极小值可观测量（observables）的预测。其次，我们利用Edwards-Jones形式推导出这些典型极小值周围的极限Hessian预解式（resolvent）。所有预测均与有限尺寸梯度下降模拟结果一致。最后，我们研究了三种平坦性度量，即左、右边缘和谱均值，并检查当数据集大小增加时泛化误差的降低是否对应于平坦性的增加。我们发现答案强烈依赖于学习任务以及参数数量与数据点数量的比率。在回归中，谱均值和右边缘与泛化误差相关，而左边缘仅在过参数化区域中相关。在分类中，这种相关性仅在高度过参数化阶段可靠成立，而对于欠参数化网络，它甚至可能反转。

英文摘要

The flatness of the loss landscape at a minimizer is a widely used heuristic for reasoning about neural-network generalization, yet evidence for this relation is mostly empirical and controversial. We study this relation in a teacher-student tree committee machine, where both the ERM estimator and the Hessian spectrum are analytically tractable in the proportional high-dimensional limit. First, we use a zero-temperature Gibbs formulation to obtain predictions for the observables of the typical minimizers of the empirical loss. Secondly, we use Edwards-Jones formalism to derive the limiting Hessian resolvent around these typical minimizers. All predictions agree with finite-size gradient-descent simulations. Finally, we study three measures of flatness, namely the left and right edges and the spectral mean, and check if a decrease in generalization error as the dataset size is increased corresponds to an increase in flatness. We find that the answer strongly depends on the learning task and on the ratio of the number of parameters to the number of data points. In regression, the spectral mean and right edge correlate with the generalization error, while the left edge does so only in the overparametrized regime. In classification this correlation reliably holds only in the highly overparametrized phase, while for underparametrized networks it can even reverse.

发表机构

• ICTP Trieste（里雅斯特国际理论物理中心）
• Bocconi University（博科尼大学）
• Bocconi Institute for Data Science and Analytics (BIDSA)（博科尼数据科学与分析研究所（BIDSA））

机构由 AI 辅助整理，请以论文原文为准。