浅ReLU模型中Hessian的解析刻画：对称性的故事
Analytic Characterization of the Hessian in Shallow ReLU Models: A Tale of Symmetry
摘要：
我们考虑与拟合双层ReLU网络相关的优化问题，其中标签由目标网络生成。我们利用丰富的对称性结构，分析表征的Hessian在各种家庭的虚假最小值的自然政权的输入$d$和隐藏的神经元$k$的数量是有限的。特别地，我们证明了对于$d\ge k$标准高斯输入：（a）的$dk$特征值的Hessian，$dk - O(d)$集中在零附近，（b）$Ω(d)$的特征值与$k$线性增长。虽然这种极其扭曲的光谱现象以前已经被观察过很多次，但据我们所知，这是第一次被严格地确立。我们的分析方法使用的技术，新的领域，从对称性破缺和表示理论，并进行重要的影响，我们的能力，通过局部曲率的统计推广。
英文摘要：
We consider the optimization problem associated with fitting two-layers ReLU networks with respect to the squared loss, where labels are generated by a target network. We leverage the rich symmetry structure to analytically characterize the Hessian at various families of spurious minima in the natural regime where the number of inputs $d$ and the number of hidden neurons $k$ is finite. In particular, we prove that for $d\ge k$ standard Gaussian inputs: (a) of the $dk$ eigenvalues of the Hessian, $dk - O(d)$ concentrate near zero, (b) $Ω(d)$ of the eigenvalues grow linearly with $k$. Although this phenomenon of extremely skewed spectrum has been observed many times before, to our knowledge, this is the first time it has been established {rigorously}. Our analytic approach uses techniques, new to the field, from symmetry breaking and representation theory, and carries important implications for our ability to argue about statistical generalization through local curvature.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
