高斯ReLU网络的仿射几何：基于条件Kac-Rice公式

中文摘要 AI 辅助

我们研究了有限ReLU网络在随机初始化时如何产生仿射几何结构，以及监督训练如何重组这些结构。我们将隐藏层预激活的变号零点称为激活开关，将标量网络输出不可微的点称为标量扭结。对于一维输入，对前几层取条件使得每个预激活成为高斯过程，并在随机有限划分的单元上仿射。这导出了沿输入区间激活开关期望数量的精确有限宽度条件Kac-Rice公式。对于固定深度和按比例增长的宽度，所得的开关强度收敛到显式确定性极限。一个可见性估计表明，不产生标量扭结的开关期望数量可忽略不计，从而得出标量扭结期望数量及仿射区域的显式主导公式。在更高输入维度d≥2时，类似的条件曲面公式给出标量扭结集合的(d-1)维Hausdorff测度的主导期望值。在威斯康星乳腺癌数据集上，初始化公式准确预测了保留区间的开关计数。训练后，类内区间的开关计数减少，而类间区间的开关计数增加。因此，训练重新分配而非仅仅收缩仿射复杂度。

英文摘要

We study how the affine geometry of finite ReLU networks is created at random initialization and reorganized by supervised training. We call a sign-changing zero of a hidden preactivation an activation switch and a point where the scalar network output is nondifferentiable a scalar kink. For one-dimensional input, conditioning on the preceding layers makes each preactivation Gaussian and affine on the cells of a random finite partition. This yields an exact finite-width conditional Kac-Rice formula for the expected number of activation switches along an input interval. For fixed depth and proportionally growing widths, the resulting switch intensities converge to explicit deterministic limits. A visibility estimate shows that the expected number of switches that do not produce scalar kinks is negligible, yielding an explicit leading formula for the expected number of scalar kinks and hence affine regions. In higher input dimensions d >= 2, the analogous conditional surface formula yields the leading expected (d-1)-dimensional Hausdorff measure of the scalar kink set. On the Breast Cancer Wisconsin data, the initialization formula accurately predicts switch counts along held-out segments. After training, switch counts decrease along within-class segments and increase along between-class segments. Thus, training redistributes rather than merely contracts affine complexity.

发表机构

• Middle East Technical University（中东技术大学）
• Aarhus University（奥胡斯大学）

机构由 AI 辅助整理，请以论文原文为准。

补充信息

备注

40 pages, 3 figures