直通估计器训练两层量化神经网络的稳定性与泛化性

中文摘要 AI 辅助

我们从统计学习理论（SLT）的角度，研究使用铰链损失训练两层二值激活网络的恒等直通估计器（STE）。我们的核心问题是，算法稳定性能否解释由不连续的STE训练规则产生的估计器的统计泛化性。在饱和输出区域，零初始化的样本级STE递归恰好是凸潜在损失$(-yu^\top x)_+$上的随机次梯度下降。这一表示使得稳定性分析成为可能。我们推导了两个耦合更新的精确距离恒等式，并证明了公共样本映射的近似非扩张性，仅当两个潜在间隔跨越零时出现二次缺陷。然后，我们获得了显式的$\ell_2$平均模型稳定性和泛化界，将稳定性等距地从潜在向量转移到完整的第一层矩阵。将稳定性与标准优化界相结合，得到了显式的超额诱导风险保证，并在$T=n^2$时得到速率$O(n^{-1/2})$。在间隔可分性条件下，一个补充论证给出了随机单遍STE迭代的最优阶$O(R^2/(\gamma^2n))$期望超额误分类误差，以及相应的多数投票界。

英文摘要

We study the identity straight-through estimator (STE) for training a two-layer binary-activation network with hinge loss from the perspective of Statistical Learning Theory (SLT). Our central question is whether algorithmic stability can explain the statistical generalization of the estimator produced by the discontinuous STE training rule. In the saturated-output regime, the zero-initialized samplewise STE recursion is exactly the stochastic subgradient descent on the convex latent loss $(-yu^\top x)_+$. This representation makes a stability analysis possible. We derive an exact distance identity for two coupled updates and prove approximate non-expansiveness of the common-example map, with a quadratic defect only when the two latent margins straddle zero. We then obtain explicit $\ell_2$ on-average model-stability and generalization bounds, transferring stability isometrically from the latent vector to the full first-layer matrix. Combining stability with a standard optimization bound yields an explicit excess induced-risk guarantee and the rate $O(n^{-1/2})$ when $T=n^2$. Under margin separability, a complementary argument gives the optimal-order $O(R^2/(γ^2n))$ expected excess misclassification error for a randomized one-pass STE iterate and a corresponding majority-vote bound.

发表机构

• University of Sydney（悉尼大学）

机构由 AI 辅助整理，请以论文原文为准。