AI 中文总结
本文提出SGD及其变体的精确信息论恒等式，将单步遗憾分解为内在时间与比较器信息项，统一解释收敛、逃逸、泛化等现象，并在真实网络上区分优化器。
AI 中文摘要
作为标准几何分析的替代方案，我们给出了随机梯度下降（SGD）及其变体的精确信息论分析。我们证明，预条件SGD步骤是高斯贝叶斯模型的后验均值更新，并且其单步遗憾分解为固有时间成本和比较器信息的变化。该分解扩展到目标函数本身的恒等式。凸收敛、严格鞍点逃逸、平坦性与泛化之间的联系、标准学习率调度、自适应优化器以及SGD的噪声、动量、重尾和无梯度变体，均对应于该恒等式中的某一项或特例。我们在合成数据和真实训练运行中测量其各项。在真实网络上，它将经典收敛界的松弛归因于其推导中丢弃的项，并区分达到相同训练损失的优化器。这种区分遵循其步数的数量和一致性。它与哪些优化器泛化更好的关系在不同网络上有所不同。对于无梯度SGD，该恒等式决定了曲率预条件器应如何进入更新。它产生的基于锐度的泛化证书，在数据无关的各向同性先验下，除非曲率谱在所有参数上几乎平坦，否则在网络规模上是空洞的。
英文摘要
As an alternative to the standard geometric analyses, we give an exact, information-theoretic analysis of stochastic gradient descent (SGD) and its variants. We show that a preconditioned SGD step is the posterior-mean update of a Gaussian Bayes model, and that its one-step regret splits into an intrinsic-time cost and a change in comparator information. The split extends to an identity for the objective itself. Convex convergence, strict-saddle-point escape, the link between flatness and generalization, the standard learning-rate schedules, adaptive optimizers, and the noisy, momentum, heavy-tailed, and gradient-free variants of SGD each correspond to a term or a special case of this identity. We measure its terms on synthetic and real training runs. On real networks it attributes the slack of classical convergence bounds to the terms their derivations drop and separates optimizers that reach the same training loss. That separation follows the number and consistency of their steps. Its relation to which of them generalizes better differs between networks. For gradient-free SGD the identity determines how a curvature preconditioner should enter the update. The sharpness-based generalization certificate it yields, with a data-independent isotropic prior, is vacuous at network scale unless the curvature spectrum is nearly flat across all parameters.
