arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2310.01770cs.LGcs.AI
收藏

从损失平坦性到压缩神经表示的简单联系

A simple connection from loss flatness to compressed neural representations

Department of Applied Mathematics(应用数学系);Computational Neuroscience Center(计算神经科学中心);University of Washington(华盛顿大学);Technion, Israel Institute of Technology(技术离子以色列理工学院)

Shirui Chen, Stefano Recanatesi, Eric Shea-Brown

AI总结:

本文通过引入局部体积比、最大局部敏感度和局部维度等度量,揭示了损失平坦性与神经表示压缩之间的关系,为理解锐度与泛化能力的关系提供了新的理论框架。

摘要:

尽管已有广泛研究,局部极小点的锐度——即损失Hessian的迹——的意义仍然不明确。我们探讨了另一种视角:锐度如何与神经表示的几何结构相关,特别是定义为在局部输入扰动下神经激活如何集中化的表示压缩。我们引入了三个度量——局部体积比(LVR)、最大局部敏感度(MLS)和局部维度,并推导出上界,显示这些度量在数学上受锐度约束:更平坦的极小点必然限制压缩。我们将这些上界扩展到重参数化不变的锐度,并引入网络级变体(NMLS、NVR),这些变体提供了比先前单层分析更紧且更稳定的上界。经验上,我们在前馈、卷积和变换器架构中验证了一致的正相关性。我们的结果表明,锐度从根本上量化了表示压缩,为关于锐度-泛化关系的矛盾发现提供了系统性的解答。

英文摘要:

Despite extensive study, the significance of sharpness -- the trace of the loss Hessian at local minima -- remains unclear. We investigate an alternative perspective: how sharpness relates to the geometric structure of neural representations, specifically representation compression, defined as how strongly neural activations concentrate under local input perturbations. We introduce three measures -- Local Volumetric Ratio (LVR), Maximum Local Sensitivity (MLS), and Local Dimensionality -- and derive upper bounds showing these are mathematically constrained by sharpness: flatter minima necessarily limit compression. We extend these bounds to reparametrization-invariant sharpness and introduce network-wide variants (NMLS, NVR) that provide tighter, more stable bounds than prior single-layer analyses. Empirically, we validate consistent positive correlations across feedforward, convolutional, and transformer architectures. Our results suggest that sharpness fundamentally quantifies representation compression, offering a principled resolution to contradictory findings on the sharpness-generalization relationship.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑