参数对称性与梯度流中的守恒律

中文摘要 AI 辅助

参数空间对称性和守恒律在理解神经网络的损失景观和隐式偏差中起着重要作用。受物理学中诺特定理的启发，先前的工作试图从参数对称性推导出梯度流下的守恒律，但这种联系的适用范围和局限性仍不清楚。我们开发了一个统一的几何框架，阐明了这两个概念之间的精确关系，包括对称性对应于守恒律的条件。我们引入了组合可辨识性的概念，并利用它建立了一个通用继承原理，用于多层网络中对称性和守恒律的完整刻画。我们将该框架应用于多头注意力、分组查询注意力、多项式神经网络和平方深度线性网络。

英文摘要

Parameter space symmetries and conservation laws play an important role in understanding the loss landscapes and implicit biases of neural networks. Inspired by Noether's theorem in physics, prior works have sought to derive conservation laws under gradient flow from parameter symmetries, but the scope and limitations of this connection remain unclear. We develop a unified geometric framework that clarifies the precise relationship between the two notions, including the conditions under which symmetries correspond to conservation laws. We introduce a notion of compositional identifiability and use it to establish a general inheritance principle for complete characterizations of symmetries and conservation laws in multilayer networks. We apply the framework to multi-head and grouped-query attention, polynomial neural networks, and square deep linear networks.

发表机构

• UCLA（加州大学洛杉矶分校）
• MPI MiS（马克斯·普朗克数学研究所）

机构由 AI 辅助整理，请以论文原文为准。