Noether：The More Things Change，the More Stay the Same（诺特：事物变化越多，就越保持不变）
Noether: The More Things Change, the More Stay the Same
摘要：
对称性已被证明是神经网络分析中的重要成分。到目前为止，它们的使用大多是隐含的或看似巧合的。我们对对称性所起的作用进行了系统的研究。特别是，我们阐明了对称性如何与学习算法相互作用。我们研究中的关键因素是诺特著名的定理，非正式地说，该定理指出对称性导致守恒量（例如，能量守恒或动量守恒）。在梯度下降下的神经网络领域中，模型对称性意味着对梯度路径的限制。例如，我们表明，激活函数的对称性导致权重矩阵的有界性，对于线性激活的特定情况，它导致连续层的平衡方程，数据增强导致具有“动量”型限制的梯度路径，并且时间对称性导致神经切线核的版本。对称性本身并不能确定优化路径，但模型中包含的对称性越多，对优化路径的限制就越多。由于对称性也意味着过度参数化，这实际上意味着这种过度参数化的一部分被守恒量的存在所抵消。因此，对称性可以被认为是理解神经网络在梯度下降下的性能的另一个重要工具。
英文摘要：
Symmetries have proven to be important ingredients in the analysis of neural networks. So far their use has mostly been implicit or seemingly coincidental. We undertake a systematic study of the role that symmetry plays. In particular, we clarify how symmetry interacts with the learning algorithm. The key ingredient in our study is played by Noether's celebrated theorem which, informally speaking, states that symmetry leads to conserved quantities (e.g., conservation of energy or conservation of momentum). In the realm of neural networks under gradient descent, model symmetries imply restrictions on the gradient path. E.g., we show that symmetry of activation functions leads to boundedness of weight matrices, for the specific case of linear activations it leads to balance equations of consecutive layers, data augmentation leads to gradient paths that have "momentum"-type restrictions, and time symmetry leads to a version of the Neural Tangent Kernel. Symmetry alone does not specify the optimization path, but the more symmetries are contained in the model the more restrictions are imposed on the path. Since symmetry also implies over-parametrization, this in effect implies that some part of this over-parametrization is cancelled out by the existence of the conserved quantities. Symmetry can therefore be thought of as one further important tool in understanding the performance of neural networks under gradient descent.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
