arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.07451cs.LGphysics.comp-ph
收藏

神经网络的集合变量:经验时间演化与标度律

Collective variables of neural networks: empirical time evolution and scaling laws

Institute for Computational Physics(计算物理研究所);University of Stuttgart(斯图加特大学);Cavendish Laboratory(卡文迪什实验室);DAMTP(应用数学与理论物理系);University of Cambridge(剑桥大学);Institute for Particle Physics Phenomenology(粒子物理现象学研究所);Durham University(杜伦大学)

Samuel Tovey, Sven Krippendorf, Michael Spannowsky, Konstantin Nikolaou, Christian Holm

AI总结:

本文提出通过经验神经正切核谱的熵和迹来理解神经网络学习动力学,发现信息压缩和结构形成两种普遍机制,并将结构形成视为深度学习阶段的起始标志。

摘要:

这项工作提出了一种理解神经网络学习动力学和标度关系的新方法。我们展示了经验神经正切核(NTK)谱上的某些度量,特别是熵和迹,能够揭示神经网络所学到的表示以及如何通过架构缩放来改进这些表示。这些结果首先在测试案例上得到验证,随后在更复杂的网络上进行了展示,包括transformers、自编码器、图神经网络以及强化学习研究。在广泛的架构上进行测试后,我们强调了训练动力学的普遍性,并进一步讨论了如何利用它来理解神经网络背后的学习机制。我们识别出贯穿机器学习训练过程的两种主要机制。第一种是信息压缩,表现为训练过程中NTK谱熵的减少,主要发生在小型神经网络中。第二种被称为结构形成,表现为熵的增加,从而在神经网络初始化时建立的先验之外,在表示中创造出新的结构。由于后者在深度神经网络架构中普遍存在,且在创建特征丰富的表示方面具有灵活性,我们认为网络熵的这种演化形式应被视为深度学习阶段的开始。

英文摘要:

This work presents a novel means for understanding learning dynamics and scaling relations in neural networks. We show that certain measures on the spectrum of the empirical neural tangent kernel, specifically entropy and trace, yield insight into the representations learned by a neural network and how these can be improved through architecture scaling. These results are demonstrated first on test cases before being shown on more complex networks, including transformers, auto-encoders, graph neural networks, and reinforcement learning studies. In testing on a wide range of architectures, we highlight the universal nature of training dynamics and further discuss how it can be used to understand the mechanisms behind learning in neural networks. We identify two such dominant mechanisms present throughout machine learning training. The first, information compression, is seen through a reduction in the entropy of the NTK spectrum during training, and occurs predominantly in small neural networks. The second, coined structure formation, is seen through an increasing entropy and thus, the creation of structure in the neural network representations beyond the prior established by the network at initialization. Due to the ubiquity of the latter in deep neural network architectures and its flexibility in the creation of feature-rich representations, we argue that this form of evolution of the network's entropy be considered the onset of a deep learning regime.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑