Collective variables of neural networks: empirical time evolution and scaling laws
Institute for Computational Physics(计算物理研究所);University of Stuttgart(斯图加特大学);Cavendish Laboratory(卡文迪什实验室);DAMTP(应用数学与理论物理系);University of Cambridge(剑桥大学);Institute for Particle Physics Phenomenology(粒子物理现象学研究所);Durham University(杜伦大学)
Samuel Tovey, Sven Krippendorf, Michael Spannowsky, Konstantin Nikolaou, Christian Holm
This work presents a novel means for understanding learning dynamics and scaling relations in neural networks. We show that certain measures on the spectrum of the empirical neural tangent kernel, specifically entropy and trace, yield insight into the representations learned by a neural network and how these can be improved through architecture scaling. These results are demonstrated first on test cases before being shown on more complex networks, including transformers, auto-encoders, graph neural networks, and reinforcement learning studies. In testing on a wide range of architectures, we highlight the universal nature of training dynamics and further discuss how it can be used to understand the mechanisms behind learning in neural networks. We identify two such dominant mechanisms present throughout machine learning training. The first, information compression, is seen through a reduction in the entropy of the NTK spectrum during training, and occurs predominantly in small neural networks. The second, coined structure formation, is seen through an increasing entropy and thus, the creation of structure in the neural network representations beyond the prior established by the network at initialization. Due to the ubiquity of the latter in deep neural network architectures and its flexibility in the creation of feature-rich representations, we argue that this form of evolution of the network's entropy be considered the onset of a deep learning regime.
Charting Empirical Laws for LLM Fine-Tuning in Scientific Multi-Discipline Learning
The University of Sydney(悉尼大学);Shanghai AI Laboratory(上海人工智能实验室);Edith Cowan University(埃德温·考文大学);University of Oxford(牛津大学);The Chinese University of Hong Kong(香港中文大学);Beihang University(北航大学)