To improve performance in contemporary deep learning, one is interested in scaling up the neural network in terms of both the number and the size of the layers. When ramping up the width of a single layer, graceful scaling of training has been linked to the need to normalize the weights and their updates in the "natural norm" particular to that layer. In this paper, we significantly generalize this idea by defining the modular norm, which is the natural norm on the full weight space of any neural network architecture. The modular norm is defined recursively in tandem with the network architecture itself. We show that the modular norm has several promising applications. On the practical side, the modular norm can be used to normalize the updates of any base optimizer so that the learning rate becomes transferable across width and depth. This means that the user does not need to compute optimizer-specific scale factors in order to scale training. On the theoretical side, we show that for any neural network built from "well-behaved" atomic modules, the gradient of the network is Lipschitz-continuous in the modular norm, with the Lipschitz constant admitting a simple recursive formula. This characterization opens the door to porting standard ideas in optimization theory over to deep learning. We have created a Python package called Modula that automatically normalizes weight updates in the modular norm of the architecture. The package is available via "pip install modula" with source code at https://github.com/jxbz/modula.
Modular MeanFlow: Towards Stable and Scalable One-Step Generative Modeling
Graduate School of Arts and Sciences, Columbia University, New York, USA(哥伦比亚大学艺术与科学研究生院);Department of Computer Science, University of Warwick, Coventry, UK(沃里克大学计算机科学系)
Deep unfolding of MCMC kernels: scalable, modular & explainable GANs for high-dimensional posterior sampling
Maxwell Institute for Mathematical Sciences(麦克斯韦数学科学研究所);School of Mathematical and Computer Sciences, Heriot-Watt University(赫瑞-瓦特大学数学与计算机科学学院);School of Mathematics, University of Edinburgh(爱丁堡大学数学学院)
Modular Delta Merging with Orthogonal Constraints: A Scalable Framework for Continual and Reversible Model Composition
Rensselaer Polytechnic Institute(伦斯勒理工学院);National University of Sciences and Technology (NUST)(国立科技大学);Virginia Tech(弗吉尼亚理工大学);University of New Mexico(新墨西哥大学)
Exploring Sparse Adapters for Scalable Merging of Parameter Efficient Experts
McGill University(麦吉尔大学);Mila(Mila实验室);Microsoft(微软公司);Université de Montréal(蒙特利尔大学);ServiceNow(ServiceNow公司);Georgia Institute of Technology(佐治亚理工学院)