Deep neural networks (DNNs) must cater to a variety of users with different performance needs and budgets, leading to the costly practice of training, storing, and maintaining numerous user/task-specific models. There are solutions in the literature to deal with single dynamic or many-in-one models instead of many individual networks; however, they suffer from significant drops in performance, lack of generalization across different model architectures or different dimensions (e.g. depth, width, attention blocks), heavy model search requirements during training, and training a limited number of sub-models. To address these limitations, we propose SortedNet, a generalized and scalable training solution to harness the inherent modularity of DNNs. Thanks to a generalized nested architecture (which we refer as \textit{sorted} architecture in this paper) with shared parameters and its novel update scheme combining random sub-model sampling and a new gradient accumulation mechanism, SortedNet enables the training of sub-models simultaneously along with the training of the main model (without any significant extra training or inference overhead), simplifies dynamic model selection, customizes deployment during inference, and reduces the model storage requirement significantly. The versatility and scalability of SortedNet are validated through various architectures and tasks, including LLaMA, BERT, RoBERTa (NLP tasks), ResNet and MobileNet (image classification) demonstrating its superiority over existing dynamic training methods. For example, we introduce a novel adaptive self-speculative approach based on sorted-training to accelerate large language models decoding. Moreover, SortedNet is able to train 160 sub-models at once, achieving at least 96\% of the original model's performance.
Modular MeanFlow: Towards Stable and Scalable One-Step Generative Modeling
Graduate School of Arts and Sciences, Columbia University, New York, USA(哥伦比亚大学艺术与科学研究生院);Department of Computer Science, University of Warwick, Coventry, UK(沃里克大学计算机科学系)
Deep unfolding of MCMC kernels: scalable, modular & explainable GANs for high-dimensional posterior sampling
Maxwell Institute for Mathematical Sciences(麦克斯韦数学科学研究所);School of Mathematical and Computer Sciences, Heriot-Watt University(赫瑞-瓦特大学数学与计算机科学学院);School of Mathematics, University of Edinburgh(爱丁堡大学数学学院)
Modular Delta Merging with Orthogonal Constraints: A Scalable Framework for Continual and Reversible Model Composition
Rensselaer Polytechnic Institute(伦斯勒理工学院);National University of Sciences and Technology (NUST)(国立科技大学);Virginia Tech(弗吉尼亚理工大学);University of New Mexico(新墨西哥大学)
Exploring Sparse Adapters for Scalable Merging of Parameter Efficient Experts
McGill University(麦吉尔大学);Mila(Mila实验室);Microsoft(微软公司);Université de Montréal(蒙特利尔大学);ServiceNow(ServiceNow公司);Georgia Institute of Technology(佐治亚理工学院)