arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2309.00255cs.LG
收藏

SortedNet:一个用于训练模块化深度神经网络的可扩展通用框架

SortedNet: A Scalable and Generalized Framework for Training Modular Deep Neural Networks

University of Waterloo(滑铁卢大学);Huawei Noah’s Ark Lab(华为诺亚方舟实验室)

Mojtaba Valipour, Mehdi Rezagholizadeh, Hossein Rajabzadeh, Parsa Kavehzadeh, Marzieh Tahaei, Boxing Chen, Ali Ghodsi

AI总结:

提出SortedNet框架,利用嵌套sorted架构与梯度累积机制,在无额外开销下同步训练主模型与子模型,实现动态选择、降低存储,并在多种架构任务上验证了其优越性。

摘要:

深度神经网络(DNNs)必须满足具有不同性能需求和预算的各类用户,这导致了训练、存储和维护大量用户/任务特定模型的昂贵做法。文献中存在一些解决方案,使用单一动态模型或多合一模型来替代多个独立网络;然而,这些方法存在性能显著下降、在不同模型架构或不同维度(如深度、宽度、注意力块)之间缺乏泛化能力、训练期间需要繁重的模型搜索,以及只能训练有限数量子模型等问题。为了解决这些局限性,我们提出了SortedNet,这是一种通用且可扩展的训练解决方案,旨在利用DNNs的固有模块化特性。得益于具有共享参数的通用嵌套架构(本文中称为“sorted”架构)以及结合了随机子模型采样和一种新梯度累积机制的新型更新方案,SortedNet能够在训练主模型的同时同步训练子模型(且没有任何显著的额外训练或推理开销),简化了动态模型选择,在推理过程中自定义部署,并显著降低了模型存储需求。SortedNet的通用性和可扩展性通过各种架构和任务得到了验证,包括LLaMA、BERT、RoBERTa(NLP任务),ResNet和MobileNet(图像分类),证明了其相对于现有动态训练方法的优越性。例如,我们引入了一种基于sorted训练的新型自适应自推测方法,以加速大语言模型解码。此外,SortedNet能够一次性训练160个子模型,并达到原始模型至少96%的性能。

英文摘要:

Deep neural networks (DNNs) must cater to a variety of users with different performance needs and budgets, leading to the costly practice of training, storing, and maintaining numerous user/task-specific models. There are solutions in the literature to deal with single dynamic or many-in-one models instead of many individual networks; however, they suffer from significant drops in performance, lack of generalization across different model architectures or different dimensions (e.g. depth, width, attention blocks), heavy model search requirements during training, and training a limited number of sub-models. To address these limitations, we propose SortedNet, a generalized and scalable training solution to harness the inherent modularity of DNNs. Thanks to a generalized nested architecture (which we refer as \textit{sorted} architecture in this paper) with shared parameters and its novel update scheme combining random sub-model sampling and a new gradient accumulation mechanism, SortedNet enables the training of sub-models simultaneously along with the training of the main model (without any significant extra training or inference overhead), simplifies dynamic model selection, customizes deployment during inference, and reduces the model storage requirement significantly. The versatility and scalability of SortedNet are validated through various architectures and tasks, including LLaMA, BERT, RoBERTa (NLP tasks), ResNet and MobileNet (image classification) demonstrating its superiority over existing dynamic training methods. For example, we introduce a novel adaptive self-speculative approach based on sorted-training to accelerate large language models decoding. Moreover, SortedNet is able to train 160 sub-models at once, achieving at least 96\% of the original model's performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑