Deep Incubation:通过分治训练大型模型
Deep Incubation: Training Large Models by Divide-and-Conquering
摘要:
近年来,大型深度学习模型取得了显着成功。然而,由于计算成本高、收敛速度慢和过拟合问题,训练这些模型具有挑战性。在本文中,我们提出了深度孵化,这是一种新的方法,通过将大型模型划分为可以单独训练和无缝组装的更小的子模块,从而实现高效和有效的大型模型训练。实现这一想法的一个关键挑战是确保独立训练的子模块的兼容性。为了解决这个问题,我们首先引入了一个全局的共享Meta模型,它被用来隐式地将所有模块链接在一起,并且可以被设计为一个非常小的网络,计算开销可以忽略不计。然后提出了一种模块孵化算法,该算法训练每个子模块替换Meta模型的相应组件,完成给定的学习任务。尽管简单,我们的方法有效地鼓励每个子模块意识到它在目标大模型中的作用,这样最终学习的子模块可以在组装后顺利地相互协作。从经验上讲,我们的方法在最终准确性和训练效率方面都优于端到端(E2 E)训练。例如,在ViT-Huge的基础上,它将ImageNet的准确率提高了2.7%,或者在训练时间减少4倍的情况下实现了类似的性能。值得注意的是,这些收益对于下游任务也很重要(例如,COCO和ADE 20 K上的目标检测和图像分割)。代码可在https://github.com/LeapLabTHU/Deep-Incubation上获得。
英文摘要:
Recent years have witnessed a remarkable success of large deep learning models. However, training these models is challenging due to high computational costs, painfully slow convergence, and overfitting issues. In this paper, we present Deep Incubation, a novel approach that enables the efficient and effective training of large models by dividing them into smaller sub-modules that can be trained separately and assembled seamlessly. A key challenge for implementing this idea is to ensure the compatibility of the independently trained sub-modules. To address this issue, we first introduce a global, shared meta model, which is leveraged to implicitly link all the modules together, and can be designed as an extremely small network with negligible computational overhead. Then we propose a module incubation algorithm, which trains each sub-module to replace the corresponding component of the meta model and accomplish a given learning task. Despite the simplicity, our approach effectively encourages each sub-module to be aware of its role in the target large model, such that the finally-learned sub-modules can collaborate with each other smoothly after being assembled. Empirically, our method outperforms end-to-end (E2E) training in terms of both final accuracy and training efficiency. For example, on top of ViT-Huge, it improves the accuracy by 2.7% on ImageNet or achieves similar performance with 4x less training time. Notably, the gains are significant for downstream tasks as well (e.g., object detection and image segmentation on COCO and ADE20K). Code is available at https://github.com/LeapLabTHU/Deep-Incubation.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。