arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2303.00980cs.LG
收藏

学习增长预训练模型以实现高效的Transformer训练

Learning to Grow Pretrained Models for Efficient Transformer Training

University of Texas at Austin(德克萨斯大学奥斯汀分校);MIT-IBM Watson AI Lab(麻省理工学院-IBM沃森人工智能实验室);Columbia University(哥伦比亚大学);MIT(麻省理工学院)

Peihao Wang, Rameswar Panda, Lucas Torroba Hennigen, Philip Greengard, Leonid Karlinsky, Rogerio Feris, David Daniel Cox, Zhangyang Wang, Yoon Kim

摘要:

缩放Transformers在许多领域取得了重大突破,导致了一种范式,在这种范式中,定期训练和发布现有模型的更大版本。这类模型的新实例通常是完全从头开始训练的,尽管它们通常只是较小模型的放大版本。我们如何使用现有的较小模型参数中的隐含知识来更快地训练更新的较大模型?本文描述了一种通过学习增长预训练的Transformers来加速Transformer训练的方法,在该方法中,我们学习线性映射较小模型的参数以初始化较大模型。为了便于学习,我们将线性变换分解为(线性)宽度和深度增长算子的组合,并进一步采用这些增长算子的Kronecker分解来编码架构知识。跨语言和视觉Transformer的广泛实验表明,我们学习的线性增长算子(LiGO)可以从头开始节省高达50%的训练计算成本,同时也始终优于强基线,这些基线也重用较小的预训练模型来初始化较大的模型。

英文摘要:

Scaling transformers has led to significant breakthroughs in many domains, leading to a paradigm in which larger versions of existing models are trained and released on a periodic basis. New instances of such models are typically trained completely from scratch, despite the fact that they are often just scaled-up versions of their smaller counterparts. How can we use the implicit knowledge in the parameters of smaller, extant models to enable faster training of newer, larger models? This paper describes an approach for accelerating transformer training by learning to grow pretrained transformers, where we learn to linearly map the parameters of the smaller model to initialize the larger model. For tractable learning, we factorize the linear transformation as a composition of (linear) width- and depth-growth operators, and further employ a Kronecker factorization of these growth operators to encode architectural knowledge. Extensive experiments across both language and vision transformers demonstrate that our learned Linear Growth Operator (LiGO) can save up to 50% computational cost of training from scratch, while also consistently outperforming strong baselines that also reuse smaller pretrained models to initialize larger models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑