Accelerating large language model pre-training is a critical issue in present research. In this paper, we focus on speeding up pre-training by progressively growing from a small Transformer structure to a large one. There are two main research problems associated with progressive growth: determining the optimal growth schedule, and designing efficient growth operators. In terms of growth schedule, the impact of each single dimension on a schedule's efficiency is under-explored by existing work. Regarding the growth operators, existing methods rely on the initialization of new weights to inherit knowledge, and achieve only non-strict function preservation, limiting further improvements on training dynamics. To address these issues, we propose Masked Structural Growth (MSG), including (i) growth schedules involving all possible dimensions and (ii) strictly function-preserving growth operators that is independent of the initialization of new weights. Experiments show that MSG is significantly faster than related work: we achieve up to 2.2x speedup in pre-training different types of language models while maintaining comparable or better downstream performances. Code is publicly available at https://github.com/cofe-ai/MSG.
School of Computer Science and Technology, East China Normal University(华东师范大学计算机科学与技术学院);School of Computer Science and Technology, Harbin Engineering University(哈尔滨工程大学计算机科学与技术学院);School of Computer Science, Fudan University(复旦大学计算机学院)
Department of Computer Science, Cornell Tech, NYC, USA(康奈尔科技学院计算机科学系);Department of Computer Science, Cornell University, Ithaca, USA(康奈尔大学计算机科学系);School of Computer and Communication Sciences, EPFL Lausanne, Switzerland(洛桑联邦理工学院计算机与通信科学学院);Department of Computer Science, University of Chicago, Illinois(芝加哥大学计算机科学系);NVIDIA, Santa Clara, USA(英伟达)
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);Sea AI Lab, Singapore(新加坡Sea AI实验室);Singapore University of Technology and Design(新加坡科技设计大学)