arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2305.02869cs.CL
收藏

掩码结构增长实现2倍速语言模型预训练

Masked Structural Growth for 2x Faster Language Model Pre-training

Beijing Academy of Artificial Intelligence(北京人工智能研究院);Harbin Institute of Technology(哈尔滨工业大学)

Yiqun Yao, Zheng Zhang, Jing Li, Yequan Wang

AI总结:

本文提出掩码结构增长(MSG)方法,通过涉及所有维度的增长调度和严格函数保持的增长算子,实现语言模型预训练最高2.2倍加速。

摘要:

加速大语言模型预训练是当前研究中的一个关键问题。本文聚焦于通过从小型Transformer结构逐步增长到大型结构来加速预训练。与渐进式增长相关的两个主要研究问题是:确定最优增长调度,以及设计高效的增长算子。在增长调度方面,现有工作对每个单一维度对调度效率的影响探索不足。关于增长算子,现有方法依赖新权重的初始化来继承知识,并且只能实现非严格的函数保持,限制了训练动态的进一步改进。为解决这些问题,我们提出了掩码结构增长(MSG),包括(i)涉及所有可能维度的增长调度,以及(ii)严格函数保持的增长算子,且该算子不依赖于新权重的初始化。实验表明,MSG显著快于相关工作:我们在预训练不同类型的语言模型时实现了高达2.2倍的加速,同时保持可比或更好的下游性能。代码公开于 https://github.com/cofe-ai/MSG。

英文摘要:

Accelerating large language model pre-training is a critical issue in present research. In this paper, we focus on speeding up pre-training by progressively growing from a small Transformer structure to a large one. There are two main research problems associated with progressive growth: determining the optimal growth schedule, and designing efficient growth operators. In terms of growth schedule, the impact of each single dimension on a schedule's efficiency is under-explored by existing work. Regarding the growth operators, existing methods rely on the initialization of new weights to inherit knowledge, and achieve only non-strict function preservation, limiting further improvements on training dynamics. To address these issues, we propose Masked Structural Growth (MSG), including (i) growth schedules involving all possible dimensions and (ii) strictly function-preserving growth operators that is independent of the initialization of new weights. Experiments show that MSG is significantly faster than related work: we achieve up to 2.2x speedup in pre-training different types of language models while maintaining comparable or better downstream performances. Code is publicly available at https://github.com/cofe-ai/MSG.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑