arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2410.03440cs.CLcs.AI
收藏

探索预训练中激活稀疏性的益处

Exploring the Benefit of Activation Sparsity in Pre-training

Tsinghua University(清华大学);Renmin University of China(中国人民大学);Tencent(腾讯)

Zhengyan Zhang, Chaojun Xiao, Qiujieli Qin, Yankai Lin, Zhiyuan Zeng, Xu Han, Zhiyuan Liu, Ruobing Xie, Maosong Sun, Jie Zhou

AI总结:

该研究发现预训练Transformer大部分阶段存在激活稀疏性且激活相关性随训练演化,提出可切换稀疏-稠密学习方法,在保证性能的同时降低预训练成本,还能实现最高2倍的稀疏推理加速。

摘要:

预训练Transformer天然具备激活稀疏性的特征,即对于每个token,仅有小部分神经元被激活。尽管已有研究通过训练后方法探索激活稀疏性,但它在预训练中的潜力仍未被发掘。在本研究中,我们首先探究了预训练过程中激活特性的变化规律。研究发现,Transformer在预训练的大部分阶段都呈现出激活稀疏性,而激活相关性会随着训练推进持续演化。基于这一观察,我们提出了可切换稀疏-稠密学习(Switchable Sparse-Dense Learning,SSD)。SSD在预训练过程中自适应地在基于混合专家(Mixtures-of-Experts,MoE)的稀疏训练和传统稠密训练之间切换,既利用了稀疏训练的高效性,又避免了稀疏训练的静态激活相关性问题。与稠密训练相比,SSD在模型规模相同的情况下取得了相当的性能,同时降低了预训练成本。此外,通过SSD训练的模型可直接作为MoE模型用于稀疏推理,在达到与稠密模型相同性能的同时,推理速度最高可达2倍。代码已开源至https://github.com/thunlp/moefication。

英文摘要:

Pre-trained Transformers inherently possess the characteristic of sparse activation, where only a small fraction of the neurons are activated for each token. While sparse activation has been explored through post-training methods, its potential in pre-training remains untapped. In this work, we first study how activation properties change during pre-training. Our examination reveals that Transformers exhibit sparse activation throughout the majority of the pre-training process while the activation correlation keeps evolving as training progresses. Leveraging this observation, we propose Switchable Sparse-Dense Learning (SSD). SSD adaptively switches between the Mixtures-of-Experts (MoE) based sparse training and the conventional dense training during the pre-training process, leveraging the efficiency of sparse training and avoiding the static activation correlation of sparse training. Compared to dense training, SSD achieves comparable performance with identical model size and reduces pre-training costs. Moreover, the models trained with SSD can be directly used as MoE models for sparse inference and achieve the same performance as dense models with up to $2\times$ faster inference speed. Codes are available at https://github.com/thunlp/moefication.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑