Pre-trained Transformers inherently possess the characteristic of sparse activation, where only a small fraction of the neurons are activated for each token. While sparse activation has been explored through post-training methods, its potential in pre-training remains untapped. In this work, we first study how activation properties change during pre-training. Our examination reveals that Transformers exhibit sparse activation throughout the majority of the pre-training process while the activation correlation keeps evolving as training progresses. Leveraging this observation, we propose Switchable Sparse-Dense Learning (SSD). SSD adaptively switches between the Mixtures-of-Experts (MoE) based sparse training and the conventional dense training during the pre-training process, leveraging the efficiency of sparse training and avoiding the static activation correlation of sparse training. Compared to dense training, SSD achieves comparable performance with identical model size and reduces pre-training costs. Moreover, the models trained with SSD can be directly used as MoE models for sparse inference and achieve the same performance as dense models with up to $2\times$ faster inference speed. Codes are available at https://github.com/thunlp/moefication.
Understanding Activation Patterns in Artificial Neural Networks by Exploring Stochastic Processes
Institute of Computer Science(计算机科学研究所);Ruhr West University of Applied Science(鲁尔西应用科学大学);Ruhr University Bochum(波鸿鲁尔大学);Institute for Neural Computation(神经计算研究所)
Exploring Unsupervised Cell Recognition with Prior Self-activation Maps
College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);Research Center for Industries of the Future, Westlake University(西湖大学未来产业研究中心);School of Engineering, Westlake University(西湖大学工学院);Westlake University(西湖大学);Zhejiang University(浙江大学)
Unraveling Babel: Exploring Multilingual Activation Patterns of LLMs and Their Applications
Liangzhu Laboratory, Zhejiang University(浙江大学良渚实验室);AI Research Center, WeDoctor Cloud(微医云AI研究中心);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));The Hong Kong University of Science and Technology(香港科技大学);School of Public Health, Zhejiang University(浙江大学公共卫生学院)
Turbo Sparse: Achieving LLM SOTA Performance with Minimal Activated Parameters
Institute of Parallel and Distributed Systems (IPADS), Shanghai Jiao Tong University(上海交通大学并行与分布式系统研究所(IPADS));Department of Computer Science and Technology, Tsinghua University(清华大学计算机科学与技术系);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院);Baidu Inc(百度公司)
针对 Transformer 均匀扩展隐藏维度效率低的问题,提出 MoHD 稀疏条件激活架构,利用共享与路由激活子维度,在减少激活参数的同时提升性能,为模型扩展提供新视角。