arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2110.01786cs.CL
收藏

MoEfication:Transformer前馈层是专家的混合体

MoEfication: Transformer Feed-forward Layers are Mixtures of Experts

Tsinghua University(清华大学);Beijing National Research Center for Information Science and Technology(北京信息科学与技术国家研究中心);Tencent Inc(腾讯公司);International Innovation Center of Tsinghua University(清华大学国际创新中心);Institute for AI Industry Research (AIR), Tsinghua University(清华大学人工智能产业研究院(AIR));Beijing Academy of Artificial Intelligence(北京人工智能研究院);Jiangsu Collaborative Innovation Center for Language Ability(江苏语言能力协同创新中心)

Zhengyan Zhang, Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun, Jie Zhou

摘要:

最近的工作表明,预训练的Transformers中的前馈网络(FFN)是一个关键组件,存储各种语言和事实知识。然而,FFN的计算模式仍不清楚。在这项工作中,我们研究了FFNs的计算模式,并观察到大多数输入只激活FFNs的一小部分神经元。这种现象类似于人类大脑的稀疏性,这推动了对人类大脑功能分区的研究。为了验证功能分区是否也出现在FFN中,我们建议将模型转换为具有相同参数的MoE版本,即MoEFification。具体来说,MoEnfication由两个阶段组成:(1)将FFN的参数划分为多个功能分区作为专家,以及(2)构建专家路由器以决定每个输入将使用哪些专家。实验结果表明,MoEfication可以有条件地使用10%到30%的FFN参数,同时保持95%以上的原始性能,用于不同的模型在各种下游任务。此外,MoEfication带来了两个优点:(1)它显著降低了推理的FLOPS,即,2倍的加速比,25%的FFN参数,(2)它提供了一个细粒度的角度来研究FFN的内部机制。本文的源代码可以从https://github.com/thunlp/MoEfication获得。

英文摘要:

Recent work has shown that feed-forward networks (FFNs) in pre-trained Transformers are a key component, storing various linguistic and factual knowledge. However, the computational patterns of FFNs are still unclear. In this work, we study the computational patterns of FFNs and observe that most inputs only activate a tiny ratio of neurons of FFNs. This phenomenon is similar to the sparsity of the human brain, which drives research on functional partitions of the human brain. To verify whether functional partitions also emerge in FFNs, we propose to convert a model into its MoE version with the same parameters, namely MoEfication. Specifically, MoEfication consists of two phases: (1) splitting the parameters of FFNs into multiple functional partitions as experts, and (2) building expert routers to decide which experts will be used for each input. Experimental results show that MoEfication can conditionally use 10% to 30% of FFN parameters while maintaining over 95% original performance for different models on various downstream tasks. Besides, MoEfication brings two advantages: (1) it significantly reduces the FLOPS of inference, i.e., 2x speedup with 25% of FFN parameters, and (2) it provides a fine-grained perspective to study the inner mechanism of FFNs. The source code of this paper can be obtained from https://github.com/thunlp/MoEfication.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑