arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2201.10890cs.LGcs.AIcs.CLcs.CV
收藏

一个学生知道所有专家知道:从稀疏到密集

One Student Knows All Experts Know: From Sparse to Dense

National University of Singapore(新加坡国立大学);Huawei Noah’s Ark Lab(华为诺亚方舟实验室)

Fuzhao Xue, Xiaoxin He, Xiaozhe Ren, Yuxuan Lou, Yang You

摘要:

人类教育系统由多个专家培养一个学生。混合专家(Mixture-of-experts,MoE)是一种包含多个专家的强大稀疏体系结构。然而,稀疏MoE模型容易过拟合,难以部署,并且对于从业者来说不是硬件友好的。在这项工作中,受人类教育模型的启发,我们提出了一个新的任务,知识整合,以获得一个密集的学生模型(OneS)作为一个稀疏的MoE知识渊博。我们通过提出一个包括知识收集和知识蒸馏的一般培训框架来调查这一任务。具体来说,收集关键知识,从不同的预先训练的专家,我们首先研究了四种不同的可能的知识收集方法,\ie求和,平均,Top-K知识收集(Top-KG),奇异值分解知识收集(SVD-KG)在本文中提出的。然后,我们通过知识蒸馏来改进密集学生模型,以抵消来自收集的噪声。在ImageNet上,我们的OneS保留了$61.7\%$从MoE中获得的好处,并仅使用$15$ M参数实现了$78.4\%$ top-1精度ImageNet。在四个自然语言处理数据集上,OneS获得了$88.2\%$ MoE的优势,并通过使用相同架构和训练数据的$51.7\%$,获得了优于最佳基线的性能。此外,与MoE相比,OneS由于计算量小和硬件友好的架构,可以实现$3.7 \times$推理加速。

英文摘要:

Human education system trains one student by multiple experts. Mixture-of-experts (MoE) is a powerful sparse architecture including multiple experts. However, sparse MoE model is easy to overfit, hard to deploy, and not hardware-friendly for practitioners. In this work, inspired by the human education model, we propose a novel task, knowledge integration, to obtain a dense student model (OneS) as knowledgeable as one sparse MoE. We investigate this task by proposing a general training framework including knowledge gathering and knowledge distillation. Specifically, to gather key knowledge from different pre-trained experts, we first investigate four different possible knowledge gathering methods, \ie summation, averaging, Top-K Knowledge Gathering (Top-KG), and Singular Value Decomposition Knowledge Gathering (SVD-KG) proposed in this paper. We then refine the dense student model by knowledge distillation to offset the noise from gathering. On ImageNet, our OneS preserves $61.7\%$ benefits from MoE and achieves $78.4\%$ top-1 accuracy ImageNet with only $15$M parameters. On four natural language processing datasets, OneS obtains $88.2\%$ MoE benefits and outperforms the best baseline by $51.7\%$ using the same architecture and training data. In addition, compared with the MoE counterpart, OneS can achieve $3.7 \times$ inference speedup due to less computation and hardware-friendly architecture.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑