arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2310.01334cs.LGcs.AIcs.CL
收藏

合并,然后压缩:从路由策略的提示中揭示高效SMoE的奥秘

Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy

The University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校);The University of Texas at Austin(德克萨斯大学奥斯汀分校);The Chinese University of Hong Kong(香港中文大学);MIT(麻省理工学院);Harvard University(哈佛大学)

Pingzhi Li, Zhenyu Zhang, Prateek Yadav, Yi-Lin Sung, Yu Cheng, Mohit Bansal, Tianlong Chen

AI总结:

本研究提出MC-SMoE方法,通过路由统计指导专家合并并进一步压缩,解决SMoE模型内存高和专家冗余问题,实现高达80%内存和20%FLOPs减少且性能几乎无损。

摘要:

稀疏激活的专家混合(SMoE)在扩展神经网络的学习能力方面显示出潜力,然而,它们存在以下问题:(a)高内存使用,由于网络层被复制为多个专家副本;(b)专家冗余,因为基于学习的常见路由策略遭受表征崩溃。因此,原始的SMoE模型在内存效率上低下且不可扩展,特别是在资源受限的下游场景中。在本文中,我们提出疑问:能否通过整合专家信息来构建紧凑的SMoE模型?将多个专家合并为更少但更有知识的专家的最佳方法是什么?我们的初步调查揭示,传统的模型合并方法在SMoE的专家合并中效果不佳。潜在原因包括:(1)冗余信息掩盖了关键专家;(2)缺乏对每个专家进行适当的神经元排列以实现对齐。为解决此问题,我们提出M-SMoE,利用路由统计来指导专家合并。具体而言,它首先对专家进行神经元排列对齐;然后,形成主导专家及其“组成员”;最后,利用每个专家的激活频率作为合并权重,将每个专家组合并为一个专家,从而减少不重要专家的影响。此外,我们观察到,我们提出的合并促进了合并专家权重空间中的低维性,自然为额外压缩铺平了道路。因此,我们的最终方法MC-SMoE(即合并,然后压缩SMoE)进一步将合并的专家分解为低秩和结构稀疏的替代方案。在8个基准上的广泛实验验证了MC-SMoE的有效性。例如,我们的MC-SMoE实现了高达80%的内存和20%的FLOPs减少,且性能几乎无损。

英文摘要:

Sparsely activated Mixture-of-Experts (SMoE) has shown promise to scale up the learning capacity of neural networks, however, they have issues like (a) High Memory Usage, due to duplication of the network layers into multiple copies as experts; and (b) Redundancy in Experts, as common learning-based routing policies suffer from representational collapse. Therefore, vanilla SMoE models are memory inefficient and non-scalable, especially for resource-constrained downstream scenarios. In this paper, we ask: Can we craft a compact SMoE model by consolidating expert information? What is the best recipe to merge multiple experts into fewer but more knowledgeable experts? Our pilot investigation reveals that conventional model merging methods fail to be effective in such expert merging for SMoE. The potential reasons are: (1) redundant information overshadows critical experts; (2) appropriate neuron permutation for each expert is missing to bring all of them in alignment. To address this, we propose M-SMoE, which leverages routing statistics to guide expert merging. Specifically, it starts with neuron permutation alignment for experts; then, dominant experts and their "group members" are formed; lastly, every expert group is merged into a single expert by utilizing each expert's activation frequency as their weight for merging, thus diminishing the impact of insignificant experts. Moreover, we observed that our proposed merging promotes a low dimensionality in the merged expert's weight space, naturally paving the way for additional compression. Hence, our final method, MC-SMoE (i.e., Merge, then Compress SMoE), further decomposes the merged experts into low-rank and structural sparse alternatives. Extensive experiments across 8 benchmarks validate the effectiveness of MC-SMoE. For instance, our MC-SMoE achieves up to 80% memory and a 20% FLOPs reduction, with virtually no loss in performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑