AdaMoE:面向混合专家语言模型的Token自适应路由与空专家机制
AdaMoE: Token-Adaptive Routing with Null Experts for Mixture-of-Experts Language Models
AI总结:
提出AdaMoE,通过引入不消耗FLOPs的空专家并增加k值,实现MoE模型中的token自适应路由,在降低平均专家负载的同时提升下游任务性能。
摘要:
混合专家(Mixture of experts, MoE)已成为构建生产级大语言模型(LLMs)的标准,因为它有望在不造成显著开销的情况下提升模型容量。然而,现有的MoE方法通常对所有token强制执行固定的top-k路由,这具有局限性,因为不同的token(例如“<EOS>”与“apple”)可能需要不同数量的专家进行特征抽象。解除这一约束有助于充分利用有限资源,并释放模型在下游任务中的潜力。在此意义上,我们引入了AdaMoE来实现MoE的token自适应路由,允许不同的token选择不同数量的专家。AdaMoE对带有top-k路由的原始MoE进行了最小化修改——它仅向专家集中引入固定数量的空专家(不消耗任何FLOPs),并增加k的值。AdaMoE不强制每个token占用固定数量的空专家,而是通过负载均衡损失确保空专家的平均使用量,从而使每个token使用的空/真实专家数量具有自适应性。AdaMoE与采用专家选择路由的MoE高度相似,同时允许简单的自回归建模。AdaMoE易于实现,可有效应用于预训练的(MoE-)LLMs。广泛的研究表明,AdaMoE能在实现卓越性能的同时降低平均专家负载(FLOPs)。例如,在ARC-C数据集上,将我们的方法应用于Mixtral-8x7B的微调,可在减少14.5% FLOPs的同时提高1.69%的准确率。
英文摘要:
Mixture of experts (MoE) has become the standard for constructing production-level large language models (LLMs) due to its promise to boost model capacity without causing significant overheads. Nevertheless, existing MoE methods usually enforce a constant top-k routing for all tokens, which is arguably restrictive because various tokens (e.g., "<EOS>" vs. "apple") may require various numbers of experts for feature abstraction. Lifting such a constraint can help make the most of limited resources and unleash the potential of the model for downstream tasks. In this sense, we introduce AdaMoE to realize token-adaptive routing for MoE, where different tokens are permitted to select a various number of experts. AdaMoE makes minimal modifications to the vanilla MoE with top-k routing -- it simply introduces a fixed number of null experts, which do not consume any FLOPs, to the expert set and increases the value of k. AdaMoE does not force each token to occupy a fixed number of null experts but ensures the average usage of the null experts with a load-balancing loss, leading to an adaptive number of null/true experts used by each token. AdaMoE exhibits a strong resemblance to MoEs with expert choice routing while allowing for trivial auto-regressive modeling. AdaMoE is easy to implement and can be effectively applied to pre-trained (MoE-)LLMs. Extensive studies show that AdaMoE can reduce average expert load (FLOPs) while achieving superior performance. For example, on the ARC-C dataset, applying our method to fine-tuning Mixtral-8x7B can reduce FLOPs by 14.5% while increasing accuracy by 1.69%.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。