arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2610.19232cs.LG
收藏

Spectral-MoE:密集模型转混合专家模型过程中每层专家的异构利用

Spectral-MoE: Heterogeneous Per-Layer Expert Utilization in Dense-to-MoE Conversion

Stanford University(斯坦福大学)

Zachary Broveak, Christian Kubicka, Thierry Tambe

AI总结:

Spectral-MoE 是一种密集转 MoE 的优化技术,通过异构分配每层专家利用率,在多个模型的 WikiText-2 困惑度和六任务平均准确率上均优于均匀分配,确立了新的优化维度。

摘要:

大型语言模型在推理阶段会产生巨大的内存需求和算术成本。密集模型转混合专家模型(Mixture-of-Experts,MoE)是一种旨在降低这类算术成本的模型优化技术,该方法将预训练的前馈网络(Feed-Forward Network,FFN)神经元划分为多个专家,并为每个 token 选择一个子集。主流方法分为两类:解析方法利用启发式规则快速生成转换后的模型;学习方法则通过梯度优化生成模型,速度慢得多。然而,这两种方法都固定了每层相同的活跃专家数量(即“利用率”)。我们提出 Spectral-MoE,它能在各层间异构分配利用率预算,其名称源于将密集 FFN 划分为专家与将光束分解为光谱的隐喻。Spectral-MoE 包含两种不同方法:静态方法使用单个跨 token 共享的每层标量;动态方法则生成每个 token 的调度。我们证明,在基于解析和基于学习的 SOTA 密集转 MoE 方法中,两种分配器的性能均优于均匀分配。在 FFN 利用率为 25% 时,动态分配相对于均匀分配,在 Llama-2-7B、Llama-3-8B 和 Qwen2.5-7B 的学习基线上,WikiText-2 困惑度分别降低 1.04、2.96 和 2.71 个点;在解析基线上,对应降低 2.45、6.26 和 2.17 个点。在学习基线上,当 FFN 利用率分别为 12.5%、25% 和 50% 时,动态分配在三个模型上的六任务平均准确率较均匀分配平均提升 1.5、4.2 和 3.6 个百分点。这些结果确立了每层专家利用率是密集转 MoE 转换中的一个新优化维度。

英文摘要:

Large language models impose substantial memory demands and arithmetic costs during inference. Dense-to-mixture-of-experts (MoE) conversion is a model optimization technique that aims to reduce these arithmetic costs. Dense-to-MoE conversion partitions pretrained feed-forward network (FFN) neurons into experts and selects a subset for each token. Leading methods fall into two categories: analytic methods produce a converted model quickly using heuristics, while learning methods produce a model much more slowly by optimizing via gradients. Both approaches, however, fix the same active expert count, or "utilization," at every layer. We introduce Spectral-MoE, which learns to allocate a utilization budget heterogeneously across layers. Its name draws a metaphor between separating a dense FFN into experts and separating a light beam into its spectrum. Spectral-MoE consists of two distinct approaches: a static approach that uses a single per-layer scalar which is shared across tokens, and a dynamic approach which emits a per-token schedule. We demonstrate that both allocators improve performance over uniform on both the analytic-based and learning-based SOTA dense-to-MoE methods. At 25% FFN utilization, dynamic allocation reduces WikiText-2 perplexity relative to uniform by 1.04, 2.96, and 2.71 points on the learned base for Llama-2-7B, Llama-3-8B, and Qwen2.5-7B, respectively. The corresponding reductions on the analytic base are 2.45, 6.26, and 2.17 points. On the learned base, dynamic allocation improves six-task mean accuracy over uniform by an average of 1.5, 4.2, and 3.6 percentage points across the three models at FFN utilizations of 12.5%, 25%, and 50%, respectively. These results establish per-layer expert utilization as a novel optimization axis in dense-to-MoE conversion.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑