MoSE: 混合可瘦身专家实现高效自适应语言模型
MoSE: Mixture of Slimmable Experts for Efficient and Adaptive Language Models
AI总结：
提出MoSE架构，每个专家具有可变宽度的嵌套结构，支持在推理时连续调节精度-计算权衡，通过多宽度训练和轻量级测试时训练实现高效自适应。
摘要：
混合专家（MoE）模型通过稀疏激活专家高效扩展大型语言模型，但一旦选定专家，其执行是完整的。因此，MoE模型中精度与计算之间的权衡通常表现出较大的不连续性。我们提出混合可瘦身专家（MoSE），这是一种MoE架构，其中每个专家具有嵌套的、可瘦身的结构，可以以可变宽度执行。这不仅实现了对激活哪些专家的条件计算，还实现了对每个专家利用多少的条件计算。因此，单个预训练的MoSE模型可以在推理时支持更连续的精度-计算权衡谱。我们提出了一种简单且稳定的训练方法，用于在稀疏路由下训练可瘦身专家，将多宽度训练与标准MoE目标相结合。在推理过程中，我们探索了运行时宽度确定的策略，包括一种轻量级的测试时训练机制，该机制学习如何在固定预算下将路由器置信度/概率映射到专家宽度。在GPT风格模型、各种路由机制、零样本下游推理基准以及DeepSeek模型的持续预训练适应上的实验表明，MoSE在全宽度下匹配或优于标准MoE，并持续将计算-质量边界向更低的推理FLOPs移动。代码可在以下网址找到：this https URL。
英文摘要：
Mixture-of-Experts (MoE) models scale large language models efficiently by sparsely activating experts, but once an expert is selected, it is executed fully. Hence, the trade-off between accuracy and computation in an MoE model typically exhibits large discontinuities. We propose Mixture of Slimmable Experts (MoSE), an MoE architecture in which each expert has a nested, slimmable structure that can be executed at variable widths. This enables conditional computation not only over which experts are activated but also over how much of each expert is utilized. Consequently, a single pretrained MoSE model can support a more continuous spectrum of accuracy-compute trade-offs at inference time. We present a simple and stable training recipe for slimmable experts under sparse routing, combining multi-width training with standard MoE objectives. During inference, we explore strategies for runtime width determination, including a lightweight test-time training mechanism that learns how to map router confidence/probabilities to expert widths under a fixed budget. Experiments on GPT-style models, various routing regimes, zero-shot downstream reasoning benchmarks, and continual pre-training adaptation of DeepSeek model show that MoSE matches or improves standard MoE at full width and consistently shifts the compute-quality frontier toward lower inference FLOPs. The code can be found at: https://github.com/tnurbek/mose.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
