循环绑定——混合专家语言模型中的专家层绑定
Tying the Loop -- Tied Expert Layers in Mixture-of-Experts Language Models
AI总结：
提出专家绑定方法，通过共享连续Transformer层的专家参数，在保持独立路由和注意力的同时，将MoE模型内存占用降低近2倍，且不损失困惑度或下游性能。
摘要：
混合专家（MoE）架构通过每个令牌仅激活一小部分专家来高效扩展大型语言模型（LLM），但全部参数计数——主要由专家参数主导——必须保留在训练和推理内存中。为了解决这个问题，我们引入了专家绑定（Expert Tying），这是一种架构修改，它在连续Transformer层之间共享专家参数，同时保留独立的逐层路由和注意力。我们在常见的先进架构上评估了这种方法，包括OLMoE、Qwen3和DeepSeek风格的MoE。我们的预训练实验表明，绑定专家可以将内存占用减少近2倍，而几乎不降低困惑度或下游质量。通过利用MoE路径中固有的参数冗余，我们的方法提供了高度有利的计算-内存权衡，推动了下一代LLM的高效训练和扩展。
英文摘要：
Mixture-of-Experts (MoE) architectures efficiently scale Large Language Models (LLMs) by activating only a small fraction of their experts per token, yet the full parameter count - dominated by the expert parameters - must be held in training and inference memory. To address this, we introduce Expert Tying, an architectural modification that shares expert parameters across consecutive transformer layers while preserving independent, layer-wise routing and attention. We evaluate this approach across common, state-of-the-art architectures, including OLMoE, Qwen3, and DeepSeek-style MoEs. Our pretraining experiments demonstrate that tying experts can reduce memory footprint by almost 2x at virtually no degradation in perplexity or downstream quality. By exploiting the parameter redundancy inherent in MoE pathways, our method provides a highly favorable compute-to-memory trade-off, advancing efficient training and scaling of next-generation LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
