arXiv 2604.230362026-09-10cs.LGcs.CL

在混合专家调谐中保持长尾专家信息

Preserving Long-Tailed Expert Information in Mixture-of-Experts Tuning

Carnegie Mellon University（卡内基梅隆大学）；MIT（麻省理工学院）；UCSD（加州大学圣地亚哥分校）；Johns Hopkins University（约翰霍普金斯大学）

Haoze He, Xingyuan Ding, Xuan Jiang, Xinkai Zou, Alex Cheng, Yibo Zhao, Juncheng Billy Li, Heather Miller

备注：: Camera-ready version. Accepted at the Third Conference on Language Modeling (COLM 2026)
发表信息：: Proceedings of the Third Conference on Language Modeling (COLM 2026), 2026

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出一种无需辅助损失的MoE SFT框架，通过偏置驱动稀疏化与始终活跃的门控凝结专家，有效保留长尾专家信息，实验显示在数学推理和常识问答基准上优于现有方法。

摘要：

尽管MoE模型在许多基准中表现优异，但其架构的监督微调（SFT）仍具挑战性，因为路由层易受损害。方法如DenseMixer和ESFT通过密集混合或辅助负载平衡损失缓解路由崩溃，但这些方法引入了噪声梯度，通常会降低性能。初步实验中，我们系统地修剪专家并发现，尽管某些超级专家被激活频率极高，丢弃较少使用的专家仍会导致显著性能下降。这表明，即使很少被激活的专家也编码了非平凡的知识，对下游任务有用。受此启发，我们提出了一种无需辅助损失的MoE SFT框架，结合偏置驱动稀疏化与始终活跃的门控凝结专家。与强制所有专家激活平衡不同，我们的方法鼓励与任务相关专家保持活跃，同时推动长尾专家趋向不活跃。凝结专家提供了一个持久、可学习的路径，缓解梯度饥饿并促进信息整合，这些信息原本会分散在稀疏激活的专家中。分析进一步表明，这种设计在稀疏路由下更有效地保留长尾专家信息。在大规模MoE模型上的实验表明，我们的方法优于现有SFT基线，如DenseMixer和ESFT，在数学推理和常识问答基准上平均获得2.5%以上的提升。

英文摘要：

Despite MoE models leading many benchmarks, supervised fine-tuning (SFT) for the MoE architectures remains difficult because its router layers are fragile. Methods such as DenseMixer and ESFT mitigate router collapse with dense mixing or auxiliary load-balancing losses, but these introduce noisy gradients that often degrade performance. In preliminary experiments, we systematically pruned experts and observed that while certain super experts are activated far more frequently, discarding less used experts still leads to notable performance degradation. This suggests that even rarely activated experts encode non-trivial knowledge useful for downstream tasks. Motivated by this, we propose an auxiliary-loss-free MoE SFT framework that combines bias-driven sparsification with always-active gated condenser experts. Rather than enforcing balanced activation across all experts, our method encourages task-relevant experts to remain active while pushing long-tailed experts toward inactivity. The condenser experts provide a persistent, learnable pathway that alleviates gradient starvation and facilitates consolidation of information that would otherwise remain fragmented across sparsely activated experts. Analysis further suggest that this design better preserves long-tailed expert information under sparse routing. Experiments on large-scale MoE models demonstrate that our approach outperforms state-of-the-art SFT baselines such as DenseMixer and ESFT, achieving average gain of 2.5%+ on both mathematical reasoning and commonsenseQA benchmarks.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。