Dirichlet-Prior Shaping：引导升级改造MoE中的专家专业化
Dirichlet-Prior Shaping: Guiding Expert Specialization in Upcycled MoEs
AI总结：
针对升级改造MoE因权重复制导致的专家专业化不足、路由区分度弱的问题，提出Dirichlet先验塑形损失DPSL，通过匹配目标Dirichlet先验塑形路由分布，在多骨干视觉-语言基准上性能优于现有方法。
摘要：
将预训练稠密模型升级改造为稀疏混合专家模型（Mixture-of-Experts, MoE）可有效提升模型容量，但由于简单的权重复制，往往存在专家专业化程度不足的问题。我们的分析表明，即便采用常规正则化，升级改造后的MoE仍表现出低置信度、区分度弱的路由分配，阻碍了性能提升。我们提出Dirichlet先验塑形损失（Dirichlet-Prior Shaping Loss, DPSL），这是一种新型路由正则化技术，通过让专家分配与目标Dirichlet先验匹配，直接对路由概率分布进行塑形。DPSL可对专家平衡度与专业化程度实现细粒度控制，还能编码归纳偏置（例如鼓励专家聚焦特定模态或任务），且无需人工干预；值得注意的是，DPSL是适用于任何输出类别概率分布模块的通用工具，其效用可拓展至MoE训练之外的场景。在以Qwen2、Phi3、Llama3.2大语言模型为骨干的升级改造MoE视觉-语言模型上开展的实验表明，在标准视觉-语言基准测试中，DPSL的表现始终优于现有升级改造策略与正则化技术，解决了专业化不足的关键问题，可培育出适应性更强、性能更优的模型。
英文摘要：
Upcycling pre-trained dense models into sparse Mixture-of-Experts (MoEs) efficiently increases model capacity but often suffers from poor expert specialization due to naive weight replication. Our analysis reveals that upcycled MoEs, even with conventional regularization, exhibit low-confidence, weakly differentiated routing, hindering performance. We introduce Dirichlet-Prior Shaping Loss (DPSL), a novel router regularization technique that directly shapes routing probability distributions by matching expert assignments to a target Dirichlet prior. DPSL offers fine-grained control over expert balance and specialization, and enables encoding of inductive biases such as encouraging experts to focus on specific modalities or tasks, without requiring manual intervention; notably, DPSL is a general tool applicable to any module that outputs categorical probability distributions, extending its utility beyond MoE training. Experiments on upcycled MoE vision-language models (with Qwen2, Phi3, Llama3.2 LLM backbones) show DPSL consistently outperforms upcycling strategies and regularization techniques across standard vision-language benchmarks, addressing the critical issue of poor specialization and fostering more adaptive, higher-performing models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
