Symphony-MoE：将不同预训练模型协调为连贯的混合专家模型
Symphony-MoE: Harmonizing Disparate Pre-trained Models into a Coherent Mixture-of-Experts
AI总结：
针对现有MoE升级方法专家多样性受限的问题，本文提出Symphony-MoE两阶段框架，融合多个架构相同但不同的预训练模型作为专家，通过层感知融合和激活对齐构建共享骨干，所获MoE在多域任务和分布外泛化上显著优于基线。
摘要：
混合专家（MoE）模型通过稀疏激活大型参数集实现可扩展性能，同时最大限度减少计算开销。为了降低从头训练MoE的高昂成本，近期研究采用升级方法，通过将单个预训练密集模型的前馈网络（FFN）层复制为专家来复用该模型。然而，由于所有专家均源自单个预训练密集模型，这限制了专家的多样性。本文通过使用来自多个架构相同但不同的预训练模型（例如Qwen2.5-Coder和Qwen2）的专家构建强大的MoE模型，解决了这一限制。一个关键挑战在于这些源模型占据不同、不和谐的参数空间区域，直接升级容易导致严重的性能下降。为克服这一问题，我们提出Symphony-MoE，这是一种新颖的两阶段框架，旨在将这些模型协调为单一、连贯的专家混合体。首先，我们以无训练的方式建立这种协调：通过层感知融合策略构建共享骨干，并至关重要地使用基于激活的功能对齐缓解专家之间的参数错位。随后，通过后训练阶段协调整个架构。实验表明，我们的方法成功整合了来自异质源的专家，所实现的MoE模型在多域任务和分布外泛化方面显著优于基线。
英文摘要：
Mixture-of-Experts (MoE) models enable scalable performance by activating large parameter sets sparsely, minimizing computational overhead. To mitigate the prohibitive cost of training MoEs from scratch, recent work employs upcycling, reusing a single pre-trained dense model by replicating its feed-forward network (FFN) layers into experts. However, this limits expert diversity, as all experts originate from a single pre-trained dense model. This paper addresses this limitation by constructing powerful MoE models using experts sourced from multiple identically-architected but disparate pre-trained models (e.g., Qwen2.5-Coder and Qwen2). A key challenge lies in the fact that these source models occupy disparate, dissonant regions of the parameter space, making direct upcycling prone to severe performance degradation. To overcome this, we propose Symphony-MoE, a novel two-stage framework designed to harmonize these models into a single, coherent expert mixture. First, we establish this harmony in a training-free manner: we construct a shared backbone via a layer-aware fusion strategy and, crucially, alleviate parameter misalignment among experts using activation-based functional alignment. Subsequently, a stage of post-training coordinates the entire architecture. Experiments demonstrate that our method successfully integrates experts from heterogeneous sources, achieving an MoE model that significantly surpasses baselines in multi-domain tasks and out-of-distribution generalization.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
