arXiv 2606.164562026-06-16cs.LGcs.AI

SPRI: 基于SVD分解残差初始化的数据受限MoE升级方法

SPRI: SVD-Partitioned Residual Initialization for Data-Constrained MoE Upcycling

Northeastern University, China（东北大学）；Huawei TSC, China（华为技术有限公司）；CUHK-Shenzhen, China（香港中文大学（深圳））；University of Maryland, USA（马里兰大学）；Harbin Engineering University, China（哈尔滨工程大学）；Inclusion AI, Ant Group（蚂蚁集团Inclusion AI）；NiuTrans Research, China（小牛翻译研究中心）

Weiqiao Shan, Ruixiang Mao, Yuang Li, Yuhao Zhang, Yingfeng Luo, Tong Zheng, Chen Xu, Yucheng Qiao, Chunxiang Jin, Yi Yuan, Jingdong Chen, Tong Xiao, Jingbo Zhu

备注：: 8pages, 12 tables, 3 figures

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出SPRI方法，利用预训练FFN权重的SVD分解残差初始化MoE专家，结合两阶段训练策略，在数据受限的多语言语音翻译任务中显著提升性能。

摘要：

混合专家（MoE）模型能够实现高效扩展，但从头训练成本过高。MoE升级通过将预训练的密集模型转换为稀疏MoE模型来降低这一成本。然而，现有的升级方法通常依赖大规模持续训练，并且在数据受限的监督适应中表现不佳，原因在于专家同质化或对预训练参数的过度扰动。在此设置下，有效的升级必须利用预训练权重结构，同时为路由专家引入足够的多样性。为此，我们提出了基于SVD分解残差初始化（SPRI）的方法，该方法将从预训练前馈网络（FFN）权重中提取的SVD分解残差分配到路由专家中，从而在预训练谱结构的基础上引入可控的专家多样性。我们进一步引入两阶段训练策略以提高适应稳定性。我们在多语言语音到文本翻译任务上评估SPRI，该任务中有限的监督数据对MoE升级构成挑战，而多个目标语言提供了天然的路由异质性。在CoVoST2数据集上的15个英语到其他语言方向中，SPRI相比完全微调的密集模型平均BLEU和COMET分别提高了2.58和3.32分，并且比之前最佳的MoE升级基线高出3.39 BLEU和4.34 COMET分。

英文摘要：

Mixture-of-Experts (MoE) models enable efficient scaling, but training them from scratch remains prohibitively expensive. MoE upcycling mitigates this cost by converting pretrained dense models into sparse MoE models. However, existing upcycling methods typically rely on large-scale continued training and often perform poorly under data-constrained supervised adaptation, due to either homogeneous experts or overly disruptive perturbations to pretrained parameters. In this setting, effective upcycling must leverage pretrained weight structure while introducing sufficient diversity among routed experts. To this end, we propose SVD-Partitioned Residual Initialization (SPRI), which distributes SVD-partitioned residuals derived from pretrained feed-forward network (FFN) weights across routed experts, introducing controlled expert diversity grounded in pretrained spectral structure. We further introduce a two-stage training strategy to improve adaptation stability. We evaluate SPRI on multilingual speech-to-text translation, where limited supervised data challenges MoE upcycling and multiple target languages provide natural routing heterogeneity. On CoVoST2 across 15 En-to-XX directions, SPRI improves average BLEU and COMET over fully fine-tuned dense models by 2.58 and 3.32 points, respectively, and outperforms the prior best MoE upcycling baseline by 3.39 BLEU and 4.34 COMET points.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。