arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.30992cs.LG
收藏

专家特征向量是无需训练的非崩溃路由器

Eigenvectors of Experts are Training-free Non-collapsing Routers

Applied Artificial Intelligence Intiative (A2I2), Deakin University, Victoria, Australia(应用人工智能倡议(A2I2),德肯大学,维多利亚,澳大利亚)

Giang Do, Hung Le, Truyen Tran

AI总结:

针对稀疏混合专家模型中专家崩溃问题,提出基于专家权重矩阵特征向量的无需训练路由框架SSMoE,通过奇异值分解利用谱特性提升模型性能。

摘要:

稀疏混合专家(SMoE)架构通过将输入令牌路由到选定的专家子集来提高大型语言模型(LLMs)的训练效率。尽管取得了显著成功,SMoE模型在训练和推理中仍面临专家崩溃问题(Chi等人,2022),这会降低模型性能。先前研究主要关注改进路由器;然而,这些方法依赖于从头训练或微调,需要高昂的计算和数据处理成本。此外,我们通过理论和实证结果证明,尽管有这些努力,在推进预训练良好的SMoE模型时,该问题仍然存在。为填补这一空白,我们分析了先进的SMoE模型,观察到专家权重矩阵的特征向量编码了丰富的语义信息,指向传统路由策略的有效替代方案。基于这一见解,我们提出了奇异值分解SMoE(SSMoE),一种新颖且无需训练的框架,利用专家权重的谱特性来解决崩溃问题并提升模型性能。在多种语言和视觉任务上的大量实验,包括干净和损坏数据设置,证明了SSMoE的强大泛化能力和鲁棒性。我们的发现强调了更深入理解模型内部结构如何指导开发更有效的SMoE架构。我们的实现已在https://github.com/giangdip2410/SSMoE公开。

英文摘要:

Sparse Mixture of Experts (SMoE) architectures improve the training efficiency of Large Language Models (LLMs) by routing input tokens to a selected subset of specialized experts. Despite their remarkable success, both training and inference in SMoE models suffer from the expert collapse issue (Chi et al., 2022), which degrades model performance. Prior studies primarily focus on improving the router; however, such methods rely on training from scratch or fine-tuning, which requires high computational and data-processing costs. Furthermore, we demonstrate that, despite these efforts, the issue persists when advancing well-pretrained SMoE models, as evidenced by both theoretical and empirical results. To fill that gap, we analyze the advanced SMoE models and observe that the eigenvectors of expert weight matrices encode rich semantic information, pointing to an effective alternative to conventional routing strategies. Building on this insight, we propose Singular Value Decomposition SMoE (SSMoE), a novel and training-free framework that leverages spectral properties of the expert weights to address the collapse issue and enhance model performance. Extensive experiments across diverse language and vision tasks, under both clean and corrupt data settings, demonstrate the strong generalization and robustness of SSMoE. Our findings highlight how a deeper understanding of model internals can guide the development of more effective SMoE architectures. Our implementation is publicly available at https://github.com/giangdip2410/SSMoE.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑