arXiv 2602.125562026-02-13cs.LGcs.AI

SD-MoE：通过谱分解实现有效的专家专业化

SD-MoE: Spectral Decomposition for Effective Expert Specialization

College of Computer Science and Artificial Intelligence, Fudan University, Shanghai, China（复旦大学计算机科学与人工智能学院）；University of Bath, Bath, United Kingdom（巴斯大学）；Oxford Suzhou Centre for Advanced Research, Suzhou, China（牛津苏泽研究中心）；Department of Electrical Engineering and Computer Science, University of Michigan（密歇根大学电气工程与计算机科学系）；Shanghai Innovation Institute, Shanghai, China（上海创新研究院）；Department of Computer Science, University of Colorado Boulder, Colorado, USA（科罗拉多大学博尔德分校计算机科学系）；Research Institute of Tsinghua University in Shenzhen, Shenzhen, China（清华大学深圳研究院）；Greater Bay Area National Center of Technology Innovation, Research Institute of Tsinghua University in Shenzhen, Shenzhen, China（粤港澳大湾区国家技术创新中心，清华大学深圳研究院）；School of Microelectronics, Fudan University, Shanghai, China（复旦大学微电子学院）

Ruijun Huang, Fang Dong, Xin Zhang, Hengjie Cao, Zhendong Huang, Anrui Chen, Jixian Zhou, Mengyi Chen, Yifeng Yang, Mingzhi Dong, Yujiang Wang, Jinlong Hou, Qin… 展开作者

Ruijun Huang, Fang Dong, Xin Zhang, Hengjie Cao, Zhendong Huang, Anrui Chen, Jixian Zhou, Mengyi Chen, Yifeng Yang, Mingzhi Dong, Yujiang Wang, Jinlong Hou, Qin Lv, Robert P. Dick, Yuan Cheng, Fan Yang, Tun Lu, Chun Zhang, Li Shang

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

SD-MoE通过谱分解解决MoE中专家专业化不足的问题，提升模型性能并兼容多种现有架构。

摘要：

混合专家（MoE）架构通过条件计算诱导的专家专业化来扩展大型语言模型。然而，在实践中，专家专业化常常失败：一些专家功能上相似，而另一些则充当事实上的共享专家，限制了有效容量和模型性能。在本文中，我们从参数和梯度空间的谱视角进行分析，发现（1）专家在参数中共享高度重叠的主导谱成分，（2）主导的梯度子空间在专家之间高度对齐，受人类语料库中普遍存在的低秩结构驱动，（3）门控机制倾向于将输入路由到这些主导方向，进一步限制了专业化。为了解决这个问题，我们提出了谱解耦MoE（SD-MoE），它在谱空间中分解参数和梯度。SD-MoE在下游任务中提高了性能，实现了有效的专家专业化，仅引入了最小的额外计算，并可以无缝集成到各种现有的MoE架构中，包括Qwen和DeepSeek。

英文摘要：

Mixture-of-Experts (MoE) architectures scale Large Language Models via expert specialization induced by conditional computation. In practice, however, expert specialization often fails: some experts become functionally similar, while others functioning as de facto shared experts, limiting the effective capacity and model performance. In this work, we analysis from a spectral perspective on parameter and gradient spaces, uncover that (1) experts share highly overlapping dominant spectral components in their parameters, (2) dominant gradient subspaces are strongly aligned across experts, driven by ubiquitous low-rank structure in human corpus, and (3) gating mechanisms preferentially route inputs along these dominant directions, further limiting specialization. To address this, we propose Spectral-Decoupled MoE (SD-MoE), which decomposes both parameter and gradient in the spectral space. SD-MoE improves performance across downstream tasks, enables effective expert specialization, incurring minimal additional computation, and can be seamlessly integrated into a wide range of existing MoE architectures, including Qwen and DeepSeek.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。