EMoE：基于特征空间的专家混合架构路由
EMoE: Eigenbasis-Guided Routing for Mixture-of-Experts
AI总结：
EMoE通过基于学习的正交特征基的路由机制，解决混合专家架构中的负载不平衡和专家同质化问题，无需辅助损失函数。
摘要：
深度学习模型的持续扩展导致了不可持续的计算需求，将混合专家（MoE）架构定位为提高效率的有希望路径。然而，MoE模型面临两个根本性挑战：1）负载不平衡问题，即
英文摘要：
The relentless scaling of deep learning models has led to unsustainable computational demands, positioning Mixture-of-Experts (MoE) architectures as a promising path towards greater efficiency. However, MoE models are plagued by two fundamental challenges: 1) a load imbalance problem known as the``rich get richer" phenomenon, where a few experts are over-utilized, and 2) an expert homogeneity problem, where experts learn redundant representations, negating their purpose. Current solutions typically employ an auxiliary load-balancing loss that, while mitigating imbalance, often exacerbates homogeneity by enforcing uniform routing at the expense of specialization. To resolve this, we introduce the Eigen-Mixture-of-Experts (EMoE), a novel architecture that leverages a routing mechanism based on a learned orthonormal eigenbasis. EMoE projects input tokens onto this shared eigenbasis and routes them based on their alignment with the principal components of the feature space. This principled, geometric partitioning of data intrinsically promotes both balanced expert utilization and the development of diverse, specialized experts, all without the need for a conflicting auxiliary loss function. Our code is publicly available at https://github.com/Belis0811/EMoE.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
