STAR: 将MoE路由重新思考为结构感知的子空间学习
STAR: Rethinking MoE Routing as Structure-Aware Subspace Learning
AI总结：
提出STAR方法，通过广义Hebbian算法学习主子空间来增强路由对输入结构的感知，实现专家稳定专业化，在合成数据和语言视觉任务上提升路由质量和下游性能。
摘要：
混合专家（MoE）通过选择性地将输入路由到专门的专家子集来高效扩展模型容量。然而，输入-专家专业化（MoE的核心动机）关键取决于路由器是否真正感知输入结构。实践中，MoE路由通常实现为浅层线性投影，对输入表示的感知有限，常导致路由不稳定。我们提出STAR（结构感知路由），将MoE路由重新思考为子空间学习问题，通过广义Hebbian算法（GHA）跟踪主导输入结构的演化主子空间来增强标准可学习路由。通过将路由决策直接与输入结构对齐，STAR实现了稳定的专家专业化。我们在受控合成设置和大规模语言与视觉任务上评估STAR，它持续提高了路由质量和下游性能，超过了强MoE基线。此外，可选的测试时子空间更新进一步增强了输入分布偏移下的路由鲁棒性和泛化能力。
英文摘要：
Mixture-of-Experts (MoE) scales model capacity efficiently by selectively routing inputs to a specialized subset of experts. However, input-expert specialization, the core motivation of MoE, critically depends on whether the router is actually aware of input structure. In practice, MoE routing is typically implemented as a shallow linear projection with limited awareness of input representation, which often leads to unstable routing. We propose STAR, a Structure Aware Routing that rethinks MoE routing as a subspace learning problem by augmenting standard learnable routing with an evolving principal subspace that tracks dominant input structure via Generalized Hebbian Algorithm (GHA). By aligning routing decisions directly with input structure, STAR enables stable expert specialization. We evaluate STAR on controlled synthetic setup and large-scale language and vision tasks, where it consistently improves routing quality and downstream performance over strong MoE baselines. Moreover, optional test-time subspace updates further enhance routing robustness and generalization under input distribution shifts.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
