L2R：低秩与利普希茨控制路由用于专家混合模型
L2R: Low-Rank and Lipschitz-Controlled Routing for Mixture-of-Experts
AI总结：
本文提出L2R框架，通过低秩潜在路由空间和SIPS评分机制提升MoE模型的路由几何和专家表现力，实验表明其在语言和视觉任务中均提升性能。
摘要：
混合专家（MoE）模型通过条件激活少量专家来扩展神经网络，其中路由器在确定专家专精和整体模型性能中起关键作用。然而，许多现代MoE系统仍使用线性路由器在原始高维表示空间中，导致表示不匹配、角度集中和尺度敏感评分可能共同损害路由判别性和稳定的专家专精。在本文中，我们提出低秩与利普希茨控制路由（L2R），一种统一的路由框架，重塑路由空间和评分几何。L2R在共享低秩潜在路由空间中进行专家分配，并引入饱和内积评分（SIPS）以显式控制路由函数的利普希茨行为，产生更平滑和稳定的路由几何。此外，L2R结合了参数高效的多锚点路由机制以增强专家表达能力。在基于OLMoE的语言MoE模型和ImageNet视觉MoE设置上的广泛实验表明，L2R一致提升了路由几何、专家判别性和整体模型性能。代码将被发布。
英文摘要：
Mixture-of-Experts (MoE) models scale neural networks by conditionally activating a small subset of experts, where the router plays a central role in determining expert specialization and overall model performance. However, many modern MoE systems still adopt linear routers in raw high-dimensional representation spaces, where representation mismatch, angular concentration, and scale-sensitive scoring can jointly undermine routing discriminability and stable expert specialization. In this work, we propose Low-rank & Lipschitz-controlled Routing (L2R), a unified routing framework that reshapes both the routing space and scoring geometry. L2R performs expert assignment in a shared low-rank latent routing space and introduces Saturated Inner-Product Scoring (SIPS) to explicitly control the Lipschitz behavior of routing functions, yielding smoother and more stable routing geometry. In addition, L2R incorporates a parameter-efficient multi-anchor routing mechanism to enhance expert expressiveness. Experiments on an OLMoE-based language MoE model and a ViT-based ImageNet setting show improved overall performance in both domains; OLMoE diagnostics further show improved routing geometry and expert discrimination.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
