输入域感知 MoE：在混合专家模型中将路由决策与任务优化解耦
Input Domain Aware MoE: Decoupling Routing Decisions from Task Optimization in Mixture of Experts
AI总结：
针对 sMoE 相似度路由难以兼顾专家专业化与负载均衡的问题，提出独立于任务目标训练的 Input Domain Aware MoE 概率混合路由框架，并在视觉-语言任务上验证其性能与利用均衡优势。
摘要：
稀疏混合专家（sMoE）已成为扩展大型视觉-语言模型的关键方法，它通过对专家进行动态、稀疏激活，在保持计算效率的同时提供可观容量。然而，现有路由机制通常基于相似度评分，难以有效捕捉潜在输入结构。这一限制导致专家专业化与计算均衡之间出现权衡，阻碍了可扩展性和性能。我们提出 Input Domain Aware MoE，这是一种新颖的路由框架，利用概率混合模型更好地划分输入空间。通过将路由概率建模为分布的混合，我们的方法使专家能够形成清晰的专业化边界，同时实现均衡利用。与传统方法不同，我们的路由机制独立于任务特定目标进行训练，从而实现稳定优化和明确的专家分配。在视觉-语言任务上的实验结果表明，我们的方法持续优于现有 sMoE 方法，取得了更高的任务性能和更好的专家利用均衡性。
英文摘要：
Sparse Mixture of Experts (sMoE) has become a pivotal approach for scaling large vision-language models, offering substantial capacity while maintaining computational efficiency through dynamic, sparse activation of experts. However, existing routing mechanisms, typically based on similarity scoring, struggle to effectively capture the underlying input structure. This limitation leads to a trade-off between expert specialization and balanced computation, hindering both scalability and performance. We propose Input Domain Aware MoE, a novel routing framework that leverages a probabilistic mixture model to better partition the input space. By modeling routing probabilities as a mixture of distributions, our method enables experts to develop clear specialization boundaries while achieving balanced utilization. Unlike conventional approaches, our routing mechanism is trained independently of task-specific objectives, allowing for stable optimization and decisive expert assignments. Empirical results on vision-language tasks demonstrate that our method consistently outperforms existing sMoE approaches, achieving higher task performance and improved expert utilization balance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
