稀疏专家混合模型的兴起：从算法基础到去中心化架构和垂直领域应用的综述
The Rise of Sparse Mixture-of-Experts: A Survey from Algorithmic Foundations to Decentralized Architectures and Vertical Domain Applications
AI总结：
本文综述了稀疏混合专家模型从算法基础到去中心化架构及垂直领域应用的发展，系统探讨了其核心原理、去中心化范式和关键挑战，为研究者和实践者提供全面的参考。
摘要：
稀疏混合专家（MoE）架构已发展为一种强大的方法，用于在可比计算成本下扩展深度学习模型以容纳更多参数。作为大型语言模型（LLM）的重要分支，MoE模型基于路由网络仅激活专家子集。这种稀疏条件计算机制显著提高了计算效率，为更大的可扩展性和成本效率开辟了有前途的道路。它不仅增强了各种横向领域的下游应用，如自然语言处理、计算机视觉和多模态，还展示了在垂直领域中的广泛适用性。尽管MoE模型在各个领域中的流行和应用日益增长，但许多重要领域缺乏对MoE近期进展的系统性探索。现有的MoE综述存在覆盖不全或未对关键领域进行深入探索的局限性。本文旨在填补这些空白。在本文中，首先我们考察了MoE的基础原理，深入探讨了其核心组件——路由网络和专家网络。随后，我们超越了集中化范式，转向去中心化范式，这解锁了去中心化基础设施的巨大未被开发潜力，使MoE开发民主化，为更广泛的社区带来更大的可扩展性和成本效率。此外，我们还专注于探索其垂直领域应用。最后，我们还识别了关键挑战和有前途的未来研究方向。据我们所知，本文是目前MoE领域最全面的综述。我们希望本文能为研究人员和实践者提供有价值的资源，使他们能够导航并保持对最新进展的了解。
英文摘要：
The sparse Mixture of Experts(MoE) architecture has evolved as a powerful approach for scaling deep learning models to more parameters with comparable computation cost. As an important branch of large language model(LLM), MoE model only activate a subset of experts based on a routing network. This sparse conditional computation mechanism significantly improves computational efficiency, paving a promising path for greater scalability and cost-efficiency. It not only enhance downstream applications such as natural language processing, computer vision, and multimodal in various horizontal domains, but also exhibit broad applicability across vertical domains. Despite the growing popularity and application of MoE models across various domains, there lacks a systematic exploration of recent advancements of MoE in many important fields. Existing surveys on MoE suffer from limitations such as lack coverage or none extensively exploration of key areas. This survey seeks to fill these gaps. In this paper, Firstly, we examine the foundational principles of MoE, with an in-depth exploration of its core components-the routing network and expert network. Subsequently, we extend beyond the centralized paradigm to the decentralized paradigm, which unlocks the immense untapped potential of decentralized infrastructure, enables democratization of MoE development for broader communities, and delivers greater scalability and cost-efficiency. Furthermore we focus on exploring its vertical domain applications. Finally, we also identify key challenges and promising future research directions. To the best of our knowledge, this survey is currently the most comprehensive review in the field of MoE. We aim for this article to serve as a valuable resource for both researchers and practitioners, enabling them to navigate and stay up-to-date with the latest advancements.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
