揭示大型语言模型中混合专家（MoE）内部的隐藏协作
Unveiling Hidden Collaboration within Mixture-of-Experts in Large Language Models
AI总结：
针对MoE LLMs专家协作机制不明的问题，提出HSDL方法识别协作模式、CAEP算法剪枝低贡献专家，实验验证模式与输入类型相关且语义显著，剪枝平均提升性能2.5%，为模型优化与可解释性提供支持。
摘要：
基于混合专家（MoE）的大型语言模型（MoE LLMs）通过将输入动态路由到专业专家，在多任务适应性方面展现出显著潜力。尽管取得成功，但专家间的协作机制仍未得到充分理解，限制了这些模型的可解释性和优化。本文聚焦两个关键问题：（1）识别专家协作模式；（2）通过专家剪枝优化MoE LLMs。针对第一个问题，我们提出分层稀疏字典学习（HSDL）方法，揭示专家间的协作模式。针对第二个问题，我们引入贡献感知专家剪枝（CAEP）算法，有效剪枝低贡献专家。广泛实验表明，专家协作模式与特定输入类型密切相关，并在各类任务中表现出语义显著性；剪枝实验显示，我们的方法平均提升整体性能2.5%，优于现有方法。这些发现为增强MoE LLMs的效率和可解释性提供了宝贵见解，有助于更清晰地理解专家交互并改进模型优化。
英文摘要：
Mixture-of-Experts based large language models (MoE LLMs) have shown significant promise in multitask adaptability by dynamically routing inputs to specialized experts. Despite their success, the collaborative mechanisms among experts are still not well understood, limiting both the interpretability and optimization of these models. In this paper, we focus on two critical issues: (1) identifying expert collaboration patterns, and (2) optimizing MoE LLMs through expert pruning. To address the first issue, we propose a hierarchical sparse dictionary learning (HSDL) method that uncovers the collaboration patterns among experts. For the second issue, we introduce the Contribution-Aware Expert Pruning (CAEP) algorithm, which effectively prunes low-contribution experts. Our extensive experiments demonstrate that expert collaboration patterns are closely linked to specific input types and exhibit semantic significance across various tasks. Moreover, pruning experiments show that our approach improves overall performance by 2.5\% on average, outperforming existing methods. These findings offer valuable insights into enhancing the efficiency and interpretability of MoE LLMs, offering a clearer understanding of expert interactions and improving model optimization.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
