专家联盟：将分层路由应用于等价分解的Transformer
Union of Experts: Adapting Hierarchical Routing to Equivalently Decomposed Transformer
AI总结：
针对传统MoE架构协调动态次优、未有效扩展到注意力块的问题，提出UoE，将Transformer等价分解为专家组并采用分层路由，在多任务中提升性能且降低计算量，优于现有模型。
摘要：
混合专家模型（Mixture-of-Experts, MoE）在保持计算效率的同时提升了模型性能，使其非常适合大规模应用。传统混合专家（MoE）架构存在次优协调动态问题，孤立的专家操作使模型面临过拟合风险。此外，它们尚未有效扩展到注意力块，限制了进一步的效率提升。为解决这些问题，我们提出专家联盟（Union-of-Experts, UoE），将Transformer模型分解为等价的专家组，并应用分层路由机制将输入子空间分配给专门的专家。我们的方法通过四项关键创新推进MoE设计：（1）将非MoE模型划分为功能等价的专家来构建专家组；（2）开发分层路由范式，整合基于补丁的数据选择和专家选择策略；（3）将MoE设计扩展到注意力块；（4）提出硬件优化的并行化方案，利用批量矩阵乘法实现高效专家计算。实验表明，我们的UoE模型在图像和自然语言领域的多项任务中优于全注意力模型、最先进的MoE模型和高效Transformer。在语言建模任务中，UoE与性能最佳的MoE方法相比，困惑度平均降低2.38，而FLOPs仅为其76%。在Long Range Arena基准测试中，它的平均得分比所有对比模型至少高0.68%，且FLOPs仅为最佳MoE方法的50%。在图像分类中，它在保持可比FLOPs的同时，比最佳模型平均准确率提升1.75%。源代码可在https://github.com/YujiaoYang-work/UoE获取。
英文摘要：
Mixture-of-Experts (MoE) enhances model performance while maintaining computational efficiency, making it well-suited for large-scale applications. Conventional mixture-of-experts (MoE) architectures suffer from suboptimal coordination dynamics, where isolated expert operations expose the model to overfitting risks. Moreover, they have not been effectively extended to attention blocks, which limits further efficiency improvements. To tackle these issues, we propose Union-of-Experts (UoE), which decomposes the transformer model into an equivalent group of experts and applies a hierarchical routing mechanism to allocate input subspaces to specialized experts. Our approach advances MoE design with four key innovations: (1) Constructing expert groups by partitioning non-MoE models into functionally equivalent specialists (2) Developing a hierarchical routing paradigm that integrates patch-wise data selection and expert selection strategies. (3) Extending the MoE design to attention blocks. (4) Proposing a hardware-optimized parallelization scheme that exploits batched matrix multiplications for efficient expert computation. The experiments demonstrate that our UoE model surpasses Full Attention, state-of-the-art MoEs and efficient transformers in several tasks across image and natural language domains. In language modeling tasks, UoE achieves an average reduction of 2.38 in perplexity compared to the best-performing MoE method with only 76% of its FLOPs. In the Long Range Arena benchmark, it demonstrates an average score at least 0.68% higher than all comparison models, with only 50% of the FLOPs of the best MoE method. In image classification, it yields an average accuracy improvement of 1.75% over the best model while maintaining comparable FLOPs. The source codes are available at https://github.com/YujiaoYang-work/UoE.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
