arXiv 2606.010622026-05-31cs.AI

DAG-MoE：从简单混合到专家混合中的结构聚合

DAG-MoE: From Simple Mixture to Structural Aggregation in Mixture-of-Experts

University of Science and Technology of China（中国科学技术大学）；Tsinghua University（清华大学）；Nanyang Technological University（南洋理工大学）；University of California, Berkeley（加州大学伯克利分校）；University of Washington（华盛顿大学）

Jiarui Feng, Hanqing Zeng, Karish Grover, Ruizhong Qiu, Yinglong Xia, Qiang Zhang, Qifan Wang, Ren Chen, Dongqi Fu, Jiayi Liu, Zhoukai Zhao, Xiangjun Fan, Benyu Zhang, Yixin Chen

备注：: Accepted by ICML 2026

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出DAG-MoE框架，通过轻量级模块自动学习选定专家之间的最优聚合结构，以替代标准加权求和聚合，从而在不改变专家或路由器的情况下扩展专家组合空间并实现单层多步推理，在预训练和微调中均优于传统MoE基线。

摘要：

混合专家（MoE）模型已成为在大型语言模型中解耦参数数量与计算成本的主流方法，但有效扩展MoE性能仍是一个挑战。先前的工作表明，细粒度专家扩大了专家组合空间并提高了灵活性，但它们也带来了大量的路由开销，造成了新的可扩展性瓶颈。在本文中，我们探索了扩展的互补轴——专家输出的聚合方式。我们从理论上证明，用结构聚合替代标准加权求和聚合可以在不改变专家或路由器的情况下扩展专家组合空间，并使得在单个MoE层内实现多步推理成为可能。为此，我们提出了DAG-MoE，一个稀疏MoE框架，它采用轻量级模块自动学习所选专家之间的最优聚合结构。在标准语言建模设置下的大量实验表明，DAG-MoE在预训练和微调中均持续提升了性能，超越了传统的MoE基线。

英文摘要：

Mixture-of-Experts (MoE) models have become a leading approach for decoupling parameter count from computational cost in large language models, yet effectively scaling MoE performance remains a challenge. Prior work shows that fine-grained experts enlarge the space of expert combinations and improve flexibility, but they also impose substantial routing overhead, creating a new scalability bottleneck. In this paper, we explore a complementary axis for scaling -- how expert outputs are aggregated. We theoretically show that replacing the standard weighted-summation aggregation with structural aggregation expands the expert-combination space without altering the experts or router, and enables possible multi-step reasoning within a single MoE layer. To this end, we propose DAG-MoE, a sparse MoE framework that employs a lightweight module to automatically learn the optimal aggregation structure among the selected experts. Extensive experiments under standard language modeling settings show that DAG-MoE consistently improves performance in both pretraining and fine-tuning, surpassing traditional MoE baselines.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。