arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2606.04438cs.LGcs.AI
收藏

LoopMoE:统一迭代计算与混合专家模型用于语言建模

LoopMoE: Unifying Iterative Computation with Mixture-of-Experts for Language Modeling

Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Huawei Technologies Co.,Ltd.(华为技术有限公司)

Wenkai Chen, Tianshu Li, Wenyong Huang, Yichun Yin, Lifeng Shang, Chengwei Qin

AI总结:

提出LoopMoE,通过迭代自适应层归一化和容量平衡策略,在相同参数和FLOPs下,循环MoE语言模型在多个基准上优于标准MoE。

摘要:

混合专家模型(MoE)和循环架构分别沿着参数容量和有效深度两个正交维度扩展模型。然而,主流的循环架构依赖于密集主干,将参数数量与每个token的FLOPs耦合,这使得在匹配预算下无法隔离迭代计算的效果。为此,我们提出了LoopMoE,一种循环MoE语言模型,通过两种设计将稀疏路由与迭代权重共享计算相结合。第一种是IterAdaLN,它通过联合以迭代索引和每个token隐藏状态为条件的调制信号来解决权重共享对称性。第二种是一种容量平衡策略,恢复了经过良好调整的非循环参考模型的注意力到FFN活跃参数比率。这些设计共同实现了在相同总参数、每个token FLOPs和活跃子层比率下,循环MoE与标准MoE的首次严格受控的头对头评估。在3B规模下,LoopMoE在9个下游基准测试中的8个上优于标准MoE,平均提升超过1个点。在9B规模下,LoopMoE继续优于匹配的标准MoE,表明架构优势在更大规模下持续存在。我们的工作建立了稀疏性和循环性的受控综合,并为循环语言模型指明了一个有前景的方向。

英文摘要:

Mixture-of-Experts (MoE) and looped architectures scale models along two orthogonal axes, namely parameter capacity and effective depth. However, mainstream looped architectures rely on dense backbones that couple parameter count with per-token FLOPs, which makes it impossible to isolate the effect of iterative computation under matched budgets. To this end, we present LoopMoE, a looped MoE language model that integrates sparse routing with iterative weight-shared computation through two designs. The first is IterAdaLN, which resolves weight-sharing symmetry via a modulation signal jointly conditioned on the iteration index and the per-token hidden state. The second is a capacity-balancing strategy that recovers the attention-to-FFN active parameter ratio of well-tuned non-looped references. Together, these designs enable the first strictly controlled, head-to-head evaluation of a looped MoE against a Vanilla MoE under identical total parameters, per-token FLOPs, and active sublayer ratios. Across nine downstream benchmarks, LoopMoE's average improvement over its matched vanilla MoE increases from over 1 point at the 3B scale to approximately 3 points at the 9B scale. These results provide initial evidence that the benefits of iterative sparse computation may strengthen with scale, positioning LoopMoE as a promising architecture for scalable looped language models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑