arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2604.01622cs.LGcs.CL
收藏

专家选择路由使扩散语言模型实现自适应计算

Expert-Choice Routing Enables Adaptive Computation in Diffusion Language Models

University of Wisconsin-Madison(威斯康星大学麦迪逊分校);Scitix;Cornell University(康奈尔大学);Duke University(杜克大学);UC Davis(加州大学戴维斯分校);Southern University of Science and Technology(南方科技大学)

Shuibai Zhang, Caspian Zhuang, Chihan Cui, Zhihan Yang, Fred Zhangzhi Peng, Yanxin Zhang, Haoyue Bai, Zack Jia, Yang Zhou, Guanhua Chen, Ming Liu

AI总结:

本文提出专家选择路由方法,通过确定性负载平衡提升扩散语言模型的吞吐量和收敛速度,并展示如何通过时间步依赖的专家容量优化计算分配,从而在匹配FLOPs下提升性能。

摘要:

扩散语言模型(DLMs)能够实现并行、非自回归的文本生成,但现有DLM混合专家(MoE)模型继承自自回归系统的token选择(TC)路由,导致负载不均衡和刚性计算分配。我们证明专家选择(EC)路由更适合DLMs:它通过设计实现确定性负载平衡,比TC提供更高的吞吐量和更快的收敛速度。基于EC容量可外部控制的特性,我们引入时间步依赖的专家容量,根据去噪步骤变化专家分配。我们发现,将更多容量分配给低掩码比率步骤在匹配FLOPs下始终取得最佳性能,并提供机制解释:低掩码比率上下文中的token表现出数量级更高的学习效率,因此集中在这些步骤上获得最大的边际回报。最后,我们展示现有预训练TC DLMs可通过仅替换路由器即可转换为EC,从而在多样下游任务中实现更快的收敛速度和改进的准确性。这些结果确立EC路由作为DLM MoE模型的更优范式,并证明DLMs中的计算可以被视为适应性策略而非固定架构常数。代码可在https://github.com/zhangshuibai/EC-DLM获取。

英文摘要:

Diffusion language models (DLMs) enable parallel, non-autoregressive text generation, yet existing DLM mixture-of-experts (MoE) models inherit token-choice (TC) routing from autoregressive systems, leading to load imbalance and rigid computation allocation. We show that expert-choice (EC) routing is a better fit for DLMs: it provides deterministic load balancing by design, yielding higher throughput and faster convergence than TC. Building on the property that EC capacity is externally controllable, we introduce timestep-dependent expert capacity, which varies expert allocation according to the denoising step. We find that allocating more capacity to low-mask-ratio steps consistently achieves the best performance under matched FLOPs, and provide a mechanistic explanation: tokens in low-mask-ratio contexts exhibit an order-of-magnitude higher learning efficiency, so concentrating compute on these steps yields the largest marginal return. Finally, we show that existing pretrained TC DLMs can be retrofitted to EC by replacing only the router, achieving faster convergence and improved accuracy across diverse downstream tasks. Together, these results establish EC routing as a superior paradigm for DLM MoE models and demonstrate that computation in DLMs can be treated as an adaptive policy rather than a fixed architectural constant. Code is available at https://github.com/zhangshuibai/EC-DLM.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑