arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.07871cs.LGcs.AIcs.CL
收藏

细粒度混合专家的缩放定律

Scaling Laws for Fine-Grained Mixture of Experts

University of Warsaw(华沙大学);IDEAS NCBR;IPPT PAN(波兰科学院物理化学研究所);TradeLink;Nomagic
Jakub Krajewski, Jan Ludziejewski, Kamil Adamczewski, Maciej Pióro, Michał Krutul, Szymon Antoniak, Kamil Ciebiera, Krystian Król, Tomasz Odrzygóźdź, Piotr Sank… 展开作者

Jakub Krajewski, Jan Ludziejewski, Kamil Adamczewski, Maciej Pióro, Michał Krutul, Szymon Antoniak, Kamil Ciebiera, Krystian Król, Tomasz Odrzygóźdź, Piotr Sankowski, Marek Cygan, Sebastian Jaszczur

AI总结:

本文引入粒度超参数以精确控制专家大小,建立了细粒度混合专家模型的缩放定律,推导出给定计算预算下的最优训练配置,并证明MoE模型优于稠密Transformer且效率优势随规模扩大而增加。

摘要:

混合专家模型已成为降低大型语言模型计算成本的主要解决方案。在这项工作中,我们分析了其缩放特性,并纳入了更广泛的变量范围。具体而言,我们引入了一个新的超参数,即粒度,通过调整它可以精确控制专家的大小。在此基础上,我们建立了细粒度 MoE 的缩放定律,综合考虑了训练 token 数量、模型大小和粒度。利用这些定律,我们推导出了给定计算预算下的最优训练配置。我们的发现不仅表明 MoE 模型始终优于稠密 Transformer,而且凸显了随着模型大小和训练预算的扩大,稠密模型与 MoE 模型之间的效率差距会不断扩大。此外,我们证明了在 MoE 中将专家大小设定为与前馈层相匹配的常见做法,在几乎任何计算预算下都不是最优的。

英文摘要:

Mixture of Experts (MoE) models have emerged as a primary solution for reducing the computational cost of Large Language Models. In this work, we analyze their scaling properties, incorporating an expanded range of variables. Specifically, we introduce a new hyperparameter, granularity, whose adjustment enables precise control over the size of the experts. Building on this, we establish scaling laws for fine-grained MoE, taking into account the number of training tokens, model size, and granularity. Leveraging these laws, we derive the optimal training configuration for a given computational budget. Our findings not only show that MoE models consistently outperform dense Transformers but also highlight that the efficiency gap between dense and MoE models widens as we scale up the model size and training budget. Furthermore, we demonstrate that the common practice of setting the size of experts in MoE to mirror the feed-forward layer is not optimal at almost any computational budget.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑