University of Warsaw(华沙大学);IDEAS NCBR;IPPT PAN(波兰科学院物理化学研究所);TradeLink;Nomagic
Jakub Krajewski, Jan Ludziejewski, Kamil Adamczewski, Maciej Pióro, Michał Krutul, Szymon Antoniak, Kamil Ciebiera, Krystian Król, Tomasz Odrzygóźdź, Piotr Sank… 展开作者
Jakub Krajewski, Jan Ludziejewski, Kamil Adamczewski, Maciej Pióro, Michał Krutul, Szymon Antoniak, Kamil Ciebiera, Krystian Król, Tomasz Odrzygóźdź, Piotr Sankowski, Marek Cygan, Sebastian Jaszczur
Mixture of Experts (MoE) models have emerged as a primary solution for reducing the computational cost of Large Language Models. In this work, we analyze their scaling properties, incorporating an expanded range of variables. Specifically, we introduce a new hyperparameter, granularity, whose adjustment enables precise control over the size of the experts. Building on this, we establish scaling laws for fine-grained MoE, taking into account the number of training tokens, model size, and granularity. Leveraging these laws, we derive the optimal training configuration for a given computational budget. Our findings not only show that MoE models consistently outperform dense Transformers but also highlight that the efficiency gap between dense and MoE models widens as we scale up the model size and training budget. Furthermore, we demonstrate that the common practice of setting the size of experts in MoE to mirror the feed-forward layer is not optimal at almost any computational budget.
Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient
University of Warsaw(华沙大学);Wroclaw University of Science and Technology(弗罗茨瓦夫理工大学);Institute of Mathematics, Polish Academy of Sciences(波兰科学院数学研究所);Institute of Fundamental Technological Research, Polish Academy of Sciences(波兰科学院基础技术研究所)
Task-Stratified Knowledge Scaling Laws for Post-Training Quantized Large Language Models
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences(中国科学院大学先进交叉学科学院);The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所认知与决策智能复杂系统重点实验室);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);College of Computer Science, Inner Mongolia University(内蒙古大学计算机学院)
Mixtures of Experts Unlock Parameter Scaling for Deep RL
Mila - Québec AI Institute(米拉-魁北克人工智能研究所);Université de Montréal(蒙特利尔大学);Google DeepMind(谷歌DeepMind);University of Oxford(牛津大学);McGill University(麦吉尔大学)
Enhancing Talent Search Ranking with Role-Aware Expert Mixtures and LLM-based Fine-Grained Job Descriptions
HKUST (GZ)(香港科技大学(广州));Alibaba Group(阿里巴巴集团);HKUST(香港科技大学);Zhijiang Lab(浙江实验室);Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)(广东人工智能与数字经济实验室(深圳))
Holistic Scaling Laws for Optimal Mixture-of-Experts Architecture Optimization
College of Computer Science and Artificial Intelligence, Fudan University(复旦大学计算机科学与人工智能学院);Hi Lab, Xiaohongshu Inc.(小红书公司Hi实验室);Project Numina(项目Numina);School of Data Science, Fudan University(复旦大学数据科学学院)
BigMac: A Communication-Efficient Mixture-of-Experts Model Structure for Fast Training and Inference
University of Science and Technology of China(中国科学技术大学);Huawei Technologies(华为技术有限公司);Institute of Artificial Intelligence, Hefei Comprehensive National Science Center(合肥综合性国家科学中心人工智能研究院)
National University of Singapore(新加坡国立大学);Center for AI Research, VinUniversity(Vin大学人工智能研究中心);Independent Researcher(独立研究者);Hanoi University of Science and Technology(河内科学技术大学)
FedMoE-DA: Federated Mixture of Experts via Domain Aware Fine-grained Aggregation
Sun Yat-Sen University, China(中山大学);The University of Hong Kong, Hong Kong(香港大学);Nanyang Technological University, Singapore(南洋理工大学);National University of Defense Technology, China(国防科技大学)
Each Rank Could be an Expert: Single-Ranked Mixture of Experts LoRA for Multi-Task Learning
Zhejiang University(浙江大学);Shanghai Innovation Institute(上海创新研究院);ByteDance Inc.(字节跳动公司);Fudan University(复旦大学);The Chinese University of Hong Kong(香港中文大学)