Scalable and Efficient MoE Training for Multitask Multilingual Models
Microsoft（微软）
Young Jin Kim, Ammar Ahmad Awan, Alexandre Muzio, Andres Felipe Cruz Salinas, Liyang Lu, Amr Hendy, Samyam Rajbhandari, Yuxiong He, Hany Hassan Awadalla
The Mixture of Experts (MoE) models are an emerging class of sparsely activated deep learning models that have sublinear compute costs with respect to their parameters. In contrast with dense models, the sparse architecture of MoE offers opportunities for drastically growing model size with significant accuracy gain while consuming much lower compute budget. However, supporting large scale MoE training also has its own set of system and modeling challenges. To overcome the challenges and embrace the opportunities of MoE, we first develop a system capable of scaling MoE models efficiently to trillions of parameters. It combines multi-dimensional parallelism and heterogeneous memory technologies harmoniously with MoE to empower 8x larger models on the same hardware compared with existing work. Besides boosting system efficiency, we also present new training methods to improve MoE sample efficiency and leverage expert pruning strategy to improve inference time efficiency. By combining the efficient system and training methods, we are able to significantly scale up large multitask multilingual models for language generation which results in a great improvement in model accuracy. A model trained with 10 billion parameters on 50 languages can achieve state-of-the-art performance in Machine Translation (MT) and multilingual natural language generation tasks. The system support of efficient MoE training has been implemented and open-sourced with the DeepSpeed library.
MoE-Loco: Mixture of Experts for Multitask Locomotion
IIIS, Tsinghua University, Beijing, China（清华大学信息学院，北京，中国）；Shanghai Qi Zhi Institute, Shanghai, China（上海启智研究院，上海，中国）；Massachusetts Institute of Technology, MA, USA（麻省理工学院，马萨诸塞州，美国）
Sparsely gated Mixture of Experts (MoE) models have been shown to be a compute-efficient method to scale model capacity for multilingual machine translation. However, for low-resource tasks, MoE models severely over-fit. We show effective regularization strategies, namely dropout techniques for MoE layers in EOM and FOM, Conditional MoE Routing and Curriculum Learning methods that prevent over-fitting and improve the performance of MoE models on low-resource tasks without adversely affecting high-resource tasks. On a massively multilingual machine translation benchmark, our strategies result i…
BrainCSD: A Hierarchical Consistency-Driven MoE Foundation Model for Unified Connectome Synthesis and Multitask Brain Trait Prediction
Department of Computer Science and Technology, Harbin Institute of Technology（哈尔滨工业大学计算机科学与技术系）；School of Intelligence Science and Engineering, College of Artificial Intelligence, Harbin Institute of Technology（哈尔滨工业大学智能科学与工程学院）；School of Biomedical Engineering, National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, Guangdong Key Laboratory for Biomedical, Measurements and Ultrasound Imaging, Shenzhen University Medical School, Shenzhen University（深圳大学医学院生物医学工程学院）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）
BrainCSD 提出层次一致性驱动的 MoE 基础模型，通过 ROI 特异性、编码-激活和网络感知细化三个组件联合合成 FC/SC 并支持多任务预测，在缺失模态下取得 SOTA 性能。
A Survey on Model MoErging: Recycling and Routing Among Specialized Experts for Collaborative Learning
UNC-Chapel Hill（北卡罗来纳大学教堂山分校）；University of Toronto（多伦多大学）；Vector Institue（向量研究所）；MIT（麻省理工学院）；MIT-IBM Watson AI Lab（麻省理工-IBM沃森人工智能实验室）；Microsoft Research（微软研究院）；Mila（蒙特利尔人工智能研究院）
Mixture of Experts (MoE) models with conditional execution of sparsely activated layers have enabled training models with a much larger number of parameters. As a result, these models have achieved significantly better quality on various natural language processing tasks including machine translation. However, it remains challenging to deploy such models in real-life scenarios due to the large memory requirements and inefficient inference. In this work, we introduce a highly efficient inference framework with several optimization approaches to accelerate the computation of sparse models and cu…
Sparse Mixture-of-Experts (MoE) has been a successful approach for scaling multilingual translation models to billions of parameters without a proportional increase in training computation. However, MoE models are prohibitively large and practitioners often resort to methods such as distillation for serving. In this work, we investigate routing strategies at different granularity (token, sentence, task) in MoE models to bypass distillation. Experiments on WMT and a web-scale dataset suggest that task-level routing (task-MoE) enables us to extract smaller, ready-to-deploy sub-networks from larg…
Department of Research（研究部）；ByteDance Inc（字节跳动公司）；Department of CS（计算机科学系）；Imperial College London（伦敦帝国理工学院）；Purdue University（普渡大学）；Emory University（埃默里大学）；Department of Computer Science（计算机科学系）；Heriot-Watt University（赫罗特-瓦特大学）；AI Agent Lab（AI代理实验室）；Vokram Group（Vokram集团）；Department of Anatomical Pathology（解剖病理学系）；Singapore General Hospital（新加坡中央医院）
This paper aims for a potential architectural improvement for multilingual learning and asks: Can different tasks from different languages be modeled in a monolithic framework, i.e. without any task/language-specific module? The benefit of achieving this could open new doors for future multilingual research, including allowing systems trained on low resources to be further assisted by other languages as well as other tasks. We approach this goal by developing a learning framework named Polyglot Prompting to exploit prompting methods for learning a unified semantic space for different languages…
The Mixture-of-Experts (MoE) technique can scale up the model size of Transformers with an affordable computational overhead. We point out that existing learning-to-route MoE methods suffer from the routing fluctuation issue, i.e., the target expert of the same input may change along with training, but only one expert will be activated for the input during inference. The routing fluctuation tends to harm sample efficiency because the same input updates different experts but only one is finally used. In this paper, we propose StableMoE with two training stages to address the routing fluctuation…
University of Colorado Boulder（科罗拉多大学博尔德分校）；University of Michigan Ann Arbor（密歇根大学安娜堡分校）；San Diego State University（圣地亚哥州立大学）
This paper describes our system for the SemEval2022 task of matching dictionary glosses to word embeddings. We focus on the Reverse Dictionary Track of the competition, which maps multilingual glosses to reconstructed vector representations. More specifically, models convert the input of sentences to three types of embeddings: SGNS, Char, and Electra. We propose several experiments for applying neural network cells, general multilingual and multitask structures, and language-agnostic tricks to the task. We also provide comparisons over different types of word embeddings and ablation studies to…
In deep learning, models typically reuse the same parameters for all inputs. Mixture of Experts (MoE) defies this and instead selects different parameters for each incoming example. The result is a sparsely-activated model -- with outrageous numbers of parameters -- but a constant computational cost. However, despite several notable successes of MoE, widespread adoption has been hindered by complexity, communication costs and training instability -- we address these with the Switch Transformer. We simplify the MoE routing algorithm and design intuitive improved models with reduced communicatio…
Harbin Institute of Technology（哈尔滨工业大学）；Microsoft Research Asia（微软亚洲研究院）；Microsoft（微软）；Microsoft Research Redmond（微软研究院（雷德蒙德））
We present M3P, a Multitask Multilingual Multimodal Pre-trained model that combines multilingual pre-training and multimodal pre-training into a unified framework via multitask pre-training. Our goal is to learn universal representations that can map objects occurred in different modalities or texts expressed in different languages into a common semantic space. In addition, to explicitly encourage fine-grained alignment between images and non-English languages, we also propose Multimodal Code-switched Training (MCT) to combine monolingual pre-training and multimodal pre-training via a code-swi…
In Natural Language Processing (NLP), one traditionally considers a single task (e.g. part-of-speech tagging) for a single language (e.g. English) at a time. However, recent work has shown that it can be beneficial to take advantage of relatedness between tasks, as well as between languages. In this work I examine the concept of relatedness and explore how it can be utilised to build NLP models that require less manually annotated data. A large selection of NLP tasks is investigated for a substantial language sample comprising 60 languages. The results show potential for joint multitask and mu…
While few-shot learning as a transfer learning paradigm has gained significant traction for scenarios with limited data, it has primarily been explored in the context of building unimodal and unilingual models. Furthermore, a significant part of the existing literature in the domain of few-shot multitask learning perform in-context learning which requires manually generated prompts as the input, yielding varying outcomes depending on the level of manual prompt-engineering. In addition, in-context learning suffers from substantial computational, memory, and storage costs which eventually leads …
Georgia Institute of Technology（佐治亚理工学院）；Microsoft（微软）
Sparsely activated models (SAMs), such as Mixture-of-Experts (MoE), can easily scale to have outrageously large amounts of parameters without significant increase in computational cost. However, SAMs are reported to be parameter inefficient such that larger models do not always lead to better performance. While most on-going research focuses on improving SAMs models by exploring methods of routing inputs to experts, our analysis reveals that such research might not lead to the solution we expect, i.e., the commonly-used routing methods based on gating mechanisms do not work better than randoml…
Qatar Computing Research Institute（卡塔尔计算研究所）；Sofia University "St. Kliment Ohridski"（索菲亚大学"圣克莱孟·奥赫里迪斯"）；University of Padova（帕多瓦大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）
University of Michigan, Ann Arbor（密歇根大学安娜堡分校）；Microsoft（微软）
Sparsely activated transformers, such as Mixture of Experts (MoE), have received great interest due to their outrageous scaling capability which enables dramatical increases in model size without significant increases in computational cost. To achieve this, MoE models replace the feedforward sub-layer with Mixture-of-Experts sub-layer in transformers and use a gating network to route each token to its assigned experts. Since the common practice for efficient training of such models requires distributing experts and tokens across different machines, this routing strategy often incurs huge cross…
CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generation
University of California, Santa Barbara（加利福尼亚大学圣巴巴拉分校）；School of Software Engineering, Xi’an Jiaotong University（西安交通大学软件学院）；Zhejiang University（浙江大学）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Alibaba Group（阿里巴巴集团）；TU Wien（维也纳技术大学）；University of Chinese Academy of Sciences（中国科学院大学）
CreoleVal: Multilingual Multitask Benchmarks for Creoles
Aalborg University（奥尔堡大学）；KU Leuven（鲁汶大学）；National Institute of Information and Communications Technology（日本信息通信研究机构）；University of Copenhagen（哥本哈根大学）；University of Electronic Science and Technology of China（电子科技大学）；Meta；Zamboanga State College of Marine Sciences and Technology（三宝颜加国立海洋科学与技术学院）；University of Newcastle（纽卡斯尔大学）；University of Surrey（萨里大学）；Kean University（肯恩大学）；Linköping University（林雪平大学）；Université Paris Nanterre（巴黎第十大学）；Massachusetts Institute of Technology（麻省理工学院）
