Sparsely gated Mixture of Experts (MoE) models have been shown to be a compute-efficient method to scale model capacity for multilingual machine translation. However, for low-resource tasks, MoE models severely over-fit. We show effective regularization strategies, namely dropout techniques for MoE layers in EOM and FOM, Conditional MoE Routing and Curriculum Learning methods that prevent over-fitting and improve the performance of MoE models on low-resource tasks without adversely affecting high-resource tasks. On a massively multilingual machine translation benchmark, our strategies result in about +1 chrF++ improvement in very low resource language pairs. We perform an extensive analysis of the learned MoE routing to better understand the impact of our regularization methods and how we can improve them.
The Mixture of Experts (MoE) models are an emerging class of sparsely activated deep learning models that have sublinear compute costs with respect to their parameters. In contrast with dense models, the sparse architecture of MoE offers opportunities for drastically growing model size with significant accuracy gain while consuming much lower compute budget. However, supporting large scale MoE training also has its own set of system and modeling challenges. To overcome the challenges and embrace the opportunities of MoE, we first develop a system capable of scaling MoE models efficiently to tr…
Mixture of Experts (MoE) models with conditional execution of sparsely activated layers have enabled training models with a much larger number of parameters. As a result, these models have achieved significantly better quality on various natural language processing tasks including machine translation. However, it remains challenging to deploy such models in real-life scenarios due to the large memory requirements and inefficient inference. In this work, we introduce a highly efficient inference framework with several optimization approaches to accelerate the computation of sparse models and cu…
RotMoLE: Enhancing Mixture of Low-Rank Experts through Rotational Gating Mechanism
Tsinghua University（清华大学）；Beijing Information Science and Technology University（北京信息科技大学）；National University of Singapore（新加坡国立大学）；Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Beijing University of Posts and Telecommunications（北京邮电大学）
In deep learning, models typically reuse the same parameters for all inputs. Mixture of Experts (MoE) defies this and instead selects different parameters for each incoming example. The result is a sparsely-activated model -- with outrageous numbers of parameters -- but a constant computational cost. However, despite several notable successes of MoE, widespread adoption has been hindered by complexity, communication costs and training instability -- we address these with the Switch Transformer. We simplify the MoE routing algorithm and design intuitive improved models with reduced communicatio…
Understanding Multilingualism in Mixture-of-Experts LLMs: Routing Mechanism, Expert Specialization, and Layerwise Steering
National University of Singapore（新加坡国立大学）；Beijing University of Posts and Telecommunication（北京邮电大学）；Harbin Institute of Technology（哈尔滨工业大学）；University of Science and Technology of China（中国科学技术大学）
When Meaning Travels: A Granular Lens on Hybrid-MoE's Role in Idiomatic Understanding for Language Models
Department of Computer Science and Engineering, Indian Institute of Technology Patna, India（印度理工学院帕纳瓦分校计算机科学与工程系）；School of Information Technology, King Mongkut’s Institute of Technology Ladkrabang, Thailand（泰国拉差班国王理工大学信息科技学院）
Kazan Federal University（喀山联邦大学）；Joint Supercomputer Center of the Russian Academy of Sciences（俄罗斯科学院联合超级计算机中心）
The paper is dedicated to the use of LodView for navigation over the multilingual Linguistic Linked Open Data cloud. First, we define the class of Pubby-like tools, that LodView belongs to, and clarify the relation of this class to the classes of URI dereferenciation tools, RDF browsers and LOD visualization tools. Second, we reveal several limitations of LodView that impede its use for the designated purpose, and propose improvements to be made for fixing these limitations. These improvements are: 1) resolution of Cyrillic URIs; 2) decoding Cyrillic URIs in Turtle representations of resources…
Although recent Massively Multilingual Language Models (MMLMs) like mBERT and XLMR support around 100 languages, most existing multilingual NLP benchmarks provide evaluation data in only a handful of these languages with little linguistic diversity. We argue that this makes the existing practices in multilingual evaluation unreliable and does not provide a full picture of the performance of MMLMs across the linguistic landscape. We propose that the recent work done in Performance Prediction for NLP tasks can serve as a potential solution in fixing benchmarking in Multilingual NLP by utilizing …
Unraveling Babel: Exploring Multilingual Activation Patterns of LLMs and Their Applications
Liangzhu Laboratory, Zhejiang University（浙江大学良渚实验室）；AI Research Center, WeDoctor Cloud（微医云AI研究中心）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；The Hong Kong University of Science and Technology（香港科技大学）；School of Public Health, Zhejiang University（浙江大学公共卫生学院）
Less, but Better: Efficient Multilingual Expansion for LLMs via Layer-wise Mixture-of-Experts
Key Laboratory of Big Data & Artificial Intelligence in Transportation, (Beijing Jiaotong University), Ministry of Education（大数据与人工智能交通大数据实验室，北京交通大学，教育部）；School of Computer Science and Technology, Beijing Jiaotong University, Beijing, China（计算机科学与技术学院，北京交通大学，北京，中国）；Pattern Recognition Center, WeChat AI, Tencent Inc, China（模式识别中心，微信AI，腾讯公司，中国）
