LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training
Soochow University（苏州大学）；Shanghai AI Laboratory（上海人工智能实验室）；Shanghai Jiao Tong University（上海交通大学）；Fudan University（复旦大学）；The Chinese University of Hong Kong（香港中文大学）
混合专家（Mixture-of-Experts, MoE）作为一种极具前景的大语言模型（large language models, LLMs）扩展框架，正日益受到关注。然而，大规模从零开始训练MoE仍存在数据需求高、训练不稳定的问题。受此局限启发，我们研究基于现有稠密大语言模型构建MoE模型的方法。具体而言，我们以知名的LLaMA-2 7B模型为基础，通过两个步骤得到MoE模型：（1）专家构建：将原始前馈网络（Feed-Forward Networks, FFNs）的参数划分为多个专家；（2）持续预训练：对转换后的MoE模型及新增的门控网络进行进一步训练。本文全面探究了不同的专家构建方法，以及适用于持续预训练的各类数据采样策略。经过这些阶段，我们的LLaMA-MoE模型能够保持语言能力，且仅激活部分参数即可将输入词元路由至特定专家。实验表明，在训练200B词元后，LLaMA-MoE-3.5B模型的性能显著优于激活参数量相近的稠密模型。源代码与模型可在https://github.com/pjlab-sys4nlp/llama-moe 获取。
英文摘要：
Mixture-of-Experts (MoE) has gained increasing popularity as a promising framework for scaling up large language models (LLMs). However, training MoE from scratch in a large-scale setting still suffers from data-hungry and instability problems. Motivated by this limit, we investigate building MoE models from existing dense large language models. Specifically, based on the well-known LLaMA-2 7B model, we obtain an MoE model by: (1) Expert Construction, which partitions the parameters of original Feed-Forward Networks (FFNs) into multiple experts; (2) Continual Pre-training, which further trains the transformed MoE model and additional gate networks. In this paper, we comprehensively explore different methods for expert construction and various data sampling strategies for continual pre-training. After these stages, our LLaMA-MoE models could maintain language abilities and route the input tokens to specific experts with part of the parameters activated. Empirically, by training 200B tokens, LLaMA-MoE-3.5B models significantly outperform dense models that contain similar activation parameters. The source codes and models are available at https://github.com/pjlab-sys4nlp/llama-moe .
Continual Pre-training of MoEs: How robust is your router?
Université de Montréal（蒙特利尔大学）；Mila – Quebec AI Institute（魁北克人工智能研究所）；Concordia University（康科迪亚大学）；University of Chicago（芝加哥大学）；Capital One（Capital One公司）
Efficient Training of Robust Traditional Chinese LLaMA-1B on a Single Consumer GPU: Continual Pre-training, SFT, and DPO
National Tsing Hua University（台湾清华大学）；Accton Technology Corporation（智邦科技公司）；National Yang Ming Chiao Tung University（阳明交通大学）；University of Taipei（台北市立大学）
Enhancing Multimodal Continual Instruction Tuning with BranchLoRA
Mohamed bin Zayed University of Artificial Intelligence（Mohamed bin Zayed大学人工智能学院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Kyoto University（京都大学）；Tencent AI Lab（腾讯AI实验室）；Tomorrow Advancing Life（明天生命科技）
DIVE into MoE: Diversity-Enhanced Reconstruction of Large Language Models from Dense into Mixture-of-Experts
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）
Central Scientific Instruments Organisation, Academy of Scientific and Innovative Research（中央科学仪器组织，科学与创新研究院）；School of Mathematical and Computational Sciences, Massey University（数学与计算科学学院，梅西大学）
MoSE: Mixture of Slimmable Experts for Efficient and Adaptive Language Models
Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), UAE（Mohamed bin Zayed人工智能大学（MBZUAI））；Michigan State University (MSU), USA（密歇根州立大学（MSU））；Amazon Science, UK（亚马逊科学）
DSMoE: Matrix-Partitioned Experts with Dynamic Routing for Computation-Efficient Dense LLMs
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Kuaishou Technology（快手科技）；Tianjin University（天津大学）；Tsinghua University（清华大学）
DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models
DeepSeek-AI（深度求索人工智能）；National Key Laboratory for Multimedia Information Processing, Peking University（北京大学多媒体信息处理全国重点实验室）；Institute for Interdisciplinary Information Sciences, Tsinghua University（清华大学交叉信息研究院）；National Key Laboratory for Novel Software Technology, Nanjing University（南京大学计算机软件新技术全国重点实验室）
LLMs Can Evolve Continually on Modality for X-Modal Reasoning
Dalian University of Technology（大连理工大学）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）；Tsinghua University（清华大学）；The Hong Kong University of Science and Technology（香港科技大学）
Revisiting Replay and Gradient Alignment for Continual Pre-Training of Large Language Models
Université de Montréal（蒙特利尔大学）；Mila – Quebec AI Institute（魁北克人工智能研究院）；Chandar Research Lab（Chandar研究实验室）；IBM Research（IBM研究院）；Fujitsu Research（富士通研究院）；Polytechnique Montréal（蒙特利尔理工学院）
Mechanistic Analysis of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning
Division of Statistics and Machine Learning (STIMA), Department of Computer and Information Science (IDA), Linköping University（统计与机器学习系（STIMA）、计算机与信息科学系（IDA）、利厄普堡大学）
Safety Alignment as Continual Learning: Mitigating the Alignment Tax via Orthogonal Gradient Projection
School of Life Sciences, IDG/McGovern Institute for Brain Research（生命科学学院，IDG/麦戈文脑科学研究所）；Dept. of Comp. Sci. and Tech., Institute for AI, Tsinghua-Bosch Joint ML Center, THBI Lab, BNRist Center（计算机科学与技术系，人工智能研究所，清华大学-博世联合机器学习中心，THBI实验室，BNRist中心）；Tsinghua University, Beijing, China（清华大学，北京，中国）
