Large language models (LLMs) have demonstrated considerable proficiency in general natural language processing (NLP) tasks. Instruction tuning, a successful paradigm, enhances the ability of LLMs to follow natural language instructions and exhibit robust generalization across general tasks. However, these models often encounter performance limitations across multiple tasks due to constrained model capacity. Expanding this capacity during the instruction tuning phase poses significant challenges. To address this issue, we introduce parameter-efficient sparsity crafting (PESC), which crafts dense models into sparse models using the mixture-of-experts (MoE) architecture. PESC integrates adapters into the MoE layers of sparse models, differentiating experts without altering the individual weights within these layers. This method significantly reduces computational costs and GPU memory requirements, facilitating model capacity expansion through a minimal parameter increase when guaranteeing the quality of approximation in function space compared to original sparse upcycling. Our empirical evaluation demonstrates the effectiveness of the PESC method. Using PESC during instruction tuning, our best sparse model outperforms other sparse and dense models and exhibits superior general capabilities compared to GPT-3.5. Our code is available at https://github.com/wuhy68/Parameter-Efficient-MoE.
What are the Essential Factors in Crafting Effective Long Context Multi-Hop Instruction Datasets? Insights and Best Practices
Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Research Center for Social Computing and Interactive Robotics(社会计算与交互机器人研究中心);Harbin Institute of Technology(哈尔滨工业大学);School of Computer Science and Engineering, Central South University(中南大学计算机科学与工程学院)
FLEx: Personalized Federated Learning for Mixture-of-Experts LLMs via Expert Grafting
ShanghaiTech University(上海科技大学);China Mobile Communications Company Limited Research Institute(中国移动通信公司有限研究所);Shanghai University(上海大学);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳))
Generative Data Imputation for Sparse Learner Performance Data Using Generative Adversarial Imputation Networks
Institute for Intelligent Systems, University of Memphis(孟菲斯大学智能系统研究所);Faculty of Education, The University of Hong Kong(香港大学教育学院);Human-Computer Interaction Institute, Carnegie Mellon University(卡内基梅隆大学人机交互研究所);Centre for Learning Analytics, Faculty of Information Technology, Monash University(莫纳什大学信息技术学院学习分析中心);Educational Testing Service(教育考试服务中心);Department of Psychology and Counseling, Arkansas State University(阿肯色州立大学心理学与咨询系);Department of Applied Social Sciences, Hong Kong Polytechnic University(香港理工大学应用社会科学系)
Sparse is Enough in Fine-tuning Pre-trained Large Language Models
National Engineering Research Center for Multimedia Software(国家多媒体软件工程技术研究中心);School of Computer Science, Wuhan University(武汉大学计算机学院);Hubei Luojia Laboratory(湖北珞珈实验室);Department of Computer Science and Engineering, Shanghai Jiao Tong University(上海交通大学计算机科学与工程系)
RS-MoE: A Vision-Language Model with Mixture of Experts for Remote Sensing Image Captioning and Visual Question Answering
China Academy of Electronics and Information Technology(中国电子科技集团公司电子科学研究院);Aerospace Information Research Institute, Chinese Academy of Sciences(中国科学院空天信息创新研究院);University of Chinese Academy of Sciences(中国科学院大学);Harbin Institute of Technology(哈尔滨工业大学);New York University Abu Dhabi(纽约大学阿布扎比分校)