Large Language Models (LLM) often need to be Continual Pre-Trained (CPT) to obtain unfamiliar language skills or adapt to new domains. The huge training cost of CPT often asks for cautious choice of key hyper-parameters such as the mixture ratio of extra language or domain corpus. However, there is no systematic study that bridges the gap between the optimal mixture ratio and the actual model performance, and the gap between experimental scaling law and the actual deployment in the full model size. In this paper, we perform CPT on Llama-3 8B and 70B to enhance its Chinese ability. We study the optimal correlation between the Additional Language Mixture Ratio (ALMR) and the Learning Rate (LR) on the 8B size which directly indicates the optimal experimental setup. By thorough choice of hyper-parameter, and subsequent fine-tuning, the model capability is improved not only on the Chinese-related benchmark but also in some specific domains including math, coding, and emotional intelligence. We deploy the final 70B version of LLM on a real-life chat system which obtains satisfying performance.
LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training
Soochow University(苏州大学);Shanghai AI Laboratory(上海人工智能实验室);Shanghai Jiao Tong University(上海交通大学);Fudan University(复旦大学);The Chinese University of Hong Kong(香港中文大学)
Evolution of meta's llama models and parameter-efficient fine-tuning of large language models: a survey
University of Kurdistan(库尔德斯坦大学);Queen Mary University(女王玛丽大学);Torrens University Australia(澳大利亚托伦斯大学);University of Technology Sydney(悉尼技术大学);University of Kurdistan Sanandaj, Iran(伊朗桑和杰德大学);TU Dortmund University(多特蒙德技术大学);University of Kurdistan Hewler(库尔德斯坦大学海勒);Tehran University(德黑兰大学)
ToMoE: Converting Dense Large Language Models to Mixture-of-Experts through Dynamic Structural Pruning
Department of Computer Science, Florida State University(佛罗里达州立大学计算机科学系);Department of Computer Science and Engineering, University of Notre Dame(圣母大学计算机科学与工程系);Department of Computer Science, University of Maryland, College Park(马里兰大学学院市计算机科学系);Samsung Research America(三星美国研究机构);Department of Computer Science, University of Central Florida(中央佛罗里达大学计算机科学系);Department of Health Outcomes and Biomedical Informatics, University of Florida(佛罗里达大学健康结果与生物医学信息学系)
Towards Adapting Open-Source Large Language Models for Expert-Level Clinical Note Generation
Mayo Clinic Health System(梅奥诊所健康系统);School of Computing and Data Science, UIUC(UIUC计算与数据科学学院);Mayo Clinic Rochester(梅奥诊所罗切斯特分部);Carle Illinois College of Medicine, UIUC(UIUC卡勒伊利诺伊医学院)
TREX: Tokenizer Regression for Optimal Data Mixture
KAIST CT(韩国科学技术院CT);Seoul National University of Science and Technology(首尔科学技术大学);Upstage AI;KAIST InnoCORE PRISM-AI Center(韩国科学技术院InnoCORE PRISM-AI中心)
Crystal: Illuminating LLM Abilities on Language and Code
Carnegie Mellon University(卡内基梅隆大学);Cerebras Systems(赛布拉斯系统公司);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校)
WDMoE: Wireless Distributed Large Language Models with Mixture of Experts
Cooperative Medianet Innovation Center, Shanghai Jiao Tong University(上海交通大学合作媒介网络创新中心);Pengcheng Laboratory(鹏城实验室);Future Network of Intelligent Institute (FNii), the Chinese University of Hong Kong (Shenzhen)(香港中文大学(深圳)未来网络智能研究所);Beijing University of Posts and Telecommunications(北京邮电大学)
Enhancing Large Language Models for Clinical Decision Support by Incorporating Clinical Practice Guidelines
University of Pittsburgh(匹兹堡大学);University of Pittsburgh Medical Center(匹兹堡大学医学中心);University of Pittsburgh Medical Center Western Maryland(匹兹堡大学医学中心西马里兰分院)
DIVE into MoE: Diversity-Enhanced Reconstruction of Large Language Models from Dense into Mixture-of-Experts
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院)
Mixture of In-Context Experts Enhance LLMs' Long Context Awareness
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);XiaoMi AI Lab(小米AI实验室);Career Science Lab, BOSS Zhipin(BOSS直聘职业科学实验室);NLP Center, BOSS Zhipin(BOSS直聘自然语言处理中心)
Large language models replicate and predict human cooperation across experiments in game theory
Department for Computational Social Sciences and Humanities, Barcelona Supercomputing Center(巴塞罗那超级计算中心计算社会科学与人文学系);Grupo de Sistemas Complejos, Universidad Politécnica de Madrid(马德里理工大学复杂系统研究组);School of Law and Political Science, Universitat Oberta de Catalunya(加泰罗尼亚开放大学法律与政治科学学院)
Wangxuan Institute of Computer Technology, Peking University(北京大学王选计算机研究所);Center for Data Science, Peking University(北京大学数据科学中心);The MOE Key Laboratory of Computational Linguistics, Peking University(北京大学计算语言学教育部重点实验室)