D-CPT Law: Domain-specific Continual Pre-Training Scaling Law for Large Language Models
Alibaba Group(阿里巴巴集团);University of Waterloo(滑铁卢大学);University of Manchester(曼彻斯特大学);QMUL(伦敦玛丽女王大学);The Hong Kong University of Science and Technology(香港科技大学)
Haoran Que, Jiaheng Liu, Ge Zhang, Chenchen Zhang, Xingwei Qu, Yinghao Ma, Feiyu Duan, Zhiqi Bai, Jiakai Wang, Yuanxing Zhang, Xu Tan, Jie Fu, Wenbo Su, Jiamang… 展开作者
Haoran Que, Jiaheng Liu, Ge Zhang, Chenchen Zhang, Xingwei Qu, Yinghao Ma, Feiyu Duan, Zhiqi Bai, Jiakai Wang, Yuanxing Zhang, Xu Tan, Jie Fu, Wenbo Su, Jiamang Wang, Lin Qu, Bo Zheng
Continual Pre-Training (CPT) on Large Language Models (LLMs) has been widely used to expand the model's fundamental understanding of specific downstream domains (e.g., math and code). For the CPT on domain-specific LLMs, one important question is how to choose the optimal mixture ratio between the general-corpus (e.g., Dolma, Slim-pajama) and the downstream domain-corpus. Existing methods usually adopt laborious human efforts by grid-searching on a set of mixture ratios, which require high GPU training consumption costs. Besides, we cannot guarantee the selected ratio is optimal for the specific domain. To address the limitations of existing methods, inspired by the Scaling Law for performance prediction, we propose to investigate the Scaling Law of the Domain-specific Continual Pre-Training (D-CPT Law) to decide the optimal mixture ratio with acceptable training costs for LLMs of different sizes. Specifically, by fitting the D-CPT Law, we can easily predict the general and downstream performance of arbitrary mixture ratios, model sizes, and dataset sizes using small-scale training costs on limited experiments. Moreover, we also extend our standard D-CPT Law on cross-domain settings and propose the Cross-Domain D-CPT Law to predict the D-CPT law of target domains, where very small training costs (about 1% of the normal training costs) are needed for the target domains. Comprehensive experimental results on six downstream domains demonstrate the effectiveness and generalizability of our proposed D-CPT Law and Cross-Domain D-CPT Law.
Learning Dynamics in Continual Pre-Training for Large Language Models
School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China(人工智能学院,中国科学院大学,北京);State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China(多模态人工智能系统国家重点实验室,自动化研究所,中国科学院,北京)
Time-Incremental Continued Pretraining of LLMs: Knowledge Updates Without Catastrophic Forgetting
Concordia University(康考迪亚大学);Mila – Quebec AI Institute(米拉-魁北克人工智能研究所);Université de Montréal(蒙特利尔大学);Laval University(拉瓦尔大学);University of Tübingen(蒂宾根大学);Tübingen AI Center(蒂宾根人工智能中心)
A Learning Rate Path Switching Training Paradigm for Version Updates of Large Language Models
School of Informatics, Xiamen University(厦门大学信息学院);vivo AI Lab(vivo人工智能实验室);Institute of Artificial Intelligence, Xiamen University(厦门大学人工智能研究院);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)
Continual Pre-training of MoEs: How robust is your router?
Université de Montréal(蒙特利尔大学);Mila – Quebec AI Institute(魁北克人工智能研究所);Concordia University(康科迪亚大学);University of Chicago(芝加哥大学);Capital One(Capital One公司)
Efficient Training of Robust Traditional Chinese LLaMA-1B on a Single Consumer GPU: Continual Pre-training, SFT, and DPO
National Tsing Hua University(台湾清华大学);Accton Technology Corporation(智邦科技公司);National Yang Ming Chiao Tung University(阳明交通大学);University of Taipei(台北市立大学)
Breaking Language Barriers: Cross-Lingual Continual Pre-Training at Scale
Chinese Academy of Sciences(中国科学院);Harbin Institute of Technology(哈尔滨工业大学);Peking University(北京大学);Central South University(中南大学);Langboat Inc.(舟博科技公司)
TELLME: Test-Enhanced Learning for Language Model Enrichment
Korea Advanced Institute of Science and Technology(韩国科学技术院);Seoul National University of Science and Technology(首尔科学技术大学);National Security Research Institute(国家安全研究院)
Mix-CPT: A Domain Adaptation Framework via Decoupling Knowledge Learning and Format Alignment
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);BOSS Zhipin(BOSS直聘);School of Information, Renmin University of China(中国人民大学信息学院)
Trade-offs in Medical LLM Adaptation: An Empirical Study in French QA
Aix-Marseille Univ., CNRS, LIS UMR 7020(艾克斯-马赛大学,法国国家科学研究中心,LIS UMR 7020);Nantes Univ., École Centrale Nantes, CNRS, LS2N UMR 6004(南特大学,南特中央理工大学,法国国家科学研究中心,LS2N UMR 6004);Grenoble Alpes Univ., CNRS, INRIA, Grenoble INP, LIG UMR 5217(格勒诺布尔阿尔卑斯大学,法国国家科学研究中心,INRIA,格勒诺布尔INP,LIG UMR 5217)
Towards Understanding Continual Factual Knowledge Acquisition of Language Models: From Theory to Algorithm
Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China(中国人民大学北京校区人工智能学院);School of Artificial Intelligence(人工智能学院);Data Science, University of International Business(国际商务大学数据科学)
Continual Mixed-Language Pre-Training for Extremely Low-Resource Neural Machine Translation
Center for Artificial Intelligence Research (CAiRE)(人工智能研究中心);Department of Electronic and Computer Engineering(电子与计算机工程系);The Hong Kong University of Science and Technology(香港科技大学)