arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.01375cs.CL
收藏

D-CPT Law:面向大语言模型的领域特定持续预训练缩放定律

D-CPT Law: Domain-specific Continual Pre-Training Scaling Law for Large Language Models

Alibaba Group(阿里巴巴集团);University of Waterloo(滑铁卢大学);University of Manchester(曼彻斯特大学);QMUL(伦敦玛丽女王大学);The Hong Kong University of Science and Technology(香港科技大学)
Haoran Que, Jiaheng Liu, Ge Zhang, Chenchen Zhang, Xingwei Qu, Yinghao Ma, Feiyu Duan, Zhiqi Bai, Jiakai Wang, Yuanxing Zhang, Xu Tan, Jie Fu, Wenbo Su, Jiamang… 展开作者

Haoran Que, Jiaheng Liu, Ge Zhang, Chenchen Zhang, Xingwei Qu, Yinghao Ma, Feiyu Duan, Zhiqi Bai, Jiakai Wang, Yuanxing Zhang, Xu Tan, Jie Fu, Wenbo Su, Jiamang Wang, Lin Qu, Bo Zheng

AI总结:

针对领域特定大模型持续预训练中混合比例选择成本高且非最优的问题,提出D-CPT Law及Cross-Domain D-CPT Law,利用小规模训练成本准确预测不同混合比例、模型与数据集规模下的性能,并在六个领域验证了其有效性与泛化能力。

摘要:

大型语言模型(LLMs)的持续预训练(CPT)已被广泛用于扩展模型对特定下游领域(如数学和代码)的基础理解能力。对于领域特定LLMs的CPT,一个重要问题是如何选择通用语料库(例如Dolma、Slim-pajama)与下游领域语料库之间的最佳混合比例。现有方法通常通过在一组混合比例上进行网格搜索来耗费大量人力,这需要高昂的GPU训练消耗成本。此外,我们无法保证所选比例对该特定领域是最优的。为了解决现有方法的局限性,受用于性能预测的Scaling Law启发,我们提出研究领域特定持续预训练的缩放定律(D-CPT Law),以在可接受的训练成本下,为不同规模的LLMs决定最佳混合比例。具体而言,通过拟合D-CPT Law,我们可以利用在有限实验上的小规模训练成本,轻松预测任意混合比例、模型规模和数据集规模下的通用性能与下游性能。此外,我们还将标准的D-CPT Law扩展到跨领域设置中,并提出了Cross-Domain D-CPT Law,用于预测目标领域的D-CPT law,其中目标领域仅需极小的训练成本(约为正常训练成本的1%)。在六个下游领域的综合实验结果证明了我们提出的D-CPT Law和Cross-Domain D-CPT Law的有效性和泛化能力。

英文摘要:

Continual Pre-Training (CPT) on Large Language Models (LLMs) has been widely used to expand the model's fundamental understanding of specific downstream domains (e.g., math and code). For the CPT on domain-specific LLMs, one important question is how to choose the optimal mixture ratio between the general-corpus (e.g., Dolma, Slim-pajama) and the downstream domain-corpus. Existing methods usually adopt laborious human efforts by grid-searching on a set of mixture ratios, which require high GPU training consumption costs. Besides, we cannot guarantee the selected ratio is optimal for the specific domain. To address the limitations of existing methods, inspired by the Scaling Law for performance prediction, we propose to investigate the Scaling Law of the Domain-specific Continual Pre-Training (D-CPT Law) to decide the optimal mixture ratio with acceptable training costs for LLMs of different sizes. Specifically, by fitting the D-CPT Law, we can easily predict the general and downstream performance of arbitrary mixture ratios, model sizes, and dataset sizes using small-scale training costs on limited experiments. Moreover, we also extend our standard D-CPT Law on cross-domain settings and propose the Cross-Domain D-CPT Law to predict the D-CPT law of target domains, where very small training costs (about 1% of the normal training costs) are needed for the target domains. Comprehensive experimental results on six downstream domains demonstrate the effectiveness and generalizability of our proposed D-CPT Law and Cross-Domain D-CPT Law.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑