arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.17467cs.CLcs.LG
收藏

CMR Scaling Law:预测语言模型持续预训练的关键混合比例

CMR Scaling Law: Predicting Critical Mixture Ratios for Continual Pre-training of Language Models

Sun Yat-sen University(中山大学);SenseTime Research(商汤科技研究院);Nanjing University(南京大学)

Jiawei Gu, Zacc Yang, Chuanghao Ding, Rui Zhao, Fei Tan

AI总结:

本研究针对语言模型持续预训练中数据混合比例凭经验选择导致效率低下的问题,发现损失、混合比例与训练token规模的幂律关系,提出CMR缩放定律及最优关键混合比例,兼顾通用与领域能力,提升资源利用率。

摘要:

大语言模型(LLM)在各类任务中表现出色,但由于领域专属或专有语料有限,在专业领域的表现往往不佳。持续预训练(CPT)通过注入新的领域专属或专有知识,同时回放通用语料以防止灾难性遗忘,来提升LLM的能力。然而,通用语料与领域专属语料的数据混合比例一直是凭经验选择的,导致实际训练效率未达最优。在此背景下,我们尝试重新探究CPT框架下LLM的缩放行为,发现损失、混合比例与训练token规模之间存在幂律关系。我们对通用能力与领域专属能力之间的权衡进行了形式化推导,得出了明确定义的通用数据与领域数据的关键混合比例(Critical Mixture Ratio,CMR)。通过实现平衡,CMR既能维持模型的通用能力,又能达成预期的领域迁移,确保可用资源的最高利用率。兼顾效率与效果,CMR可被视为最优混合比例。通过大量实验,我们证实了CMR的可预测性,提出了CMR缩放定律,并验证了其泛化性。这些发现为优化专业领域的LLM训练提供了实用指导,在高效管理训练资源的同时,兼顾通用性能与领域专属性能。

英文摘要:

Large Language Models (LLMs) excel in diverse tasks but often underperform in specialized fields due to limited domain-specific or proprietary corpus. Continual pre-training (CPT) enhances LLM capabilities by imbuing new domain-specific or proprietary knowledge while replaying general corpus to prevent catastrophic forgetting. The data mixture ratio of general corpus and domain-specific corpus, however, has been chosen heuristically, leading to sub-optimal training efficiency in practice. In this context, we attempt to re-visit the scaling behavior of LLMs under the hood of CPT, and discover a power-law relationship between loss, mixture ratio, and training tokens scale. We formalize the trade-off between general and domain-specific capabilities, leading to a well-defined Critical Mixture Ratio (CMR) of general and domain data. By striking the balance, CMR maintains the model's general ability and achieves the desired domain transfer, ensuring the highest utilization of available resources. Considering the balance between efficiency and effectiveness, CMR can be regarded as the optimal mixture ratio. Through extensive experiments, we ascertain the predictability of CMR, propose CMR scaling law and have substantiated its generalization. These findings offer practical guidelines for optimizing LLM training in specialized domains, ensuring both general and domain-specific performance while efficiently managing training resources.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑