arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.16952cs.CLcs.AIcs.LG
收藏

数据混合定律:通过预测语言建模性能优化数据混合比例

Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance

Fudan University(复旦大学);Shanghai AI Laboratory(上海人工智能实验室)

Jiasheng Ye, Peiju Liu, Tianxiang Sun, Jun Zhan, Yunhua Zhou, Xipeng Qiu

AI总结:

该研究提出数据混合定律,可通过拟合样本混合数据的函数预测不同数据混合比例下的语言模型性能,结合缩放定律仅用小规模训练就能预测大模型表现,还能优化训练数据混合、避免持续训练中的灾难性遗忘。

摘要:

大语言模型的预训练数据由多个领域(如网页文本、学术论文、代码)构成,其混合比例对最终模型的性能至关重要。现有研究依赖启发式方法或定性策略调整比例,而我们发现模型性能随混合比例变化的定量可预测性呈函数形式,我们将其称为数据混合定律。在样本混合数据上拟合这类函数,可在实际训练前预测模型在未见过的混合比例下的性能,从而指导理想数据混合方案的选择。此外,我们提出将训练步数、模型规模的缩放定律与我们的数据混合定律嵌套使用,仅需小规模训练就能预测在海量数据上、不同混合比例下训练的大模型性能。实验结果验证,我们的方法有效优化了在RedPajama数据集上训练100B tokens的1B参数模型的训练数据混合比例,其性能与在默认混合比例下多训练48%步数的模型相当。将数据混合定律扩展应用到持续训练中,可准确预测避免灾难性遗忘的关键混合比例,并展现了动态数据调度的潜力。

英文摘要:

Pretraining data of large language models composes multiple domains (e.g., web texts, academic papers, codes), whose mixture proportions crucially impact the competence of outcome models. While existing endeavors rely on heuristics or qualitative strategies to tune the proportions, we discover the quantitative predictability of model performance regarding the mixture proportions in function forms, which we refer to as the data mixing laws. Fitting such functions on sample mixtures unveils model performance on unseen mixtures before actual runs, thus guiding the selection of an ideal data mixture. Furthermore, we propose nested use of the scaling laws of training steps, model sizes, and our data mixing law to enable predicting the performance of large models trained on massive data under various mixtures with only small-scale training. Moreover, experimental results verify that our method effectively optimizes the training mixture of a 1B model trained for 100B tokens in RedPajama, reaching a performance comparable to the one trained for 48% more steps on the default mixture. Extending the application of data mixing laws to continual training accurately predicts the critical mixture proportion that avoids catastrophic forgetting and outlooks the potential for dynamic data schedules

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑