RegMix:将数据混合视为语言模型预训练的回归问题
RegMix: Data Mixture as Regression for Language Model Pre-training
AI总结:
RegMix将数据混合视为回归任务,通过训练小模型预测最佳混合,用于大规模预训练,在多种规模下优于人工选择,且效率高于DoReMi。
摘要:
大型语言模型预训练的数据混合对性能有显著影响,但如何确定有效的混合方式仍不清楚。我们提出RegMix,通过将数据混合问题形式化为回归任务,自动识别高性能的数据混合方案。RegMix在多样化的数据混合上训练许多小模型,使用回归预测未见混合的性能,并应用预测的最佳混合来训练计算量高出数个数量级的大规模模型。为了实证验证RegMix,我们训练了512个具有1M参数、1B token的模型来拟合回归模型并预测最佳数据混合。使用该混合,我们训练了一个1B参数、25B token的模型(即规模大1000倍、训练时长长25倍),发现它在64个候选1B参数模型(使用其他混合)中表现最佳。此外,RegMix在涉及高达7B模型、100B token训练的实验中始终优于人工选择,同时仅使用10%的计算资源即可匹配或超越DoReMi。我们的实验还表明:(1)数据混合显著影响性能;(2)网络语料库(而非像Wikipedia这样被视为高质量的数据)与下游性能的正相关性最强;(3)领域之间的交互复杂,常常违背常识,因此需要像RegMix这样的自动方法;(4)数据混合效应超越了缩放定律。我们的代码可在https://github.com/sail-sg/regmix获取。
英文摘要:
The data mixture for large language model pre-training significantly impacts performance, yet how to determine an effective mixture remains unclear. We propose RegMix to automatically identify a high-performing data mixture by formulating it as a regression task. RegMix trains many small models on diverse data mixtures, uses regression to predict performance of unseen mixtures, and applies the best predicted mixture to train a large-scale model with orders of magnitude more compute. To empirically validate RegMix, we train 512 models with 1M parameters for 1B tokens to fit the regression model and predict the best data mixture. Using this mixture we train a 1B parameter model for 25B tokens (i.e. 1000x larger and 25x longer) which we find performs best among 64 candidate 1B parameter models with other mixtures. Furthermore, RegMix consistently outperforms human selection in experiments involving models up to 7B models trained on 100B tokens, while matching or exceeding DoReMi using just 10% of the computational resources. Our experiments also show that (1) Data mixtures significantly impact performance; (2) Web corpora rather than data perceived as high-quality like Wikipedia have the strongest positive correlation with downstream performance; (3) Domains interact in complex ways often contradicting common sense, thus automatic approaches like RegMix are needed; (4) Data mixture effects transcend scaling laws. Our code is available at https://github.com/sail-sg/regmix.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。