arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.08022cs.LG
收藏

面向容量的混合法则实现高效大语言模型数据优化

Capacity-Aware Mixture Law Enables Efficient LLM Data Optimization

Institute for Interdisciplinary Information Sciences, Tsinghua University(交叉信息研究院,清华大学);Shanghai Qizhi Institute(上海启智研究院)

Jingwei Li, Xinran Gu, Jingzhao Zhang

AI总结:

面向容量的混合法则通过高效数据混合优化,减少50%的优化成本并提升3%的下游性能

摘要:

数据混合指的是如何将不同的数据源结合起来训练大型语言模型,选择有效的混合物对于获得最佳下游性能至关重要。现有方法要么在目标模型上进行成本高昂的搜索,要么依赖于混合扩展定律,这些定律在大模型规模下无法很好地外推。我们通过引入一个计算高效的管道来解决这些限制。首先,我们提出了CAMEL,一种面向容量的混合法则,该法则通过模型大小和混合物之间的非线性相互作用来建模验证损失。我们还引入了一种损失到基准预测法则,该法则从验证损失估计基准准确率,从而能够为目标模型提供端到端的性能预测。接下来,我们研究如何将固定的计算预算分配到模型规模上,以适应该法则并减少预测误差。最后,我们将我们的方法应用于具有高达7B-A150M参数的混合专家模型以适应该法则,并通过外推到55B-A1.2B目标模型来验证由该法则得出的最优混合物。与先前方法相比,我们减少了混合优化成本50%,并提高了下游基准性能高达3%。

英文摘要:

A data mixture refers to how different data sources are combined to train large language models, and selecting an effective mixture is crucial for optimal downstream performance. Existing methods either conduct costly searches directly on the target model or rely on mixture scaling laws that fail to extrapolate well to large model sizes. We address these limitations by introducing a compute-efficient pipeline for data mixture scaling. First, we propose CAMEL, a capacity-aware mixture law that models validation loss with the nonlinear interplay between model size and mixture. We also introduce a loss-to-benchmark prediction law that estimates benchmark accuracy from validation loss, enabling end-to-end performance prediction for the target model. Next, we study how to allocate a fixed compute budget across model scales to fit the law and reduce prediction error. Finally, we apply our method to Mixture-of-Experts models with up to 7B-A150M parameters to fit the law, and verify the optimal mixture derived from the law by extrapolating to a 55B-A1.2B target model. Compared to prior methods, we reduces mixture optimization costs by 50\% and improves downstream benchmark performance by up to 3\%.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑