arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2405.14908cs.LGcs.AIcs.CL
收藏

BiMix:一种用于语言模型预训练的二元数据混合定律

BiMix: A Bivariate Data Mixing Law for Language Model Pretraining

Alibaba Group(阿里巴巴集团)

Ce Ge, Zhijian Ma, Daoyuan Chen, Yaliang Li, Bolin Ding

AI总结:

本文提出 BiMix,一种用于语言模型预训练的二元数据混合定律,通过建模领域比例与数据量的联合缩放行为,实现高精度的损失外推与未见混合物泛化,并确立基于熵的轻量代理优化数据混合,提升 LLM 训练效率。

摘要:

大型语言模型在各种任务中展现出了卓越的能力,这主要归因于多样化来源数据的利用。然而,预训练数据组成对模型性能的影响仍缺乏深入理解。本文提出了 $\textbf{BiMix}$,一种新颖的二元数据混合定律,用于建模 LLM 预训练中领域比例和数据量的联合缩放行为。$\textbf{BiMix}$ 提供了一个系统框架,用于理解和优化跨不同领域的数据混合。通过在两个大规模数据集上进行广泛实验,我们证明了 $\textbf{BiMix}$ 在损失外推方面的高准确性(平均相对误差 < 0.2%)及其对未见混合物的泛化能力(R${}^{2}$ > 0.97)。领域比例的优化使得模型性能优于现有方法。此外,我们确立了基于熵的度量作为数据混合的有效代理,提供了一种计算轻量的策略。我们的工作为数据混合动态提供了理论见解,并为提升 LLM 训练效率提供了实用工具,为语言模型开发中更有效的缩放策略铺平了道路。

英文摘要:

Large language models have demonstrated remarkable capabilities across various tasks, primarily attributed to the utilization of diversely sourced data. However, the impact of pretraining data composition on model performance remains poorly understood. This paper introduces $\textbf{BiMix}$, a novel bivariate data mixing law that models the joint scaling behavior of domain proportions and data volume in LLM pretraining. $\textbf{BiMix}$ provides a systematic framework for understanding and optimizing data mixtures across diverse domains. Through extensive experiments on two large-scale datasets, we demonstrate $\textbf{BiMix}$'s high accuracy in loss extrapolation (mean relative error < 0.2%) and its generalization to unseen mixtures (R${}^{2}$ > 0.97). Optimization of domain proportions yields superior model performance compared to existing methods. Furthermore, we establish entropy-based measures as efficient proxies for data mixing, offering a computationally lightweight strategy. Our work contributes both theoretical insights into data mixing dynamics and practical tools for enhancing LLM training efficiency, paving the way for more effective scaling strategies in language model development.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑