arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2502.16802cs.CLcs.AI
收藏

主题优于来源:语言模型预训练中有效数据混合的关键

Topic Over Source: The Key to Effective Data Mixing for Language Models Pre-training

Jiahui Peng, Xinlin Zhuang, Jiantao Qiu, Ren Ma, Jing Yu, He Zhu, Conghui He

AI总结:

提出基于主题的数据混合策略,通过多阶段过程生成主题标签,证明其优于基于来源的方法,显著降低验证损失并提升模型性能。

摘要:

大型语言模型(LLMs)的性能受其预训练数据的质量和组成的显著影响,这些数据本质上是多样化的,涵盖各种语言、来源和主题。有效整合这些异构数据组对于优化LLM性能至关重要。先前的研究主要集中在基于来源的数据混合上,往往忽略了数据在细粒度主题层面的特征。为了填补这一空白,我们提出了一种基于主题的数据混合策略,该策略利用通过多阶段过程生成的详细主题标签,该过程结合了无监督聚类、基于LLM的摘要生成和监督分类器训练。借助这一策略,我们首次在多种混合策略中对基于主题与基于来源的划分进行了全面比较。我们证明,在包括RegMix、DoReMi、基于温度的采样以及基于下游任务性能的人工混合方法在内的多种方法中,使用按主题混合的数据预训练的语言模型始终优于使用按来源混合的数据训练的模型。我们的理论分析表明,与基于来源的方法相比,基于主题的数据实现了显著更低的验证损失,为模型训练创造了更好的优化景观。我们将公开我们的代码、标注数据集和主题分类模型,以促进进一步的研究。

英文摘要:

The performance of large language models (LLMs) is significantly affected by the quality and composition of their pre-training data, which is inherently diverse, spanning various languages, sources, and topics. Effectively integrating these heterogeneous data groups is crucial for optimizing LLM performance. Previous research has predominantly concentrated on source-based data mixing, often neglecting the nuanced topic-level characteristics of the data. To address this gap, we propose a topic-based data mixing strategy that utilizes detailed topic labels generated through a multi-stage process combining unsupervised clustering, LLM-based summarization, and supervised classifier training. With this strategy, we conduct the first comprehensive comparison of topic-based versus source-based partitioning across multiple mixing strategies. We demonstrate that language models pretrained on data mixed by topics consistently outperform those trained on data mixed by sources across multiple methods including RegMix, DoReMi,temperature-based sampling, and a manual mixing method based on downstream task performance. Our theoretical analysis reveals that topic-based data achieves significantly lower validation loss compared to source-based approaches, creating a better optimization landscape for model training. We will make our code, annotated datasets, and topic classification models publicly available to facilitate further research.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑