将搜索与训练解耦：通过模型合并实现大规模语言模型预训练的数据混合缩放
Decouple Searching from Training: Scaling Data Mixing via Model Merging for Large Language Model Pre-training
AI总结：
提出DeMix框架，通过模型合并预测最优数据配比，在降低搜索成本的同时提升基准性能。
摘要：
确定有效的数据混合是大语言模型（LLM）预训练的关键因素，模型必须在通用能力与数学、代码等困难任务的专业性之间取得平衡。然而，识别最优混合仍然是一个开放挑战，现有方法要么依赖不可靠的小规模代理实验，要么需要代价高昂的大规模探索。为此，我们提出“将搜索与训练解耦混合”（DeMix），一种利用模型合并预测最优数据配比的新框架。DeMix不是为每个采样的混合训练代理模型，而是按规模在候选数据集上训练组件模型，并通过加权模型合并推导数据混合代理。这种范式将搜索与训练成本解耦，使得无需额外训练负担即可评估无限采样的混合，从而通过更多搜索试验促进更好的混合发现。大量实验表明，DeMix打破了充分性、准确性和效率之间的权衡，以更低的搜索成本获得更高基准性能的最优混合。此外，我们发布了DeMix语料库，一个包含高质量预训练数据和已验证混合的综合22T令牌数据集，以促进开放研究。我们的代码和DeMix语料库可在https://github.com/Lucius-lsr/DeMix获取。
英文摘要：
Determining an effective data mixture is a key factor in Large Language Model (LLM) pre-training, where models must balance general competence with proficiency on hard tasks such as math and code. However, identifying an optimal mixture remains an open challenge, as existing approaches either rely on unreliable tiny-scale proxy experiments or require prohibitively expensive large-scale exploration. To address this, we propose Decouple Searching from Training Mix (DeMix), a novel framework that leverages model merging to predict optimal data ratios. Instead of training proxy models for every sampled mixture, DeMix trains component models on candidate datasets at scale and derives data mixture proxies via weighted model merging. This paradigm decouples search from training costs, enabling evaluation of unlimited sampled mixtures without extra training burden and thus facilitating better mixture discovery through more search trials. Extensive experiments demonstrate that DeMix breaks the trade-off between sufficiency, accuracy and efficiency, obtaining the optimal mixture with higher benchmark performance at lower search cost. Additionally, we release the DeMix Corpora, a comprehensive 22T-token dataset comprising high-quality pre-training data with validated mixtures to facilitate open research. Our code and DeMix Corpora is available at https://github.com/Lucius-lsr/DeMix.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
