arXiv 2508.176772025-08-25cs.LG

TiKMiX：将数据影响纳入语言模型预训练的动态混合

TiKMiX: Take Data Influence into Dynamic Mixture for Language Model Pre-training

Yifan Wang, Binbin Liu, Fengze Liu, Yuanfan Guo, Jiyao Deng, Xuecheng Wu, Weidong Zhou, Xiaohuan Zhou, Taifeng Wang

AI总结：

针对语言模型预训练中静态数据混合策略的不足，提出TiKMiX方法，通过组影响指标动态调整数据混合，以直接优化或回归预测方式提升性能，在低计算资源下超越现有方法，下游基准平均性能提升2%。

摘要：

语言模型预训练中使用的数据混合是其最终性能的基石。然而，静态混合策略并非最优，因为模型对不同数据领域的学习偏好会在整个训练过程中动态变化。关键在于，以计算高效的方式观察这些不断演变的偏好仍然是一项重大挑战。为解决这一问题，我们提出TiKMiX，一种根据模型不断演变的偏好动态调整数据混合的方法。TiKMiX引入组影响（Group Influence），这是一种评估数据领域对模型影响的高效指标。该指标使数据混合问题能够被表述为寻找最优的、影响最大化的分布。我们通过两种方法解决这一问题：用于直接优化的TiKMiX-D，以及使用回归模型预测更优混合的TiKMiX-M。我们在多达1万亿个token上训练了不同参数数量的模型。TiKMiX-D在仅使用20%计算资源的情况下超过了REGMIX等最先进方法的性能。TiKMiX-M在9个下游基准上带来了2%的平均性能提升。我们的实验表明，模型的数据偏好会随着训练进度和规模而演变，并且我们证明，基于组影响（这些偏好的直接衡量标准）动态调整数据混合，通过缓解静态比例下的数据未充分消化问题，显著提高了性能。

英文摘要：

The data mixture used in the pre-training of a language model is a cornerstone of its final performance. However, a static mixing strategy is suboptimal, as the model's learning preferences for various data domains shift dynamically throughout training. Crucially, observing these evolving preferences in a computationally efficient manner remains a significant challenge. To address this, we propose TiKMiX, a method that dynamically adjusts the data mixture according to the model's evolving preferences. TiKMiX introduces Group Influence, an efficient metric for evaluating the impact of data domains on the model. This metric enables the formulation of the data mixing problem as a search for an optimal, influence-maximizing distribution. We solve this via two approaches: TiKMiX-D for direct optimization, and TiKMiX-M, which uses a regression model to predict a superior mixture. We trained models with different numbers of parameters, on up to 1 trillion tokens. TiKMiX-D exceeds the performance of state-of-the-art methods like REGMIX while using just 20% of the computational resources. TiKMiX-M leads to an average performance gain of 2% across 9 downstream benchmarks. Our experiments reveal that a model's data preferences evolve with training progress and scale, and we demonstrate that dynamically adjusting the data mixture based on Group Influence, a direct measure of these preferences, significantly improves performance by mitigating the underdigestion of data seen with static ratios.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。