OPUS:在每次迭代中实现大语言模型预训练中的高效且原则性的数据选择
OPUS: Towards Efficient and Principled Data Selection in Large Language Model Pre-training in Every Iteration
AI总结:
OPUS通过在优化器诱导的更新空间中动态选择数据,提升大语言模型预训练的效率和效果,尤其在数据稀缺和低质量情况下表现优异。
摘要:
随着高质量公开文本接近枯竭,即所谓的数据墙现象,预训练正从更多token转向更优质的token。然而,现有方法要么依赖于启发式静态过滤器,忽略了训练动态,要么使用动态但优化器无关的准则,基于原始梯度。我们提出了OPUS(由优化器诱导的投影效用选择),一种动态数据选择框架,其在优化器诱导的更新空间中定义效用。OPUS通过将候选者的有效更新(由现代优化器塑造)投影到从稳定、分布内代理导出的目标方向上对候选者进行评分。为了确保可扩展性,我们采用Ghost技术结合CountSketch以提高计算效率,并采用Boltzmann采样以实现数据多样性,仅带来4.7%的额外计算开销。OPUS在多样化的语料库、质量层级、优化器和模型规模上均取得显著成果。在使用300亿个token预训练GPT-2 Large/XL于FineWeb和FineWeb-Edu时,OPUS优于工业级基线,并甚至超越全2000亿token训练。此外,当与工业级静态过滤器结合时,OPUS进一步提高了预训练效率,即使在低质量数据上也是如此。此外,在继续预训练Qwen3-8B-Base于SciencePedia时,OPUS仅使用0.5B tokens就实现了优于全训练3B tokens的性能,展示了在专业领域内显著的数据效率提升。
英文摘要:
As high-quality public text approaches exhaustion, a phenomenon known as the Data Wall, pre-training is shifting from more tokens to better tokens. However, existing methods either rely on heuristic static filters that ignore training dynamics, or use dynamic yet optimizer-agnostic criteria based on raw gradients. We propose OPUS (Optimizer-induced Projected Utility Selection), a dynamic data selection framework that defines utility in the optimizer-induced update space. OPUS scores candidates by projecting their effective updates, shaped by modern optimizers, onto a target direction derived from a stable, in-distribution proxy. To ensure scalability, we employ Ghost technique with CountSketch for computational efficiency, and Boltzmann sampling for data diversity, incurring only 4.7\% additional compute overhead. OPUS achieves remarkable results across diverse corpora, quality tiers, optimizers, and model scales. In pre-training of GPT-2 Large/XL on FineWeb and FineWeb-Edu with 30B tokens, OPUS outperforms industrial-level baselines and even full 200B-token training. Moreover, when combined with industrial-level static filters, OPUS further improves pre-training efficiency, even with lower-quality data. Furthermore, in continued pre-training of Qwen3-8B-Base on SciencePedia, OPUS achieves superior performance using only 0.5B tokens compared to full training with 3B tokens, demonstrating significant data efficiency gains in specialized domains.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。