DataMan：用于大语言模型预训练的数据管理器
DataMan: Data Manager for Pre-training Large Language Models
AI总结：
本文提出数据管理器DataMan，通过让LLM自我识别14项质量标准和15个应用领域，对447B token语料进行标注，并据此筛选30B token训练1.3B模型，显著提升了ICL、困惑度和指令遵循能力，验证了质量排序与领域混合的有效性。
摘要：
由数据规模法则驱动的大语言模型（LLMs）性能涌现，使得预训练数据的选择愈发重要。然而，现有方法依赖有限的启发式规则和人类直觉，缺乏全面且清晰的指导准则。为解决这一问题，我们受到“逆向思维”的启发——促使LLMs自我识别哪些标准有利于其性能。由于预训练能力与困惑度（PPL）相关，我们从文本困惑度异常的成因中推导出14项质量标准，并引入15个常见应用领域以支持领域混合。本文中，我们训练了一个数据管理器（DataMan），从逐点评分中学习质量评级和领域识别，并用它为一个447B token的预训练语料标注14项质量评级和领域类型。我们的实验验证了该方法：使用DataMan筛选30B token训练一个1.3B参数的语言模型，在上下文学习（ICL）、困惑度和指令遵循能力方面，相较最先进的基线模型均有显著提升。基于综合评分l=5的最佳模型，超越了使用均匀采样且数据量多50%训练的模型。我们继续使用DataMan标注的高评分、特定领域数据进行预训练，以增强特定领域的ICL性能，从而验证DataMan的领域混合能力。我们的发现强调了质量排序的重要性、质量标准之间的互补性，以及它们与困惑度的低相关性，并分析了PPL与ICL性能之间的错位。我们还全面分析了预训练数据集，考察了其构成、质量评级分布和原始文档来源。
英文摘要：
The performance emergence of large language models (LLMs) driven by data scaling laws makes the selection of pre-training data increasingly important. However, existing methods rely on limited heuristics and human intuition, lacking comprehensive and clear guidelines. To address this, we are inspired by ``reverse thinking'' -- prompting LLMs to self-identify which criteria benefit its performance. As its pre-training capabilities are related to perplexity (PPL), we derive 14 quality criteria from the causes of text perplexity anomalies and introduce 15 common application domains to support domain mixing. In this paper, we train a Data Manager (DataMan) to learn quality ratings and domain recognition from pointwise rating, and use it to annotate a 447B token pre-training corpus with 14 quality ratings and domain type. Our experiments validate our approach, using DataMan to select 30B tokens to train a 1.3B-parameter language model, demonstrating significant improvements in in-context learning (ICL), perplexity, and instruction-following ability over the state-of-the-art baseline. The best-performing model, based on the Overall Score l=5 surpasses a model trained with 50% more data using uniform sampling. We continue pre-training with high-rated, domain-specific data annotated by DataMan to enhance domain-specific ICL performance and thus verify DataMan's domain mixing ability. Our findings emphasize the importance of quality ranking, the complementary nature of quality criteria, and their low correlation with perplexity, analyzing misalignment between PPL and ICL performance. We also thoroughly analyzed our pre-training dataset, examining its composition, the distribution of quality ratings, and the original document sources.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
