arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.12414cs.CL
收藏

propella-1:大规模LLM数据整理中的多属性文档标注

propella-1: Multi-Property Document Annotation for LLM Data Curation at Scale

ellamind;Leibniz University Hannover(汉诺威莱布尼茨大学)

Maximilian Idahl, Benedikt Droste, Björn Plüster, Jan Philipp Harries

AI总结:

propella-1通过多属性文档标注提升大规模LLM数据整理的效率和质量,揭示预训练数据集在质量、推理深度和内容组成上的显著差异。

摘要:

自从FineWeb-Edu以来,LLM预训练的数据整理主要依赖于由小型分类器产生的单一标量质量评分。单一评分会混合同一质量维度,防止灵活过滤,并且提供不了可解释性。我们介绍了propella-1,一组小型多语言LLM(0.6B、1.7B、4B参数),能够对文本文档进行18个属性的标注,这些属性分为六个类别:核心内容、分类、质量和价值、受众和目的、安全性和合规性以及地理相关性。这些模型支持57种语言,并产生符合预定义模式的结构化JSON标注。在与前沿商业LLM作为参考标注器的评估中,4B模型在与许多更大通用模型相比,实现了更高的一致性。我们发布了propella-annotations,一个包含超过30亿文档标注的数据集,覆盖了主要预训练语料库,包括来自FineWeb-2、FinePDFs、HPLT 3.0和Nemotron-CC的数据。利用这些标注,我们展示了对广泛使用的预训练数据集的多维组合分析,揭示了单评分方法无法捕捉到的质量、推理深度和内容组成方面的显著差异。所有模型权重和标注均在宽松、允许商业使用许可下发布。

英文摘要:

Since FineWeb-Edu, data curation for LLM pretraining has predominantly relied on single scalar quality scores produced by small classifiers. A single score conflates multiple quality dimensions, prevents flexible filtering, and offers no interpretability. We introduce propella-1, a family of small multilingual LLMs (0.6B, 1.7B, 4B parameters) that annotate text documents across 18 properties organized into six categories: core content, classification, quality and value, audience and purpose, safety and compliance, and geographic relevance. The models support 57 languages and produce structured JSON annotations conforming to a predefined schema. Evaluated against a frontier commercial LLM as a reference annotator, the 4B model achieves higher agreement than much larger general-purpose models. We release propella-annotations, a dataset of over three billion document annotations covering major pretraining corpora including data from FineWeb-2, FinePDFs, HPLT 3.0, and Nemotron-CC. Using these annotations, we present a multi-dimensional compositional analysis of widely used pretraining datasets, revealing substantial differences in quality, reasoning depth, and content composition that single-score approaches cannot capture. All model weights and annotations are released under permissive, commercial-use licenses.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑