Since FineWeb-Edu, data curation for LLM pretraining has predominantly relied on single scalar quality scores produced by small classifiers. A single score conflates multiple quality dimensions, prevents flexible filtering, and offers no interpretability. We introduce propella-1, a family of small multilingual LLMs (0.6B, 1.7B, 4B parameters) that annotate text documents across 18 properties organized into six categories: core content, classification, quality and value, audience and purpose, safety and compliance, and geographic relevance. The models support 57 languages and produce structured JSON annotations conforming to a predefined schema. Evaluated against a frontier commercial LLM as a reference annotator, the 4B model achieves higher agreement than much larger general-purpose models. We release propella-annotations, a dataset of over three billion document annotations covering major pretraining corpora including data from FineWeb-2, FinePDFs, HPLT 3.0, and Nemotron-CC. Using these annotations, we present a multi-dimensional compositional analysis of widely used pretraining datasets, revealing substantial differences in quality, reasoning depth, and content composition that single-score approaches cannot capture. All model weights and annotations are released under permissive, commercial-use licenses.
AboutMe: Using Self-Descriptions in Webpages to Document the Effects of English Pretraining Data Filters
Allen Institute for AI(艾伦人工智能研究所);University of California, Berkeley(加州大学伯克利分校);University of Washington(华盛顿大学);Emory University(埃默里大学);Carnegie Mellon University(卡内基梅隆大学)
Oasis: Data Curation and Assessment System for Pretraining of Large Language Models
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Beijing Unisound Information Technology Co., Ltd(北京云知声信息技术有限公司)
Equipping Retrieval-Augmented Large Language Models with Document Structure Awareness
School of Computer Science, Beijing Institute of Technology(计算机学院,北京理工大学);Ant Group(蚂蚁集团);Southeast Academy of Information Technology, Beijing Institute of Technology(信息科技东南学院,北京理工大学)
Large Language Models Are Effective Human Annotation Assistants, But Not Good Independent Annotators
Department of Computer Science, University of Maryland(大学计算机科学系);National Consortium for the Study of Terrorism and Responses to Terrorism(反恐与反恐响应国家研究中心)
Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research
Allen Institute for AI(艾伦人工智能研究所);University of California, Berkeley(加州大学伯克利分校);Carnegie Mellon University(卡内基梅隆大学);Spiffy AI(Spiffy AI公司);Massachusetts Institute of Technology(麻省理工学院);University of Washington(华盛顿大学)
ABCD-LINK: Annotation Bootstrapping for Cross-Document Fine-Grained Links
Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science and Hessian Center for AI (hessian.AI), TU Darmstadt(图腾知识处理实验室(UKP实验室)、计算机科学系和海德堡人工智能中心(hessian.AI)、图腾达姆斯塔特大学);Hebrew University of Jerusalem and The Allen Institute for AI (AI2)(耶路撒冷希伯来大学和人工智能研究所(AI2))
Instruction-Tuning Data Synthesis from Scratch via Web Reconstruction
The Hong Kong University of Science and Technology (Guangzhou)(香港科学与技术大学(广州));The Hong Kong University of Science and Technology(香港科学与技术大学);Huawei Noah’s Ark Lab(华为诺亚实验室)
Low resource language dataset creation, curation and classification: Setswana and Sepedi -- Extended Abstract
University of Pretoria(比勒陀利亚大学);CSIR(科学与工业研究理事会);University of Zululand(祖鲁兰大学);University of Cape Town(开普敦大学);University of Limpopo(林波波大学);North-West University(西北大学);University of the Witwatersrand(威特沃特斯兰德大学)
Data-Centric AI in the Age of Large Language Models
National University of Singapore(新加坡国立大学);Agency for Science, Technology and Research(新加坡科技研究局);Singapore-MIT Alliance for Research and Technology Centre(新加坡-麻省理工研究与技术联盟中心);Guangdong Lab of AI and Digital Economy (SZ)(广东人工智能与数字经济实验室(深圳));Massachusetts Institute of Technology(麻省理工学院);Allen Institute for AI(艾伦人工智能研究所);University of Washington(华盛顿大学)
School of Information Sciences, University of Illinois Urbana Champaign(伊利诺伊大学厄巴纳-香槟分校信息科学学院);Informatics, University of Illinois Urbana Champaign(伊利诺伊大学厄巴纳-香槟分校信息学系);National Center for Supercomputing Applications (NCSA)(国家超级计算应用中心)
Investigating an approach for low resource language dataset creation, curation and classification: Setswana and Sepedi
University of Pretoria(比勒陀利亚大学);CSIR(科学与工业研究理事会);University of Zululand(祖鲁兰大学);University of Cape Town(开普敦大学);University of Limpopo(林波波大学);North-West University(西北大学);University of the Witwatersrand(威特沃特斯兰德大学)
KARMA: Leveraging Multi-Agent LLMs for Automated Knowledge Graph Enrichment
Department of Big Data and Biomedical AI, Peking University(北京大学大数据与生物医学人工智能系);Wallace H. Coulter Department of Biomedical Engineering, Georgia Institute of Technology(佐治亚理工学院生物医学工程系);School of Architecture, Tsinghua University(清华大学建筑学院)
FlyAOC: Evaluating Agentic Ontology Curation of Drosophila Scientific Knowledge Bases
University of Michigan(密歇根大学);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);University of California, Berkeley(加州大学伯克利分校);University of Washington(华盛顿大学)
BigDocs: An Open Dataset for Training Multimodal Models on Document and Code Tasks
ServiceNow(ServiceNow公司);Mila(米拉计算科学研究所);École de Technologie Supérieure(高等技术学院);University of Waterloo(滑铁卢大学);Université de Montréal(蒙特利尔大学);McGill University(麦吉尔大学);York University(约克大学);University of British Columbia(不列颠哥伦比亚大学);Universitat Autònoma de Barcelona(巴塞罗那自治大学);Polytechnique Montréal(蒙特利尔理工学院)