arXiv 2511.010662026-04-19cs.CL

HPLT 3.0：非常大规模的多语言资源用于大语言模型和机器翻译。单语和双语数据、多语言评估以及预训练模型

HPLT 3.0: Very Large-Scale Multilingual Resources for LLMs and MT. Mono- and Bi-lingual Data, Multilingual Evaluation, and Pre-Trained Models

University of Oslo, Department of Informatics（奥斯陆大学信息学院）；University of Helsinki, Department of Digital Humanities（赫尔辛基大学数字人文学院）；Prompsit Language Engineering（Prompsit语言工程）；The Common Crawl Foundation（Common Crawl基金会）；Edinburgh University, School of Informatics（爱丁堡大学信息学院）；Charles University, Prague, Institute of Formal and Applied Linguistics（布拉格查理大学形式与应用语言学研究所）；TurkuNLP, University of Turku, Department of Computing（图尔库大学TurkuNLP，计算机系）

Stephan Oepen, Nikolay Arefev, Mikko Aulamo, Marta Bañón, Maja Buljan, Laurie Burchell, Lucas Charpentier, Pinzhen Chen, Mariya Fedorova, Ona de Gibert, Barry H… 展开作者

Stephan Oepen, Nikolay Arefev, Mikko Aulamo, Marta Bañón, Maja Buljan, Laurie Burchell, Lucas Charpentier, Pinzhen Chen, Mariya Fedorova, Ona de Gibert, Barry Haddow, Jan Hajič, Jindřich Helcl, Andrey Kutuzov, Veronika Laippala, Zihao Li, Risto Luukkonen, Bhavitvya Malik, Vladislav Mikhailov, Amanda Myntti, Dayyán O'Brien, Lucie Poláková, Sampo Pyysalo, Gema Ramírez Sánchez, Janine Siewert, Pavel Stepachev, Jörg Tiedemann, Teemu Vahtola, Dušan Variš, Fedor Vitiugin, Tea Vojtěchová, Jaume Zaragoza

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

HPLT 3.0 提供了覆盖近200种语言的高质量大规模文本数据集，包含多语言评估基准和预训练模型，通过数据质量分析和模型评估验证了其有效性。

摘要：

我们提出了一项持续性的倡议，旨在提供开放、非常大、高质量且标注丰富的文本数据集，覆盖近200种语言。这些数据集源自不同来源的网络爬虫，配套完整的开源流程用于从网络档案中选择文档、从HTML中提取文本、识别噪声文本的语言、精确和近似去重，以及标注注册标签、文本质量估计和可识别信息。我们通过对比和分析统计、对24种语言样本的手动检查以及端到端评估各种语言模型架构来报告数据质量。对于多语言LLM评估，我们提供了九种欧洲语言的全面基准，特别强调原生任务、缓解提示敏感性的机制以及分数的规范化和汇总。此外，我们训练和评估了57个单语编码器-解码器模型以及少量单语GPT-like参考模型。除了单语数据和模型外，我们还提供了一组从数据中自动挖掘的平行文本，以及通过机器翻译合成的新型平行语料库。

英文摘要：

We present an ongoing initiative to provide open, very large, high-quality, and richly annotated textual datasets for almost 200 languages. At 30 trillion tokens, this is likely the largest generally available multilingual collection of LLM pre-training data. These datasets are derived from web crawls from different sources and accompanied with a complete, open-source pipeline for document selection from web archives, text extraction from HTML, language identification for noisy texts, exact and near-deduplication, annotation with, among others, register labels, text quality estimates, and personally identifiable information; and final selection and filtering. We report on data quality probes through contrastive and analytical statistics, through manual inspection of samples for 24 languages, and through end-to-end evaluation of various language model architectures trained on this data. For multilingual LLM evaluation, we provide a comprehensive collection of benchmarks for nine European languages, with special emphasis on natively created tasks, mechanisms to mitigate prompt sensitivity, and refined normalization and aggregation of scores. Additionally, we train and evaluate a family of 57 monolingual encoder-decoder models, as well as a handful of monolingual GPT-like reference models. Besides the monolingual data and models, we also present a very large collection of parallel texts automatically mined from this data, together with a novel parallel corpus synthesized via machine translation.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。