Large volumes of text data have contributed significantly to the development of large language models (LLMs) in recent years. This data is typically acquired by scraping the internet, leading to pretraining datasets comprised of noisy web text. To date, efforts to prune these datasets down to a higher quality subset have relied on hand-crafted heuristics encoded as rule-based filters. In this work, we take a wider view and explore scalable estimates of data quality that can be used to systematically measure the quality of pretraining data. We perform a rigorous comparison at scale of the simple data quality estimator of perplexity, as well as more sophisticated and computationally intensive estimates of the Error L2-Norm and memorization. These metrics are used to rank and prune pretraining corpora, and we subsequently compare LLMs trained on these pruned datasets. Surprisingly, we find that the simple technique of perplexity outperforms our more computationally expensive scoring methods. We improve over our no-pruning baseline while training on as little as 30% of the original training dataset. Our work sets the foundation for unexplored strategies in automatically curating high quality corpora and suggests the majority of pretraining data can be removed while retaining performance.
On the Limitations of Language Targeted Pruning: Investigating the Calibration Language Impact in Multilingual LLM Pruning
Department of Computer Science, TU Dortmund University（图恩-多特蒙德大学计算机科学系）；Bonn-Aachen International Center for Information Technology, University of Bonn（波恩-亚琛国际信息科技中心，波恩大学）；Computer Science School, University of Sheffield（谢菲尔德大学计算机科学学院）；Lamarr Institute for Machine Learning and Artificial Intelligence（拉马尔人工智能与机器学习研究所）
Investigating Structural Pruning and Recovery Techniques for Compressing Multimodal Large Language Models: An Empirical Study
Technical University of Munich, Germany（慕尼黑技术大学，德国）；Helmholtz Munich, Munich Center for Machine Learning, Germany（海德堡慕尼黑，慕尼黑机器学习中心，德国）；University of Tübingen, Tübingen AI Center, Germany（图宾根大学，图宾根人工智能中心，德国）；University of Trento, Italy（特伦托大学，意大利）；Beijing University of Posts and Telecommunications, China（北京邮电大学，中国）
Allen Institute for Artificial Intelligence（艾伦人工智能研究所）；Institute for Language, Cognition and Computation, University of Edinburgh（爱丁堡大学语言、认知与计算研究所）；Language Technologies Institute, Carnegie Mellon University（卡内基梅隆大学语言技术研究所）
AboutMe: Using Self-Descriptions in Webpages to Document the Effects of English Pretraining Data Filters
Allen Institute for AI（艾伦人工智能研究所）；University of California, Berkeley（加州大学伯克利分校）；University of Washington（华盛顿大学）；Emory University（埃默里大学）；Carnegie Mellon University（卡内基梅隆大学）
From $P(y|x)$ to $P(y)$: Investigating Reinforcement Learning in Pre-train Space
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；National University of Singapore（新加坡国立大学）；Tencent AI Lab（腾讯AI实验室）
Small or Large? Zero-Shot or Finetuned? Guiding Language Model Choice for Specialized Applications in Healthcare
British Columbia Cancer Registry Provincial Health Services Authority（不列颠哥伦比亚省癌症登记处省级卫生服务局）；Data Science Institute University of British Columbia（数据科学研究所不列颠哥伦比亚大学）
Towards Robust Calibration Dataset for Domain-General Mixture of Experts Pruning
Institute of Information Science, Academia Sinica（中央研究院資訊科學研究所）；National Taiwan University of Science and Technology（國立臺灣科技大學）；AI Research Center, Hon Hai Research Institute（鴻海研究院人工智能研究中心）
When Meanings Meet: Investigating the Emergence and Quality of Shared Concept Spaces during Multilingual Language Model Training
MaiNLP, Center for Information and Language Processing, LMU Munich（MaiNLP，信息与语言处理中心，慕尼黑大学）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；University of Tokyo（东京大学）；IT University of Copenhagen（哥本哈根IT大学）；Language Technology Lab, University of Cambridge（语言技术实验室，剑桥大学）
How Good is Your Wikipedia? Auditing Data Quality for Low-resource and Multilingual NLP
Department of Computer Science, KU Leuven（库勒万大学计算机科学系）；CENTAL, UCLouvain（UCLouvain的CENTAL）；Department of Computer Science, Aalborg University（奥尔堡大学计算机科学系）；Department of Computer and Information Science, Linköping University（利厄浦大学计算机与信息科学系）；Google Translate（谷歌翻译）
本文审计非英语 Wikipedia 的数据质量，通过多种过滤技术发现短文章与重复问题，并证明质量剪枝可在不降低性能的前提下提升低资源多语言 NLP 的训练效率。
