The impressive capabilities of recent language models can be largely attributed to the multi-trillion token pretraining datasets that they are trained on. However, model developers fail to disclose their construction methodology which has lead to a lack of open information on how to develop effective pretraining sets. To address this issue, we perform the first systematic study across the entire pipeline of pretraining set construction. First, we run ablations on existing techniques for pretraining set development to identify which methods translate to the largest gains in model accuracy on downstream evaluations. Then, we categorize the most widely used data source, web crawl snapshots, across the attributes of toxicity, quality, type of speech, and domain. Finally, we show how such attribute information can be used to further refine and improve the quality of a pretraining set. These findings constitute an actionable set of steps that practitioners can use to develop high quality pretraining sets.
EURO-5K: When Does Domain Pretraining Matter? Benchmarking Transformers for EU Reporting Obligation Extraction
Division of Computer Science, School of Electrical and Computer Engineering(计算机科学系,电气与计算机工程学院);National Technical University of Athens(雅典技术大学);Department of Humanities Social Sciences and Law, School of Applied Mathematical and Physical Sciences(人文社会科学与法律系,应用数学与物理科学学院)
A Practitioner's Guide to Continual Multimodal Pretraining
Tübingen AI Center(蒂宾根人工智能中心);University of Tübingen(蒂宾根大学);Helmholtz Munich(亥姆霍兹慕尼黑中心);University of Cambridge(剑桥大学);LAION;Jülich Supercomputing Center (JSC)(于利希超级计算中心);Research Center Jülich (FZJ)(于利希研究中心);Helmholtz Association(亥姆霍兹联合会);Google DeepMind(谷歌DeepMind);Munich Center for ML(慕尼黑机器学习中心);Technical University of Munich(慕尼黑工业大学)
Continual Pretraining on Encrypted Synthetic Data for Privacy-Preserving LLMs
The PII shown in the figure is synthetic and not real(合成的PII);International Digital Economy Academy(国际数字经济学院);The Hong Kong University of Science and Technology (Guangzhou)(香港科学与技术大学(广州));The Hong Kong University of Science and Technology(香港科学与技术大学);DataArc Tech Ltd(DataArc科技有限公司)
Data Mixing for Large Language Models Pretraining: A Survey and Outlook
The International Joint Institute of Tianjin University(天津大学国际联合研究院);Tianjin University(天津大学);TJUNLP Lab, School of Computer Science and Technology(天津大学自然语言处理实验室,计算机科学与技术学院)
LLMs on the Line: Data Determines Loss-to-Loss Scaling Laws
Max Planck Institute for Intelligent Systems(智能系统马克斯·普朗克研究所);ELLIS Institute Tübingen(图宾根ELLIS研究所);Tübingen AI Center(图宾根人工智能中心);University of Tübingen(图宾根大学)