Through pretraining on a corpus with various sources, Large Language Models (LLMs) have gained impressive performance. However, the impact of each component of the pretraining corpus remains opaque. As a result, the organization of the pretraining corpus is still empirical and may deviate from the optimal. To address this issue, we systematically analyze the impact of 48 datasets from 5 major categories of pretraining data of LLMs and measure their impacts on LLMs using benchmarks about nine major categories of model capabilities. Our analyses provide empirical results about the contribution of multiple corpora on the performances of LLMs, along with their joint impact patterns, including complementary, orthogonal, and correlational relationships. We also identify a set of ``high-impact data'' such as Books that is significantly related to a set of model capabilities. These findings provide insights into the organization of data to support more efficient pretraining of LLMs.
PPU-Bench:Real World Benchmark for Personalized Partial Unlearning in Vision Language Models
Harbin Institute of Technology（哈尔滨工业大学）；Pengcheng Laboratory（鹏城实验室）；The Hong Kong Polytechnic University（香港理工大学）；Sichuan University（四川大学）；Zhejiang Normal University（浙江师范大学）
Capability Provenance in Language Models: A Case Study in Social Reasoning
Georgia Institute of Technology, College of Computing（佐治亚理工学院计算学院）；MATS Program（MATS项目）；EleutherAI；KAIST AI（韩国科学技术院人工智能学院）；Georgia Tech AI Safety Initiative（佐治亚理工学院人工智能安全倡议）
