大型语言模型（LLMs）的训练成本高昂。在本文中，我们研究了预训练LLMs的数据高效方法，即旨在优化模型质量与训练资源/数据消耗之间帕累托前沿的技术。我们试图理解基于以下两种方法的数据选择例程的权衡：(i) 计算成本高昂的数据质量估计，以及 (ii) 在特征空间中最大化覆盖率和基于多样性的度量。我们的第一种技术Ask-LLM利用指令微调LLMs的零样本推理能力来直接评估训练样本的质量。为了实现覆盖率，我们提出了Density sampling，该方法对数据分布进行建模以选择多样化的样本。在我们对19个采样器的比较中，涉及数百个评估任务和预训练运行，我们发现Ask-LLM和Density是各自类别中的最佳方法。覆盖率采样能够恢复完整数据的性能，而在Ask-LLM数据上训练的模型始终优于完整数据训练——即使我们拒绝了原始数据集的90%，同时收敛速度提高了70%。
英文摘要：
The training of large language models (LLMs) is expensive. In this paper, we study data-efficient approaches for pre-training LLMs, i.e., techniques that aim to optimize the Pareto frontier of model quality and training resource/data consumption. We seek to understand the tradeoffs associated with data selection routines based on (i) expensive-to-compute data-quality estimates, and (ii) maximization of coverage and diversity-based measures in the feature space. Our first technique, Ask-LLM, leverages the zero-shot reasoning capabilities of instruction-tuned LLMs to directly assess the quality of a training example. To target coverage, we propose Density sampling, which models the data distribution to select a diverse sample. In our comparison of 19 samplers, involving hundreds of evaluation tasks and pre-training runs, we find that Ask-LLM and Density are the best methods in their respective categories. Coverage sampling can recover the performance of the full data, while models trained on Ask-LLM data consistently outperform full-data training -- even when we reject 90% of the original dataset, while converging up to 70% faster.
How Post-Training Reshapes LLMs: A Mechanistic View on Knowledge, Truthfulness, Refusal, and Confidence
University of California, Los Angeles（加州大学洛杉矶分校）；University of Alberta（阿尔伯塔大学）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Harvard University（哈佛大学）
University of Cambridge（剑桥大学）；ELLIS Institute Tübingen（ELLIS研究所图宾根）；MPI-IS（马克斯·普朗克智能系统研究所）；Tübingen AI Center（图宾根人工智能中心）；Mila（Mila魁北克人工智能研究所）；University of Montreal（蒙特利尔大学）
Context-Aware Membership Inference Attacks against Pre-trained Large Language Models
Mohamed bin Zayed University of Artificial Intelligence（莫扎伊德·本·扎耶德人工智能大学）；Brave Software（Brave软件公司）；Imperial College London（帝国理工学院）；National University of Singapore（新加坡国立大学）
Department of Statistics and Operations Research, UNC-Chapel Hill（统计与运筹学系，北卡罗来纳大学 Chapel Hill 分校）；Imperial College Business School, Imperial College London（帝国理工学院伦敦校区商学院）
Can LLMs Reason Over Non-Text Modalities in a Training-Free Manner? A Case Study with In-Context Representation Learning
College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算机与数据科学学院）；AI-X, Interdisciplinary Graduate Programme, Nanyang Technological University（南洋理工大学人工智能交叉研究生项目）；Lee Kong Chian School of Medicine, Nanyang Technological University（南洋理工大学李科钦医学院）；Centre of AI in Medicine (C-AIM), Nanyang Technological University（南洋理工大学医学人工智能中心）；University of Toronto（多伦多大学）；Brigham and Women’s Hospital, Harvard Medical School（哈佛医学院布里洛妇女医院）；Massachusetts Institute of Technology（麻省理工学院）
Towards Tracing Trustworthiness Dynamics: Revisiting Pre-training Period of Large Language Models
Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Shanghai Jiao Tong University（上海交通大学）；The University of Sydney（悉尼大学）；Renmin University of China（中国人民大学）；University of Chinese Academy of Sciences（中国科学院大学）
PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training
Microsoft Corporation（微软公司）；School of Computer Science, Peking University（北京大学计算机学院）；National Key Laboratory for Multimedia Information Processing, Peking University（北京大学多媒体信息处理国家重点实验室）
