Rethinking Data Selection at Scale: Random Selection is Almost All You Need
Alibaba Group(阿里巴巴集团);School of Artificial Intelligence, Jilin University(吉林大学人工智能学院);International Center of Future Science, Jilin University(吉林大学未来科学国际中心)
Tingyu Xia, Bowen Yu, Kai Dang, An Yang, Yuan Wu, Yuan Tian, Yi Chang, Junyang Lin
Supervised fine-tuning (SFT) is crucial for aligning Large Language Models (LLMs) with human instructions. The primary goal during SFT is to select a small yet representative subset of training data from the larger pool, such that fine-tuning with this subset achieves results comparable to or even exceeding those obtained using the entire dataset. However, most existing data selection techniques are designed for small-scale data pools, which fail to meet the demands of real-world SFT scenarios. In this paper, we replicated several self-scoring methods those that do not rely on external model assistance on two million scale datasets, and found that nearly all methods struggled to significantly outperform random selection when dealing with such large-scale data pools. Moreover, our comparisons suggest that, during SFT, diversity in data selection is more critical than simply focusing on high quality data. We also analyzed the limitations of several current approaches, explaining why they perform poorly on large-scale datasets and why they are unsuitable for such contexts. Finally, we found that filtering data by token length offers a stable and efficient method for improving results. This approach, particularly when training on long text data, proves highly beneficial for relatively weaker base models, such as Llama3.
Take the essence and discard the dross: A Rethinking on Data Selection for Fine-Tuning Large Language Models
The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Shenzhen Research Institute of Big Data(深圳大数据研究院);University of Science and Technology of China(中国科学技术大学)
Beyond Truncation: Rethinking LLM Decoding as Ensemble Pruning
Institute of Statistics and Big Data, Renmin University of China(中国人民大学统计与大数据研究院);Big Data and Responsible Artificial Intelligence for National Governance, Renmin University of China(中国人民大学大数据与国家治理研究院);Center for Applied Statistics, Institute of Statistics and Big Data, Renmin University of China(中国人民大学统计与大数据研究院应用统计中心)
Rethinking Data Curation in LLM Training: Online Reweighting Offers Better Generalization than Offline Methods
University of Cambridge(剑桥大学);OATML, University of Oxford(牛津大学OATML实验室);University of Toronto(多伦多大学);Shanghai Jiao Tong University(上海交通大学);Samsung AI Center(三星人工智能中心)
Rethinking KV Cache Eviction via a Unified Information-Theoretic Objective
College of Computer Science, Sichuan University, Chengdu, China(四川大学计算机学院,成都,中国);Institute of High Performance Computing, A STAR, Singapore(高性能计算研究院,新加坡)
Rethinking Efficient Graph Coarsening via a Non-Selfishness Principle
School of Information Science and Electronic Engineering, Shanghai Jiao Tong University, Shanghai, China(上海交通大学信息科学与电子工程学院);School of Artificial Intelligence, Shanghai Jiao Tong University, Shanghai, China(上海交通大学人工智能学院);School of Environment Science and Engineering, Shanghai Jiao Tong University, Shanghai, China(上海交通大学环境科学与工程学院);School of Artificial Intelligence, Nanchang University, Nanchang, China(南昌大学人工智能学院);Institute of Geographic Sciences and Natural Resources Research, Chinese Academy of Sciences, Beijing, China(中国科学院地理科学与资源研究所)
Rethinking Retrieval-Augmented Generation for Medicine: A Large-Scale, Systematic Expert Evaluation and Practical Insights
Yale School of Medicine(耶鲁医学院);Yale University(耶鲁大学);ETH Zurich(苏黎世联邦理工学院);Harvard Medical School(哈佛医学院);Geisel School of Medicine at Dartmouth(达特茅斯大学盖塞医学院);Seoul National University College of Medicine(首尔国立大学医学院);Hanyang University College of Medicine(翰林大学医学院);PA Leadership Charter School(宾夕法尼亚州西切斯特市领导学校)