Dataset pruning aims to select a subset of a dataset for efficient model training. While data efficiency in natural language processing has primarily focused on within-corpus scenarios during model pre-training, efficient dataset pruning for task-specific fine-tuning across diverse datasets remains challenging due to variability in dataset sizes, data distributions, class imbalance and label spaces. Current cross-dataset pruning techniques for fine-tuning often rely on computationally expensive sample ranking processes, typically requiring full dataset training or reference models. We address this gap by proposing Swift Cross-Dataset Pruning (SCDP). Specifically, our approach uses TF-IDF embeddings with geometric median to rapidly evaluate sample importance. We then apply dataset size-adaptive pruning to ensure diversity: for smaller datasets, we retain samples far from the geometric median, while for larger ones, we employ distance-based stratified pruning. Experimental results on six diverse datasets demonstrate the effectiveness of our method, spanning various tasks and scales while significantly reducing computational resources. Source code is available at: https://github.com/he-y/NLP-Dataset-Pruning
Systematic Weight Evaluation for Pruning Large Language Models: Enhancing Performance and Sustainability
organization= College Of Science \& Engineering, Hamad Bin Khalifa University , addressline= Education City , city= Doha , postcode= 34110 , country= Qatar
$\rm SP^3$: Enhancing Structured Pruning via PCA Projection
School of Information, Renmin University of China(中国人民大学信息学院);Tencent AI Lab, Tencent(腾讯AI实验室);School of Computer Science and Technology, Xi’an Jiaotong University(西安交通大学计算机科学与技术学院)
SpecVLM: Enhancing Speculative Decoding of Video LLMs via Verifier-Guided Token Pruning
The State Key Laboratory of Blockchain and Data Security(区块链与数据安全国家重点实验室);Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security(杭州高新技术区(滨江)区块链与数据安全研究院);Department of Computing, The Hong Kong Polytechnic University(香港理工大学计算机系);School of Computer Science, Beijing University of Posts and Telecommunications(北京邮电大学计算机学院)
Enhancing Knowledge Density via Hierarchical Context Pruning for Retrieval-Augmented Repository-Level Code Completion
MIIT Key Laboratory of Pattern Analysis and Machine Intelligence(信息科技模式分析与机器智能国家重点实验室);College of Computer Science and Technology(计算机科学与技术学院);Huaneng Information Technology Co Ltd(华能信息技术有限公司);Hunan Vanguard Group Corporation Co Ltd(湖南 Vanguard集团有限公司);Tsinghua University(清华大学);Zhejiang University(浙江大学);PowerChina Huadong Engineering Co Ltd(国家电网华东工程有限公司)
Condense, Don't Just Prune: Enhancing Efficiency and Performance in MoE Layer Pruning
University of Surrey(萨里大学);Clemson University(克莱姆森大学);Meituan(美团);University of Texas at Austin(德克萨斯大学奥斯汀分校);Sun Yat-sen University(中山大学);The University of Arizona(亚利桑那大学);ELLIS Institute(ELLIS研究所);Max Planck Institute for Intelligent Systems(智能系统马克斯·普朗克研究所);Tübingen AI Center(图宾根人工智能中心)
Fluctuation-based Adaptive Structured Pruning for Large Language Models
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);Wuhan AI Research(武汉人工智能研究院);Objecteye Inc.(Objecteye公司)
PRISM: Self-Pruning Intrinsic Selection Method for Training-Free Multimodal Data Selection
LMU Munich(慕尼黑大学);Munich Research Center, Huawei Technologies(慕尼黑研究中心,华为技术);METEOR;School of Computer Science, Wuhan University(武汉大学计算机学院);Munich Center for Machine Learning(慕尼黑机器学习中心)
Cross-Model Comparative Loss for Enhancing Neuronal Utility in Language Understanding
CAS Key Laboratory of AI Security, Institute of Computing Technology, CAS(中国科学院人工智能安全重点实验室 中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);Institute for AI Industry Research, Tsinghua University(清华大学人工智能产业研究院)