Compute-efficient training of language models has become an important issue. We consider data pruning for data-efficient training of LLMs. In this work, we consider a data pruning method based on information entropy. We propose that the samples in the training corpus be ranked in terms of their informativeness which we estimate through entropy functions. The key idea is that, less informative samples are likely to contain redundant information, and thus should be pruned first. We use the entropy functions based on the negative log-likelihood and the average inverse word frequency of a sample as a surrogate to measure its informativeness. Experiments reveal that the proposed information-based pruning can improve upon various language modeling and downstream tasks, and enhance the generalization capability of language models.
Lightweight and Post-Training Structured Pruning for On-Device Large Lanaguage Models
School of Computer Science and Technology, University of Science and Technology of China(中国科学技术大学计算机科学与技术学院);Suzhou Institute for Advanced Research, University of Science and Technology of China(中国科学技术大学苏州高等研究院)
Adaptive Pruning for Large Language Models with Structural Importance Awareness
Harbin Engineering University(哈尔滨工程大学);Shenzhen Future Network of Intelligence Institute (FNii-Shenzhen)(深圳未来网络智能研究院(香港中文大学(深圳)));The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Nanyang Technological University(南洋理工大学)
GRIP-VLM: Group-Relative Importance Pruning for Efficient Vision-Language Models
Institute for AI Industry Research (AIR), Tsinghua University(人工智能产业研究院(AIR),清华大学);Juhaokan Technology Co.,Ltd(极皓科技有限公司);Nanjing University(南京大学);University of Science and Technology of China(中国科学技术大学)
GPrune-LLM: Generalization-Aware Structured Pruning for Large Language Models
Department of Computing, The Hong Kong Polytechnic University, Hong Kong(香港理工大学计算机系);School of Artificial Intelligence and Data Science, IIT Jodhpur(IIT朱罗浦人工智能与数据科学学院)
Fluctuation-based Adaptive Structured Pruning for Large Language Models
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);Wuhan AI Research(武汉人工智能研究院);Objecteye Inc.(Objecteye公司)
Entropy Meets Importance: A Unified Head Importance-Entropy Score for Stable and Efficient Transformer Pruning
Department of Computer Science and Engineering, Korea University, Seoul, South Korea(计算机科学与工程系,韩国大学,首尔,韩国);School of Software, Soongsil University, Seoul, South Korea(软件学院,顺天大学,首尔,韩国)
Resource-Efficient Pruning for Transformer via Low-Rank Importance Estimation
School of Computer Science and Technology, Guangdong University of Technology(广东工业大学计算机科学与技术学院);College of Information and Artificial Intelligence, Yangzhou University(扬州大学信息与人工智能学院)
Synaptic Pruning: A Biological Inspiration for Deep Learning Regularization
College of Science and Engineering, James Cook University(詹姆斯库克大学科学与工程学院);College of Health Care Sciences, James Cook University(詹姆斯库克大学健康护理科学学院);College of Public Health, Medical, and Vet Sciences, James Cook University(詹姆斯库克大学公共卫生、医学与兽医学科学学院)
School of Computing and Artificial Intelligence, Southwest Jiaotong University(计算机与人工智能学院,西南交通大学);Network and Data Security Key Laboratory of Sichuan Province, University of Electronic Science and Technology of China(四川省网络与数据安全重点实验室,电子科技大学);School of Information and Communication, Guilin University of Electronic Technology(信息与通信学院,桂林电子科技大学)
TAP-ViTs: Task-Adaptive Pruning for On-Device Deployment of Vision Transformers
State Key Laboratory of Blockchain and Data Security(区块链与数据安全国家重点实验室);College of Computer Science and Technology(计算机科学与技术学院);College of Computer Science and Electronic Engineering(计算机科学与电子工程学院)
Data Pruning Can Do More: A Comprehensive Data Pruning Approach for Object Re-identification
Technical University of Berlin(柏林工业大学);Huawei Munich Research Center(华为慕尼黑研究中心);Hasso Plattner Institute(哈索·普拉特纳研究所);CISUC, University of Coimbra(科英布拉大学CISUC);Einstein Center Digital Future(爱因斯坦数字未来中心);Science of Intelligence Excellence Cluster(科学智能卓越集群)
One-Cycle Structured Pruning via Stability-Driven Subnetwork Search
IT Application Research Center, Korea Electronics Technology Institute(韩国电子技术研究所应用信息技术中心);Department of Intelligent Semiconductors, Soongsil University(顺慈大学智能半导体系);Department of Computer Science & Engineering, Seoul National University(首尔国立大学计算机科学与工程系)
Teacher-Guided One-Shot Pruning via Context-Aware Knowledge Distillation
Apurba-NSU R&D Lab, Department of Electrical and Computer Engineering, North South University, Dhaka, Bangladesh(阿普尔巴-NSU研发实验室,电气与计算机工程系,北方南大学,达卡,孟加拉国);Apurba Technologies, Sunnyvale, CA 94085, USA(阿普尔巴技术公司,硅谷,加利福尼亚州94085,美国)
Which Layer Causes Distribution Deviation? Entropy-Guided Adaptive Pruning for Diffusion and Flow Models
Stanford University(斯坦福大学);North China Electric Power University(华北电力大学);University of Technology Sydney(悉尼科技大学);Harvard University(哈佛大学);University of Science and Technology of China(中国科学技术大学)