College of Computer（计算机学院）；National University of Defense Technology（国防科技大学）；School of Computer and Communication Engineering（计算机与通信工程学院）；Changsha University of Science and Technology（长沙理工大学）；Hunan Provincial Key Laboratory of Intelligent Processing of Big Data on Transp（湖南省大数据智能处理重点实验室）
Jinying Xiao, Ping Li, Jie Nie, Bin Ji, Shasha Li, Xiaodong Liu, Jun Ma, Qingbo Wu, Jie Yu
Recently, large language and vision models have shown strong performance, but due to high pre-training and fine-tuning costs, research has shifted towards faster training via dataset pruning. Previous methods used sample loss as an evaluation criterion, aiming to select the most "difficult" samples for training. However, when the pruning rate increases, the number of times each sample is trained becomes more evenly distributed, which causes many critical or general samples to not be effectively fitted. We refer to this as Low-Frequency Learning (LFL). In other words, LFL prevents the model from remembering most samples. In our work, we decompose the scoring function of LFL, provide a theoretical explanation for the inefficiency of LFL, and propose adding a memory term to the scoring function to enhance the model's memory capability, along with an approximation of this memory term. Similarly, we explore memory in Self-Supervised Learning (SSL), marking the first discussion on SSL memory. Using contrastive learning, we derive the memory term both theoretically and experimentally. Finally, we propose Enhance Memory Pruning (EMP), which addresses the issue of insufficient memory under high pruning rates by enhancing the model's memory of data, thereby improving its performance. We evaluated the performance of EMP in tasks such as image classification, natural language understanding, and model pre-training. The results show that EMP can improve model performance under extreme pruning rates. For example, in the CIFAR100-ResNet50 pre-training task, with 70\% pruning, EMP outperforms current methods by 2.2\%.
Attribution-Guided and Coverage-Maximized Pruning for Structural MoE Compression
School of Computer Science and Engineering, Beihang University（北京航空航天大学计算机科学与工程学院）；School of Artificial Intelligence, Beihang University（北京航空航天大学人工智能学院）；Nanyang Technological University（南洋理工大学）
ARMOR: High-Performance Semi-Structured Pruning via Adaptive Matrix Factorization
University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；University of Washington（华盛顿大学）；Princeton University（普林斯顿大学）；Georgia Institute of Technology（佐治亚理工学院）
Pruning Randomly Initialized Neural Networks with Iterative Randomization
NTT Computer and Data Science Laboratories, NTT Corporation（NTT公司NTT计算机与数据科学实验室）；NTT Social Informatics Laboratories, NTT Corporation（NTT公司NTT社会信息学实验室）
Lightweight and Post-Training Structured Pruning for On-Device Large Lanaguage Models
School of Computer Science and Technology, University of Science and Technology of China（中国科学技术大学计算机科学与技术学院）；Suzhou Institute for Advanced Research, University of Science and Technology of China（中国科学技术大学苏州高等研究院）
PrimeSVT: An Automated Memory-aware Pruning Framework with Prioritized Compression Policy for Spiking Vision Transformers
eBRAIN Lab, Division of Engineering, New York University (NYU) Abu Dhabi（eBRAIN实验室，工程系，纽约大学（NYU）阿布扎克分校）；New York University (NYU) Abu Dhabi, United Arab Emirates (UAE)（纽约大学（NYU）阿布扎克分校，阿拉伯联合酋长国（UAE））
Mosaic: Composite Projection Pruning for Resource-efficient LLMs
organization= School of Computer Science, University of St Andrews , addressline= Jack Cole Building , city= St Andrews , postcode= KY16 9SX , state= Fife , country= Scotland, United Kingdom；organization= Autonomous Networking Research \& Innovation Department, Rakuten Mobile, Inc. , addressline= Rakuten Crimson House, 1-14-1 Tamagawa, Setagaya-ku , city= Tokyo , postcode= 158-0094 , state= Tokyo , country= Japan
RANP: Resource Aware Neuron Pruning at Initialization for 3D CNNs
Australian National University（澳大利亚国立大学）；Australian Centre for Robotic Vision（澳大利亚机器人视觉中心）；Data61, CSIRO（澳大利亚联邦科学与工业研究组织Data61）；Microsoft Research（微软研究院）
Faculty of Science and Technology, University of Macau, Macau, China（澳门大学科学与技术学院）；School of Information and Software Engineering, University of Electronic Science and Technology of China, Chengdu, China（电子科技大学信息与软件工程学院）
CoCurve: Cross-Module Co-Pruning Curvature for Structured LLM Pruning
College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算与数据科学学院）；Interdisciplinary Graduate Programme, Nanyang Technological University（南洋理工大学跨学科研究生项目）；School of Electrical and Electronic Engineering, Nanyang Technological University（南洋理工大学电气与电子工程学院）
Not All Data Matters: An End-to-End Adaptive Dataset Pruning Framework for Enhancing Model Performance and Efficiency
National Key Laboratory for Novel Software Technology, Nanjing University（南京大学计算机软件新技术全国重点实验室）；Department of Computer Science and Technology, Nanjing University（南京大学计算机科学与技术系）；School of Artificial Intelligence, Nanjing University（南京大学人工智能学院）；School of Electronic Science and Engineering, Nanjing University（南京大学电子科学与工程学院）
Pruning AMR: Efficient Visualization of Implicit Neural Representations via Weight Matrix Analysis
Center for Applied Mathematics, Cornell University（应用数学中心，康奈尔大学）；Center for Applied Scientific Computing, Lawrence Livermore National Laboratory（应用科学计算中心，劳伦斯·利弗莫尔国家实验室）
