Large Language Models (LLMs) are not only resource-intensive to train but even more costly to deploy in production. Therefore, recent work has attempted to prune blocks of LLMs based on cheap proxies for estimating block importance, effectively removing 10% of blocks in well-trained LLaMa-2 and Mistral 7b models without any significant degradation of downstream metrics. In this paper, we explore different block importance metrics by considering adaptive metrics such as Shapley value in addition to static ones explored in prior work. We show that adaptive metrics exhibit a trade-off in performance between tasks i.e., improvement on one task may degrade performance on the other due to differences in the computed block influences. Furthermore, we extend this analysis from a complete block to individual self-attention and feed-forward layers, highlighting the propensity of the self-attention layers to be more amendable to pruning, even allowing removal of upto 33% of the self-attention layers without incurring any performance degradation on MMLU for Mistral 7b (significant reduction in costly maintenance of KV-cache). Finally, we look at simple performance recovery techniques to emulate the pruned layers by training lightweight additive bias or low-rank linear adapters. Performance recovery using emulated updates avoids performance degradation for the initial blocks (up to 5% absolute improvement on MMLU), which is either competitive or superior to the learning-based technique.
Locality-Aware Redundancy Pruning for LLM Depth Compression
University of Southern California(美国南加州大学);Neural Superintelligence Lab, MODULABS(MODULABS神经超级智能实验室);Seoul National University(首尔国立大学);Inha University(釜山大学)
SimDiff: Depth Pruning via Similarity and Difference
State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications(网络与交换技术国家重点实验室,北京邮电大学);Hong Kong University of Science and Technology(香港科学与技术大学);Laboratory of Data Science and Information Studies, Beijing Information Science and Technology University(数据科学与信息研究实验室,北京信息科技大学);Beijing Advanced Innovation Center for Materials Genome Engineering, Beijing Information Science and Technology University(材料基因组工程北京先进创新中心,北京信息科技大学)
FlattenGPT: Depth Compression for Transformer with Layer Flattening
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University(多媒体信息处理国家重点实验室,计算机科学学院,北京大学);Tsinghua University(清华大学)
Fast Think-on-Graph: Wider, Deeper and Faster Reasoning of Large Language Model on Knowledge Graph
School of Cyberspace Security, Beijing University Of Posts And Telecommunications(北京邮电大学网络空间安全学院);Peng Cheng Laboratory(鹏城实验室);School of Computer Science and Technology, Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳)计算机科学与技术学院)
On the Limitations of Language Targeted Pruning: Investigating the Calibration Language Impact in Multilingual LLM Pruning
Department of Computer Science, TU Dortmund University(图恩-多特蒙德大学计算机科学系);Bonn-Aachen International Center for Information Technology, University of Bonn(波恩-亚琛国际信息科技中心,波恩大学);Computer Science School, University of Sheffield(谢菲尔德大学计算机科学学院);Lamarr Institute for Machine Learning and Artificial Intelligence(拉马尔人工智能与机器学习研究所)
MoPE-CLIP: Structured Pruning for Efficient Vision-Language Models with Module-wise Pruning Error Metric
School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);City University of Hong Kong(香港城市大学);Huawei Noah’s Ark Lab(华为诺亚方舟实验室);The Hong Kong University of Science and Technology(香港科技大学);Nanyang Technological University(南洋理工大学)
Spanning Training Progress: Temporal Dual-Depth Scoring (TDDS) for Enhanced Dataset Pruning
XiDian University(西安电子科技大学);Centre for Frontier AI Research (CFAR), Agency for Science, Technology and Research (A*STAR)(新加坡科技研究局前沿人工智能研究中心);Institute of High Performance Computing (IHPC), Agency for Science, Technology and Research (A*STAR)(新加坡科技研究局高性能计算研究所)
Unifying Depth and Width Pruning for LLMs via Binary Knapsack Optimization
Yardi School of Artificial Intelligence, Indian Institute of Technology Delhi(德里印度理工学院亚迪人工智能学院);Indian Institute of Technology Delhi(德里印度理工学院);Microsoft Research (MSR) India(微软研究院印度分部)
From LLMs to LRMs: Rethinking Pruning for Reasoning-Centric Models
Institute of Digital Twin, Eastern Institute of Technology(数字孪生研究院,东部技术研究所);Department of Computing, The Hong Kong Polytechnic University(计算系,香港理工大学);Tencent Hunyuan / AI Lab(腾讯文英/人工智能实验室);Université Paris Dauphine - PSL(巴黎-萨克勒大学)
Beyond FLOPs: Benchmarking Real Inference Acceleration of LLM Pruning under a GEMM-Centric Taxonomy
Ningbo Institute of Digital Twin, Eastern Institute of Technology, Ningbo(宁波数字孪生研究院,东方理工大学(宁波));Department of Computing, The Hong Kong Polytechnic University(香港理工大学计算学系);Munich Center for Machine Learning, LMU Munich(慕尼黑大学机器学习慕尼黑中心)
Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrence
University of Maryland(马里兰大学);New York University(纽约大学);Lawrence Livermore National Laboratory(劳伦斯利弗莫尔国家实验室);University of North Carolina(北卡罗来纳大学);ELLIS Institute Tübingen, Max Planck Institute for Intelligent Systems, Tübingen AI Center(图宾根ELLIS研究所、马克斯·普朗克智能系统研究所、图宾根人工智能中心);Columbia University(哥伦比亚大学)