The recent paradigm shift to large-scale foundation models has brought about a new era for deep learning that, while has found great success in practice, has also been plagued by prohibitively expensive costs in terms of high memory consumption and compute. To mitigate these issues, there has been a concerted effort in post-hoc neural network pruning techniques that do not require costly retraining. Despite the considerable progress being made, existing methods often exhibit a steady drop in model performance as the compression increases. In this paper, we present a novel approach to compressing large transformers, coined OATS, that utilizes the second moment information in the input embeddings to decompose the model weights into a sum of sparse and low-rank matrices. Without any retraining, OATS achieves state-of-the-art performance when compressing models by up to $60\%$ on large language models such as Llama-3 and Phi-3 and vision transformers such as ViT and DINOv2 while delivering up to $1.37\times$ the CPU acceleration versus a model that was comparably pruned.
Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity
Eindhoven University of Technology, the Netherlands（埃因霍温理工大学）；University of Surrey（萨里大学）；Google Research, USA（谷歌研究院）；University of Texas at Austin, USA（德克萨斯大学奥斯汀分校）；Clemson University, USA（克莱姆森大学）；University of Washington, USA（华盛顿大学）；University of Oxford, UK（牛津大学）
SimDiff: Depth Pruning via Similarity and Difference
State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications（网络与交换技术国家重点实验室，北京邮电大学）；Hong Kong University of Science and Technology（香港科学与技术大学）；Laboratory of Data Science and Information Studies, Beijing Information Science and Technology University（数据科学与信息研究实验室，北京信息科技大学）；Beijing Advanced Innovation Center for Materials Genome Engineering, Beijing Information Science and Technology University（材料基因组工程北京先进创新中心，北京信息科技大学）
