Structured pruning is a promising hardware-friendly compression technique for large language models (LLMs), which is expected to be retraining-free to avoid the enormous retraining cost. This retraining-free paradigm involves (1) pruning criteria to define the architecture and (2) distortion reconstruction to restore performance. However, existing methods often emphasize pruning criteria while using reconstruction techniques that are specific to certain modules or criteria, resulting in limited generalizability. To address this, we introduce the Linear Interpolation-based Adaptive Reconstruction (LIAR) framework, which is both efficient and effective. LIAR does not require back-propagation or retraining and is compatible with various pruning criteria and modules. By applying linear interpolation to the preserved weights, LIAR minimizes reconstruction error and effectively reconstructs the pruned output. Our evaluations on benchmarks such as GLUE, SQuAD, WikiText, and common sense reasoning show that LIAR enables a BERT model to maintain 98% accuracy even after removing 50% of its parameters and achieves top performance for LLaMA in just a few minutes.
Ensembling Pruned Attention Heads For Uncertainty-Aware Efficient Transformers
U2IS, ENSTA, Institut Polytechnique de Paris（U2IS、ENSTA、巴黎理工学院）；University of Trento（特伦托大学）；NVIDIA；University of Bergamo, Italy（意大利贝拉姆博大学）；Fondazione Bruno Kessler (FBK)（布鲁诺·凯斯勒基金会）；AMIAD, Pôle Recherche, Palaiseau（AMIAD、研究部、帕莱索）
Olica: Efficient Structured Pruning of Large Language Models without Retraining
Center of Statistical Research, School of Statistics（统计研究中心、统计学与数据科学学院）；New Cornerstone Science Laboratory, Southwestern University of Finance（新基石科学实验室、西南财经大学）
A Free Lunch in LLM Compression: Revisiting Retraining after Pruning
Department for AI in Society, Science, and Technology, Zuse Institute Berlin（人工智能社会、科学与技术系，柏林Zuse研究所）；Institute of Mathematics, Technische Universität Berlin（数学系，柏林技术大学）
PERP: Rethinking the Prune-Retrain Paradigm in the Era of LLMs
Department for AI in Society, Science, and Technology, Zuse Institute Berlin, Germany（人工智能与社会、科学与技术部门，柏林Zuse研究所）；Institute of Mathematics, Technische Universität Berlin, Germany（数学研究所，柏林技术大学）
Automated Tuning for Diffusion Inverse Problem Solvers without Generative Prior Retraining
Department of Electrical \& Computer Engineering, University of Minnesota, MN, USA；Center for Magnetic Resonance Research, University of Minnesota, MN, USA
Roots Beneath the Cut: Uncovering the Risk of Concept Revival in Pruning-Based Unlearning for Diffusion Models
University of Georgia（佐治亚大学）；Carnegie Mellon University（卡内基梅隆大学）；Northeastern University（东北大学）；Stevens Institute of Technology（史蒂文斯理工学院）；University of Arizona（亚利桑那大学）
PET-Adapter: Test-Time Domain Adaptation for Full and Limited-Angle PET Image Reconstruction
Institute of Imaging and Computer Vision, RWTH Aachen University, Germany（成像与计算机视觉研究所，亚琛工业大学，德国）；Machine Learning for Medical Data, Heinrich Heine University Düsseldorf, Germany（医学数据机器学习，杜伊斯堡-艾森大学，德国）
