arXiv 2203.044662022-03cs.LGcs.CV

The Combinatorial Brain Surgeon: Pruning Weights That Cancel One Another in Neural Networks

University of Utah（犹他大学）；Bucknell University（巴克内尔大学）；Google Research（谷歌研究院）

Xin Yu, Thiago Serra, Srikumar Ramalingam, Shandian Zhe

英文摘要：

Neural networks tend to achieve better accuracy with training if they are larger -- even if the resulting models are overparameterized. Nevertheless, carefully removing such excess parameters before, during, or after training may also produce models with similar or even improved accuracy. In many cases, that can be curiously achieved by heuristics as simple as removing a percentage of the weights with the smallest absolute value -- even though magnitude is not a perfect proxy for weight relevance. With the premise that obtaining significantly better performance from pruning depends on accounting for the combined effect of removing multiple weights, we revisit one of the classic approaches for impact-based pruning: the Optimal Brain Surgeon(OBS). We propose a tractable heuristic for solving the combinatorial extension of OBS, in which we select weights for simultaneous removal, as well as a systematic update of the remaining weights. Our selection method outperforms other methods under high sparsity, and the weight update is advantageous even when combined with the other methods.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

阅读原文

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

本文内容

英文摘要相关论文

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

继续探索

相关论文

Fast as CHITA: Neural Network Pruning with Combinatorial Optimization

MIT（麻省理工学院）；Google Research（谷歌研究院）

The sheer size of modern neural networks makes model serving a serious computational challenge. A popular class of compression techniques overcomes this challenge by pruning or sparsifying the weights of pretrained networks. While useful, these techniques often face serious tradeoffs between computational requirements and compression quality. In this work, we propose a novel optimization-based pruning framework that considers the combined effect of pruning (and updating) multiple weights subject to a sparsity constraint. Our approach, CHITA, extends the classical Optimal Brain Surgeon framewor…

2023-02-28cs.CVcs.LGmath.OC

The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Models

Institute of Science and Technology Austria（奥地利科学技术研究所）；Neural Magic Inc.（Neural Magic公司）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）

Transformer-based language models have become a key building block for natural language processing. While these models are extremely accurate, they can be too large and computationally intensive to run on standard deployments. A variety of compression methods, including distillation, quantization, structured and unstructured pruning are known to decrease model size and increase inference speed, with low accuracy loss. In this context, this paper's contributions are two-fold. We perform an in-depth study of the accuracy-compression trade-off for unstructured weight pruning of BERT models. We in…

2022-10-17cs.CLcs.LG

Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeon

Nanyang Technological University（南洋理工大学）

How to develop slim and accurate deep neural networks has become crucial for real- world applications, especially for those employed in embedded systems. Though previous work along this research line has shown some promising results, most existing methods either fail to significantly compress a well-trained deep network or require a heavy retraining process for the pruned deep network to re-boost its prediction performance. In this paper, we propose a new layer-wise pruning method for deep neural networks. In our proposed method, parameters of each individual layer are pruned independently bas…

2017-11-09cs.CVcs.LGcs.NE

Optimal Eye Surgeon：在初始化时通过稀疏生成器寻找图像先验

Optimal Eye Surgeon: Finding Image Priors through Sparse Generators at Initialization

Michigan State University（密歇根州立大学）；University of Michigan – Ann Arbor（密歇根大学安娜堡分校）

本文提出 OES 框架，通过在随机初始化时将深度生成网络剪枝至欠参数化水平，得到 Sparse-DIP 子网络，从而在图像恢复中有效抵抗噪声过拟合并优于彩票假说等剪枝方法。

2024-06-07cs.AIcs.CVcs.LG

结构化剪枝的统一响应几何

Unified Response Geometry for Structured Pruning

本文提出统一响应几何用于结构化剪枝，通过联合响应容量选择子集并补偿移除响应，在ImageNet上显著提升剪枝后准确率，验证了响应几何作为条件原则的有效性。

2026-09-17cs.CV

The LLM Surgeon

Imperial College London（帝国理工学院）；Qualcomm AI Research（高通人工智能研究）；University of Amsterdam（阿姆斯特丹大学）

State-of-the-art language models are becoming increasingly large in an effort to achieve the highest performance on large corpora of available textual data. However, the sheer size of the Transformer architectures makes it difficult to deploy models within computational, environmental or device-specific constraints. We explore data-driven compression of existing pretrained models as an alternative to training smaller models from scratch. To do so, we scale Kronecker-factored curvature approximations of the target loss landscape to large language models. In doing so, we can compute both the dyn…

2024-03-20cs.CLcs.LG

更多相关论文收起更多论文

SURGEON：通过动态激活稀疏性实现内存自适应的全测试时自适应

SURGEON: Memory-Adaptive Fully Test-Time Adaptation via Dynamic Activation Sparsity

Northwestern Polytechnical University（西北工业大学）；Tsinghua University（清华大学）；The Hong Kong University of Science and Technology（香港科技大学）；Beijing Jiaotong University（北京交通大学）；Harbin Engineering University（哈尔滨工程大学）

SURGEON通过动态激活稀疏性策略，在全测试时自适应中按层动态剪枝激活，以降低内存成本并保持准确性，实现内存自适应的高效FTTA，达到SOTA性能。

2025-03-26cs.CVcs.LG

使用块坐标下降法对大型语言和视觉模型进行可扩展的迭代剪枝

Scalable iterative pruning of large language and vision models using block coordinate descent

Amazon Quantum Solutions Lab（亚马逊量子解决方案实验室）；AWS Center for Quantum Computing（AWS量子计算中心）；Fidelity Center for Applied Technology（富达应用技术中心）；AI Center of Excellence, FMR LLC（FMR LLC卓越人工智能中心）

针对大型语言和视觉模型剪枝问题，提出基于块坐标下降的迭代组合脑外科医生（iCBS）方法，可扩展至超大型模型，在相同密度下性能优于现有方法，还支持质量-时间权衡与硬件加速，具量子适用性。

2024-11-26cs.LGmath.OCquant-ph

一分钱买两个：曲率感知剪枝也能保留影响函数

Two For The Price Of One: Curvature-Aware Pruning Preserves Influence Functions Too

Massachusetts Institute of Technology（麻省理工学院）；Google Research（谷歌研究院）

本文提出曲率感知剪枝优化器$\texttt{CACTUS}$，可高效保留影响函数，在模型压缩任务中比LLM-Surgeon加速超28倍，在数据选择上表现优于ALPS，结合LoGRA仍能保持梯度对齐。

2026-10-14cs.LGstat.ML

迭代最优脑外科医生：利用二阶信息实现更快的稀疏恢复

The Iterative Optimal Brain Surgeon: Faster Sparse Recovery by Leveraging Second-Order Information

Yandex Research（Yandex研究院）；Skoltech（斯科尔科沃科学技术研究院）；Institute of Science Technology Austria (ISTA)（奥地利科学技术研究所）

本文将最优脑外科医生框架的二阶曲率信息引入经典迭代稀疏恢复算法（如IHT）的投影步骤，提出具有理论保证的新算法，并在视觉和语言任务上验证了其对Transformer模型稀疏化的有效性。

2024-08-30cs.LG

Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning

IST Austria（奥地利科学技术研究所）；ETH Zurich（苏黎世联邦理工学院）；Neural Magic（神经魔法公司）

We consider the problem of model compression for deep neural networks (DNNs) in the challenging one-shot/post-training setting, in which we are given an accurate trained model, and must compress it without any retraining, based only on a small amount of calibration input data. This problem has become popular in view of the emerging software and hardware support for executing models compressed via pruning and/or quantization with speedup, and well-performing solutions have been proposed independently for both compression approaches. In this paper, we introduce a new compression framework which …

2023-01-08cs.LG

LEAP：可学习的端到端无结构剪枝大型语言模型

LEAP: Learnable End-to-End Adaptive Pruning of Large Language Models

University of Maryland（马里兰大学）

本文提出LEAP，一种可学习的端到端无结构剪枝方法，通过伯努利-戈姆贝茨松弛替代传统参数化，提高了无结构剪枝的端到端准确率，实验表明在多个LLM家族上平均提升了零样本准确率。

2026-06-07cs.AIcs.LG

SparseSSM：高效选择性结构化状态空间模型可一次性剪枝

SparseSSM: Efficient Selective Structured State Space Models Can Be Pruned in One-Shot

Tongji University（同济大学）；Westlake University（西湖大学）

本文提出无需训练的 SparseSSM 框架，通过二阶显著性与组件敏感性分析对 Mamba 等状态空间模型进行一次性剪枝，在零样本准确率无损下剪掉 50% SSM 权重。

2025-06-11cs.LG

EfficientXpert: 通过传播感知剪枝实现大语言模型的高效领域适应

EfficientXpert: Efficient Domain Adaptation for Large Language Models via Propagation-Aware Pruning

Univeristy of California, Berkeley（加州大学伯克利分校）；San Francisco State University（旧金山州立大学）

EfficientXpert通过传播感知剪枝方法，实现大语言模型在领域适应中的高效压缩，达到接近密集性能的稀疏效果，同时保持训练时间和内存效率。

2026-10-06cs.CLcs.LG

WoodFisher: Efficient Second-Order Approximation for Neural Network Compression

ETH Zurich（苏黎世联邦理工学院）；IST Austria（奥地利科学技术研究所）；Neural Magic, Inc.（Neural Magic公司）

Second-order information, in the form of Hessian- or Inverse-Hessian-vector products, is a fundamental tool for solving optimization problems. Recently, there has been significant interest in utilizing this information in the context of deep neural networks; however, relatively little is known about the quality of existing approximations in this context. Our work examines this question, identifies issues with existing approaches, and proposes a method called WoodFisher to compute a faithful and efficient estimate of the inverse Hessian. Our main application is to neural network compression, wh…

2020-11-25cs.LGstat.ML

基于凸优化的逐层训练后剪枝器，用于大型语言模型

A Convex-optimization-based Layer-wise Post-training Pruner for Large Language Models

The University of Hong Kong（香港大学）；Huawei Cloud（华为云）

本文提出首个基于凸优化的训练后剪枝器FISTAPruner，通过ℓ1范数凸优化模型和FISTA求解器，结合层内累积误差校正与并行剪枝，在多个LLM上优于现有方法。

2024-08-07cs.LGmath.OC

脉冲神经网络压缩：针对脉冲神经网络的后训练二次压缩

Spiking Brain Compression: Post-Training Second-order Compression for Spiking Neural Networks

School of Engineering Mathematics and Technology, University of Bristol, England（工程数学与技术学院，布里斯托大学，英格兰）

本文提出Spiking Brain Compression（SBC）框架，通过一次性后训练压缩提升脉冲神经网络的效率，实现比传统ANN基线更高的准确率。

2026-01-28cs.LGcs.NE

更少的权重，更多的问题：对LLM剪枝的一种实用攻击

Fewer Weights, More Problems: A Practical Attack on LLM Pruning

ETH Zurich（苏黎世联邦理工学院）

本文首次揭示了现代LLM剪枝方法可能被恶意利用的问题，通过构造看似无害的模型，在剪枝后表现出恶意行为，展示了剪枝过程中的安全风险。

2026-04-06cs.AIcs.CRcs.LG