Graph learning plays a central role in many data mining and machine learning tasks, such as manifold learning, data representation and analysis, dimensionality reduction, clustering, and visualization. In this work, we propose a highly scalable, adversarial-robustness-guided graph pruning framework for learning graph topologies from data. By performing a spectral adversarial robustness evaluation, our method aims to learn sparse, undirected graphs that help the underlying algorithms resist noise and adversarial perturbations. In particular, we explicitly identify and prune edges that are most vulnerable to adversarial attacks. We use spectral clustering, one of the most representative graph-based machine learning algorithms, to evaluate the proposed framework. Compared with prior state-of-the-art graph learning approaches, the proposed method is more scalable and significantly improves both the computational efficiency and the solution quality of spectral clustering.
Pruning is a well-known mechanism for reducing the computational cost of deep convolutional networks. However, studies have shown the potential of pruning as a form of regularization, which reduces overfitting and improves generalization. We demonstrate that this family of strategies provides additional benefits beyond computational performance and generalization. Our analyses reveal that pruning structures (filters and/or layers) from convolutional networks increase not only generalization but also robustness to adversarial images (natural images with content modified). Such achievements are …
Adversarial Pruning: A Survey and Benchmark of Pruning Methods for Adversarial Robustness
University of Cagliari（卡利亚里大学）；Sapienza University of Rome（罗马大学）；University of Genova（热那亚大学）；CINI, National Cybersecurity Lab, Consorzio Interuniversitario Nazionale per l’Informatica（意大利国家跨大学信息学联盟国家网络安全实验室）
Adversarial examples provoke weak reliability and potential security issues in deep neural networks. Although adversarial training has been widely studied to improve adversarial robustness, it works in an over-parameterized regime and requires high computations and large memory budgets. To bridge adversarial robustness and model compression, we propose a novel adversarial pruning method, Masking Adversarial Damage (MAD) that employs second-order information of adversarial loss. By using it, we can accurately estimate adversarial saliency for model parameters and determine which parameters can …
Deep neural networks (DNNs) have been proven to be vulnerable to adversarial examples. A special branch of adversarial examples, namely sparse adversarial examples, can fool the target DNNs by perturbing only a few pixels. However, many existing sparse adversarial attacks use heuristic methods to select the pixels to be perturbed, and regard the pixel selection and the adversarial attack as two separate steps. From the perspective of neural network pruning, we propose a novel end-to-end sparse adversarial attack method, namely AutoAdversary, which can find the most important pixels automatical…
Tsinghua University（清华大学）；IBM Research（IBM研究院）；University of California, Los Angeles（加州大学洛杉矶分校）；SenseTime Research（商汤科技研究院）；Northeastern University（东北大学）
It is well known that deep neural networks (DNNs) are vulnerable to adversarial attacks, which are implemented by adding crafted perturbations onto benign examples. Min-max robust optimization based adversarial training can provide a notion of security against adversarial attacks. However, adversarial robustness requires a significantly larger capacity of the network than that for the natural training with only benign examples. This paper proposes a framework of concurrent adversarial training and weight pruning that enables model compression while still preserving the adversarial robustness a…
University of Chinese Academy of Sciences（中国科学院大学）；Sun Yat-sen University（中山大学）；Institute of Information Engineering, CAS（中国科学院信息工程研究所）；Tencent AI Lab（腾讯人工智能实验室）；Beihang University（北京航空航天大学）；Shenzhen Research Institute of Big Data（深圳大数据研究院）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））
Fast adversarial training (FAT) effectively improves the efficiency of standard adversarial training (SAT). However, initial FAT encounters catastrophic overfitting, i.e.,the robust accuracy against adversarial attacks suddenly and dramatically decreases. Though several FAT variants spare no effort to prevent overfitting, they sacrifice much calculation cost. In this paper, we explore the difference between the training processes of SAT and FAT and observe that the attack success rate of adversarial examples (AEs) of FAT gets worse gradually in the late training stage, resulting in overfitting…
Neural networks are vulnerable to adversarial examples, which poses a threat to their application in security sensitive systems. We propose high-level representation guided denoiser (HGD) as a defense for image classification. Standard denoiser suffers from the error amplification effect, in which small residual adversarial noise is progressively amplified and leads to wrong classifications. HGD overcomes this problem by using a loss function defined as the difference between the target model's outputs activated by the clean image and denoised image. Compared with ensemble adversarial training…
Advances in the development of adversarial attacks have been fundamental to the progress of adversarial defense research. Efficient and effective attacks are crucial for reliable evaluation of defenses, and also for developing robust models. Adversarial attacks are often generated by maximizing standard losses such as the cross-entropy loss or maximum-margin loss within a constraint set using Projected Gradient Descent (PGD). In this work, we introduce a relaxation term to the standard loss, that finds more suitable gradient-directions, increases attack efficacy and leads to more efficient adv…
Though deep neural networks perform challenging tasks excellently, they are susceptible to adversarial examples, which mislead classifiers by applying human-imperceptible perturbations on clean inputs. Under the query-free black-box scenario, adversarial examples are hard to transfer to unknown models, and several methods have been proposed with the low transferability. To settle such issue, we design a max-min framework inspired by input transformations, which are benificial to both the adversarial attack and defense. Explicitly, we decrease loss values with inputs' affline transformations as…
Deep neural networks have achieved impressive performance in many applications but their large number of parameters lead to significant computational and storage overheads. Several recent works attempt to mitigate these overheads by designing compact networks using pruning of connections. However, we observe that most of the existing strategies to design compact networks fail to preserve network robustness against adversarial examples. In this work, we rigorously study the extension of network pruning strategies to preserve both benign accuracy and robustness of a network. Starting with a form…
National Tsing-Hua University（国立清华大学）；Electronic and Optoelectronic System Research Laboratories, ITRI（工业技术研究院电子与光电系统研究实验室）；Google Research（谷歌研究院）
Adversarial robustness has emerged as an important topic in deep learning as carefully crafted attack samples can significantly disturb the performance of a model. Many recent methods have proposed to improve adversarial robustness by utilizing adversarial training or model distillation, which adds additional procedures to model training. In this paper, we propose a new training paradigm called Guided Complement Entropy (GCE) that is capable of achieving "adversarial defense for free," which involves no additional procedures in the process of improving adversarial robustness. In addition to ma…
Johns Hopkins University（约翰斯·霍普金斯大学）；Carnegie Mellon University（卡内基梅隆大学）；University of Maryland, College Park（马里兰大学帕克分校）；City University of Hong Kong（香港城市大学）
The University of Sydney（悉尼大学）；SenseTime Computer Vision Research Group（商汤科技计算机视觉研究组）
In this work, we propose a new layer-by-layer channel pruning method called Channel Pruning guided by classification Loss and feature Importance (CPLI). In contrast to the existing layer-by-layer channel pruning approaches that only consider how to reconstruct the features from the next layer, our approach additionally take the classification loss into account in the channel pruning process. We also observe that some reconstructed features will be removed at the next pruning stage. So it is unnecessary to reconstruct these features. To this end, we propose a new strategy to suppress the influe…
Deep neural network (DNN) pruning has become a de facto component for deploying on resource-constrained devices since it can reduce memory requirements and computation costs during inference. In particular, channel pruning gained more popularity due to its structured nature and direct savings on general hardware. However, most existing pruning approaches utilize importance measures that are not directly related to the task utility. Moreover, few in the literature focus on visual detection models. To fill these gaps, we propose a novel gradient-based saliency measure for visual detection and us…
With the increase of structure complexity, convolutional neural networks (CNNs) take a fair amount of computation cost. Meanwhile, existing research reveals the salient parameter redundancy in CNNs. The current pruning methods can compress CNNs with little performance drop, but when the pruning ratio increases, the accuracy loss is more serious. Moreover, some iterative pruning methods are difficult to accurately identify and delete unimportant parameters due to the accuracy drop during pruning. We propose a novel adversarial iterative pruning method (AIP) for CNNs based on knowledge transfer.…
Adversarial pruning compresses models while preserving robustness. Current methods require access to adversarial examples during pruning. This significantly hampers training efficiency. Moreover, as new adversarial attacks and training methods develop at a rapid rate, adversarial pruning methods need to be modified accordingly to keep up. In this work, we propose a novel framework to prune a previously trained robust neural network while maintaining adversarial robustness, without further generating adversarial examples. We leverage concurrent self-distillation and pruning to preserve knowledg…
Xiamen University（厦门大学）；Beihang University（北京航空航天大学）；University of Chinese Academy of Sciences（中国科学院大学）；BestImage, Tencent Technology (Shanghai) Co.,Ltd（腾讯科技（上海）有限公司贝思图）
Structured pruning of filters or neurons has received increased focus for compressing convolutional neural networks. Most existing methods rely on multi-stage optimizations in a layer-wise manner for iteratively pruning and retraining which may not be optimal and may be computation intensive. Besides, these methods are designed for pruning a specific structure, such as filter or block structures without jointly pruning heterogeneous structures. In this paper, we propose an effective structured pruning approach that jointly prunes filters as well as other structures in an end-to-end manner. To …
Back to Fundamentals: Low-Level Visual Features Guided Progressive Token Pruning
Xidian University（西安电子科技大学）；University of Michigan – Shanghai Jiao Tong University Joint Institute（密歇根大学-上海交通大学联合学院）；Shanghai Jiao Tong University（上海交通大学）；Norwegian University of Science and Technology（挪威科技大学）；Xi’an Jiaotong-Liverpool University（西交利物浦大学）
Efficient Test-time Adaptive Object Detection via Sensitivity-Guided Pruning
School of Information Science and Technology and MoE Key Laboratory of Brain-inspired Intelligent Perception and Cognition, University of Science and Technology of China（信息科学与技术学院和脑启发智能感知与认知MoE重点实验室，中国科学技术大学）
EntropyPrune: Matrix Entropy Guided Visual Token Pruning for Multimodal Large Language Models
School of Computer Science and Technology, Tongji University（同济大学计算机科学与技术学院）；University of California, Santa Cruz（加州大学圣克ruz分校）；East China Normal University（华东师范大学）；Shanghai Eye Disease Prevention and Treatment Center（上海眼病预防与治疗中心）
Intel Labs（英特尔实验室）；University of Pittsburgh（匹兹堡大学）；Duke University（杜克大学）
Phenomenally successful in practical inference problems, convolutional neural networks (CNN) are widely deployed in mobile devices, data centers, and even supercomputers. The number of parameters needed in CNNs, however, are often large and undesirable. Consequently, various methods have been developed to prune a CNN once it is trained. Nevertheless, the resulting CNNs offer limited benefits. While pruning the fully connected layers reduces a CNN's size considerably, it does not improve inference speed noticeably as the compute heavy parts lie in convolutions. Pruning CNNs in a way that increa…
ERA: Entropy-Guided Visual Token Pruning with Rectified Attention for Efficient MLLMs
School of Future Technology, Dalian University of Technology（大连理工大学未来技术学院）；S-Lab, Nanyang Technological University（南洋理工大学S-Lab）；OPPO Research Institute（OPPO研究院）；Department of Computing, The Hong Kong Polytechnic University（香港理工大学计算机系）
MADTP: Multimodal Alignment-Guided Dynamic Token Pruning for Accelerating Vision-Language Transformer
School of Information Science and Technology, Fudan University（复旦大学信息科学与技术学院）；Academy for Engineering and Technology, Fudan University（复旦大学工程技术研究院）；Tsinghua-Berkeley Shenzhen Institute, Tsinghua University（清华大学深圳国际研究生院（清华-伯克利深圳学院））
Deep Neural Networks (DNNs) have achieved remarkable success in many computer vision tasks recently, but the huge number of parameters and the high computation overhead hinder their deployments on resource-constrained edge devices. It is worth noting that channel pruning is an effective approach for compressing DNN models. A critical challenge is to determine which channels are to be removed, so that the model accuracy will not be negatively affected. In this paper, we first propose Spatial and Channel Attention (SCA), a new attention module combining both spatial and channel attention that re…
Northeastern University（东北大学）；Lawrence Livermore National Laboratory（劳伦斯利弗莫尔国家实验室）
Deep neural networks (DNNs) are sensitive to adversarial examples, resulting in fragile and unreliable performance in the real world. Although adversarial training (AT) is currently one of the most effective methodologies to robustify DNNs, it is computationally very expensive (e.g., 5-10X costlier than standard training). To address this challenge, existing approaches focus on single-step AT, referred to as Fast AT, reducing the overhead of adversarial example generation. Unfortunately, these approaches are known to fail against stronger adversaries. To make AT computationally efficient witho…
Despite the recent progress of network pruning, directly applying it to the Internet of Things (IoT) applications still faces two challenges, i.e. the distribution divergence between end and cloud data and the missing of data label on end devices. One straightforward solution is to combine the unsupervised domain adaptation (UDA) technique and pruning. For example, the model is first pruned on the cloud and then transferred from cloud to end by UDA. However, such a naive combination faces high performance degradation. Hence this work proposes an Adversarial Double Masks based Pruning (ADMP) fo…
Spectral Evolution-Guided Token Pruning in Multimodal Large Language Models
School of Software Technology, Zhejiang University（浙江大学软件学院）；Zhejiang Key Laboratory of Digital-Intelligence Service Technology（浙江省数字化服务技术重点实验室）；The University of Queensland（昆士兰大学）；Singapore Management University（新加坡管理大学）；The University of Southern Queensland（南昆士兰大学）
In safety-critical but computationally resource-constrained applications, deep learning faces two key challenges: lack of robustness against adversarial attacks and large neural network size (often millions of parameters). While the research community has extensively explored the use of robust training and network pruning independently to address one of these challenges, only a few recent works have studied them jointly. However, these works inherit a heuristic pruning strategy that was developed for benign training, which performs poorly when integrated with robust training techniques, includ…
Which Layer Causes Distribution Deviation? Entropy-Guided Adaptive Pruning for Diffusion and Flow Models
Stanford University（斯坦福大学）；North China Electric Power University（华北电力大学）；University of Technology Sydney（悉尼科技大学）；Harvard University（哈佛大学）；University of Science and Technology of China（中国科学技术大学）
Mix-and-Match Pruning: Globally Guided Layer-Wise Sparsification of DNNs
1 Brandenburg Technical University, Cottbus, Germany；2 University of Zanjan, Iran；3 Tallinn University of Technology, Estonia；4 Leibniz Institute for High Performance Microelectronics, Frankfurt Oder, Germany
FedX: Explanation-Guided Pruning for Communication-Efficient Federated Learning in Remote Sensing
Faculty of Electrical Engineering and Computer Science, Technische Universität Berlin（电气工程与计算机科学学院，柏林技术大学）；BIFOLD - Berlin Institute for the Foundations of Learning and Data（柏林学习与数据基础研究所）
Illinois Institute of Technology（伊利诺伊理工学院）；Texas State University（德克萨斯州立大学）；Shandong University（山东大学）
With the remarkable success of deep learning recently, efficient network compression algorithms are urgently demanded for releasing the potential computational power of edge devices, such as smartphones or tablets. However, optimal network pruning is a non-trivial task which mathematically is an NP-hard problem. Previous researchers explain training a pruned network as buying a lottery ticket. In this paper, we investigate the Magnitude-Based Pruning (MBP) scheme and analyze it from a novel perspective through Fourier analysis on the deep learning model to guide model designation. Besides expl…
Spectral Complex Autoencoder Pruning: A Fidelity-Guided Criterion for Extreme Structured Channel Compression
School of Computer, Jiangsu University of Science and Technology（江苏科技大学计算机学院）；Department of Electrical and Computer Engineering, University of Nevada, Las Vegas（内华达大学拉斯维加斯分校电子与计算机工程系）
World Health Organization (WHO) provides the guideline for managing the Particulate Matter (PM) level because when the PM level is higher, it threats the human health. For managing PM level, the procedure for measuring PM value is needed firstly. We use Tapered Element Oscillating Microbalance (TEOM)-based PM measuring sensors because it shows higher cost-effectiveness than Beta Attenuation Monitor (BAM)-based sensor. However, TEOM-based sensor has higher probability of malfunctioning than BAM-based sensor. In this paper, we call the overall malfunction as an anomaly, and we aim to detect anom…
Teacher-Guided One-Shot Pruning via Context-Aware Knowledge Distillation
Apurba-NSU R&D Lab, Department of Electrical and Computer Engineering, North South University, Dhaka, Bangladesh（阿普尔巴-NSU研发实验室，电气与计算机工程系，北方南大学，达卡，孟加拉国）；Apurba Technologies, Sunnyvale, CA 94085, USA（阿普尔巴技术公司，硅谷，加利福尼亚州94085，美国）
Multimodal-Guided Dynamic Dataset Pruning for Robust and Efficient Data-Centric Learning
National Key Laboratory for Novel Software Technology, Nanjing University（新型软件技术国家重点实验室，南京大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；The Chinese University of Hong Kong（香港中文大学）
