Recent works show that reducing the number of layers in a convolutional neural network can enhance efficiency while maintaining the performance of the network. Existing depth compression methods remove redundant non-linear activation functions and merge the consecutive convolution layers into a single layer. However, these methods suffer from a critical drawback; the kernel size of the merged layers becomes larger, significantly undermining the latency reduction gained from reducing the depth of the network. We show that this problem can be addressed by jointly pruning convolution layers and activation functions. To this end, we propose LayerMerge, a novel depth compression method that selects which activation layers and convolution layers to remove, to achieve a desired inference speed-up while minimizing performance loss. Since the corresponding selection problem involves an exponential search space, we formulate a novel surrogate optimization problem and efficiently solve it via dynamic programming. Empirical results demonstrate that our method consistently outperforms existing depth compression and layer pruning methods on various network architectures, both on image classification and generation tasks. We release the code at https://github.com/snu-mllab/LayerMerge.
Pruning Deep Neural Networks (DNNs) is a prominent field of study in the goal of inference runtime acceleration. In this paper, we introduce a novel data-free pruning protocol RED++. Only requiring a trained neural network, and not specific to DNN architecture, we exploit an adaptive data-free scalar hashing which exhibits redundancies among neuron weight values. We study the theoretical and empirical guarantees on the preservation of the accuracy from the hashing as well as the expected pruning ratio resulting from the exploitation of said redundancies. We propose a novel data-free pruning te…
CAMP-HiVe: Cyclic Pair Merging based Efficient DNN Pruning with Hessian-Vector Approximation for Resource-Constrained Systems
Engineering University of Louisville（工程大学路易斯维尔大学）；School of Computing University of Georgia（计算学院佐治亚大学）；Department of Computer Science（1 计算机科学系）；Engineering, University of Louisville, KY, USA（工程，路易斯维尔大学，美国肯塔基州）；School of Computing, University of Georgia, Athens, GA, USA（2 计算学院，佐治亚大学，美国亚特兰大州）
Evolutionary Negative Module Pruning for Better LoRA Merging
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；School of Software Technology, Zhejiang University（浙江大学软件学院）；State Key Laboratory of Blockchain and Security, Zhejiang University（浙江大学区块链与安全国家重点实验室）；Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security（杭州高新技术区（滨江）区块链与数据安全研究院）
Mila – Quebec AI Institute（Mila – 魁北克人工智能研究所）；Polytechnique Montréal（蒙特利尔综合理工学院）；Université de Montréal（蒙特利尔大学）；McGill University（麦吉尔大学）；AI Center, Samsung, South Korea（三星人工智能中心，韩国）；Samsung AI Lab, Montreal（三星人工智能实验室，蒙特利尔）
Resource-efficient convolution neural networks enable not only the intelligence on edge devices but also opportunities in system-level optimization such as scheduling. In this work, we aim to improve the performance of resource-constrained filter pruning by merging two sub-problems commonly considered, i.e., (i) how many filters to prune for each layer and (ii) which filters to prune given a per-layer pruning budget, into a global filter ranking problem. Our framework entails a novel algorithm, dubbed layer-compensated pruning, where meta-learning is involved to determine better solutions. We …
ETH Zürich（苏黎世联邦理工学院）；Beihang University（北京航空航天大学）；Singapore Management University（新加坡管理大学）
Many mobile applications demand selective execution of multiple correlated deep learning inference tasks on resource-constrained platforms. Given a set of deep neural networks, each pre-trained for a single task, it is desired that executing arbitrary combinations of tasks yields minimal computation cost. Pruning each network separately yields suboptimal computation cost due to task relatedness. A promising remedy is to merge the networks into a multitask network to eliminate redundancy across tasks before network pruning. However, pruning a multitask network combined by existing network mergi…
Westlake University（西湖大学）；Westlake Institute for Advanced Study（西湖高等研究院）；Zhejiang University（浙江大学）
Ki67 is a significant biomarker in the diagnosis and prognosis of cancer, whose index can be evaluated by quantifying its expression in Ki67 immunohistochemistry (IHC) stained images. However, quantitative analysis on multi-source Ki67 images is yet a challenging task in practice due to cross-domain distribution differences, which result from imaging variation, staining styles, and lesion types. Many recent studies have made some efforts on domain generalization (DG), whereas there are still some noteworthy limitations. Specifically in the case of Ki67 images, learning invariant representation…
Can Visual Input Be Compressed? A Visual Token Compression Benchmark for Large Multimodal Models
Shandong University（山东大学）；Shenzhen University of Information Technology（深圳信息职业技术大学）；Tongji University（同济大学）；University of Science and Technology of China（中国科学技术大学）；Sichuan University（四川大学）；University of Notre Dame（圣母大学）；Ludwig Maximilian University of Munich（慕尼黑路德维希-马克西米利安大学）；EB Tech Co., Ltd.（EB科技有限公司）
A Novel Hierarchical Integration Method for Efficient Model Merging in Medical LLMs
Deakin Cyber Research（德肯网络研究）；Innovation Centre Deakin University / Universal Higher Education Melbourne, Australia（德肯大学创新中心/通用高等教育墨尔本，澳大利亚）；School of IT（信息科技学院）；Engineering Melbourne Institute of Technology Melbourne, Australia（工程墨尔本技术学院墨尔本，澳大利亚）；Innovation Centre Deakin University Geelong, Australia（德肯大学创新中心吉朗，澳大利亚）
DOTResize: Reducing LLM Width via Discrete Optimal Transport-based Neuron Merging
Center for Language and Speech Processing, Johns Hopkins University, Baltimore, MD, USA（语言与语音处理中心，约翰霍普金斯大学，巴尔的摩，MD，美国）；Human Language Technology Center of Excellence, Johns Hopkins University, Baltimore, MD, USA（人工智能语言技术卓越中心，约翰霍普金斯大学，巴尔的摩，MD，美国）
Layer as Puzzle Pieces: Compressing Large Language Models through Layer Concatenation
South China University of Technology（华南理工大学）；JD Explore Academy（JD探索学院）；Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区）；University of Sydney（悉尼大学）；Pazhou Lab（琶洲实验室）
Merging Smarter, Generalizing Better: Enhancing Model Merging on OOD Data
College of Artificial Intelligence, Jilin University（吉林大学人工智能学院）；Rensselaer Polytechnic Institute（拉特盖尔理工学院）；College of applied Science, Taiyuan University of Science and Technology（太原科技大学应用科学学院）；CSIRO’s Data61（CSIRO数据61）
Is Retraining-Free Enough? The Necessity of Router Calibration for Efficient MoE Compression
Department of Electrical and Computer Engineering, Seoul National University, Seoul, South Korea（电气与计算机工程系，首尔国立大学，首尔，韩国）；Interdisciplinary Program in Artificial Intelligence, Seoul National University, Seoul, South Korea（人工智能跨学科项目，首尔国立大学，首尔，韩国）
Come Together, But Not Right Now: A Progressive Strategy to Boost Low-Rank Adaptation
Southern University of Science（南方科技大学）；City University of Hong Kong（香港城市大学）；Zhejiang University（浙江大学）；Hong Kong University of Science（香港科学大学）；University of Surrey（萨里大学）；New York University（纽约大学）
