As Vision Transformers (ViTs) increasingly set new benchmarks in computer vision, their practical deployment on inference engines is often hindered by their significant memory bandwidth and (on-chip) memory footprint requirements. This paper addresses this memory limitation by introducing an activation-aware model compression methodology that uses selective low-rank weight tensor approximations of different layers to reduce the parameter count of ViTs. The key idea is to decompose the weight tensors into a sum of two parameter-efficient tensors while minimizing the error between the product of the input activations with the original weight tensor and the product of the input activations with the approximate tensor sum. This approximation is further refined by adopting an efficient layer-wise error compensation technique that uses the gradient of the layer's output loss. The combination of these techniques achieves excellent results while it avoids being trapped in a shallow local minimum early in the optimization process and strikes a good balance between the model compression and output accuracy. Notably, the presented method significantly reduces the parameter count of DeiT-B by 60% with less than 1% accuracy drop on the ImageNet dataset, overcoming the usual accuracy degradation seen in low-rank approximations. In addition to this, the presented compression technique can compress large DeiT/ViT models to have about the same model size as smaller DeiT/ViT variants while yielding up to 1.8% accuracy gain. These results highlight the efficacy of our approach, presenting a viable solution for embedding ViTs in memory-constrained environments without compromising their performance.
Communication Efficient Split Learning of ViTs with Attention-based Double Compression
Department of Computer, Control, and Management Engineering (DIAG)（计算机、控制与管理工程系）；Department of Information Engineering, Electronics, and Telecommunications (DIET)（信息工程、电子与电信系）；Consorzio Nazionale Interuniversitario per le Telecomunicazioni (CNIT)（全国大学电信联合体）
PYRA: Parallel Yielding Re-Activation for Training-Inference Efficient Task Adaptation
School of Software, Tsinghua University（清华大学软件学院）；BNRist, Tsinghua University（清华大学北京信息科学与技术国家研究中心）；Department of Automation, Tsinghua University（清华大学自动化系）
Accelerating Parallel Diffusion Model Serving with Residual Compression
Southern University of Science and Technology（南方科技大学）；Jiangnan University（江南大学）；Shenzhen Technology University（深圳技术大学）；The Chinese University of Hong Kong（香港中文大学）
HAMP-LIC: Hessian-Aware Mixed-Precision Post-Training Quantization for Learned Image Compression
Beijing Institute of Computer Technology and Application（北京计算机技术及应用研究所）；School of Elite Engineering, Northwestern Polytechnical University（西北工业大学精英工程学院）
A Convolutional Layer Activation Dimensionality Reduction for Out-of-Distribution and Adversarial Attack Detection Methods
Alma Mater Studiorum Università di Bologna（博洛尼亚大学）；Advanced Research Center on Electronic Systems “Ercole De Castro” (ARCES) - Alma Mater Studiorum Università di Bologna（博洛尼亚大学“埃尔科莱·德·卡斯特罗”电子系统高级研究中心）
Mixed-UNet: Refined Class Activation Mapping for Weakly-Supervised Semantic Segmentation with Multi-scale Inference
Shandong University（山东大学）；Institute of Tsinghua-Berkeley Shenzhen, Tsinghua Shenzhen International Graduate School（清华大学深圳国际研究生院清华-伯克利深圳学院）；Hangzhou Dianzi University（杭州电子科技大学）；Shenzhen ZNV Technology Co., Ltd（深圳中兴网信科技有限公司）；Shenzhen Graduate School of Harbin Institute of Technology（哈尔滨工业大学深圳研究生院）；Institute of Biopharmaceutical and Health Engineering, Tsinghua Shenzhen International Graduate School（清华大学深圳国际研究生院生物医药与健康工程研究院）
SizeGS: Size-aware Compression of 3D Gaussian Splatting via Mixed Integer Programming
SIGS, Tsinghua University（清华大学SIGS实验室）；Harbin Institute of Technology（哈尔滨工业大学）；MMLab, The Chinese University of Hong Kong（香港中文大学MMLab）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Jiangxing Intelligence Inc.（江行智能有限公司）
