Diffusion models have shown remarkable performance in image synthesis by progressively estimating a smooth transition from a Gaussian distribution of noise to a real image. Unfortunately, their practical deployment is limited by slow inference speed, high memory usage, and the computational demands of the noise estimation process. Post-training quantization (PTQ) emerges as a promising solution to accelerate sampling and reduce memory overhead for diffusion models. Existing PTQ methods for diffusion models typically apply uniform weights to calibration samples across timesteps, which is sub-optimal since data at different timesteps may contribute differently to the diffusion process. Additionally, due to varying activation distributions and gradients across timesteps, a uniform quantization approach is sub-optimal. Each timestep requires a different gradient direction for optimal quantization, and treating them equally can lead to conflicting gradients that degrade performance. In this paper, we propose a novel PTQ method that addresses these challenges by assigning appropriate weights to calibration samples. Specifically, our approach learns to assign optimal weights to calibration samples to align the quantized model's gradients across timesteps, facilitating the quantization process. Extensive experiments on CIFAR-10, LSUN-Bedrooms, and ImageNet demonstrate the superiority of our method compared to other PTQ methods for diffusion models.
EDA-DM: Enhanced Distribution Alignment for Post-Training Quantization of Diffusion Models
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
S$^2$Q-VDiT: Accurate Quantized Video Diffusion Transformer with Salient Data and Sparse Token Distillation
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences（人工智能安全国家重点实验室，计算技术研究所，中国科学院）；University of Chinese Academy of Sciences（中国科学院大学）；ETH Zürich（苏黎世联邦理工学院）
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；ByteDance Inc.（字节跳动公司）
Post-training quantization methods use a set of calibration data to compute quantization ranges for network parameters and activations. The calibration data usually comes from the training dataset which could be inaccessible due to sensitivity of the data. In this work, we want to study such a problem: can we use out-of-domain data to calibrate the trained networks without knowledge of the original dataset? Specifically, we go beyond the domain of natural images to include drastically different domains such as X-ray images, satellite images and ultrasound images. We find cross-domain calibrati…
University of Michigan（密歇根大学）；Adobe Research（奥多比研究院）
Geometric camera calibration is often required for applications that understand the perspective of the image. We propose perspective fields as a representation that models the local perspective properties of an image. Perspective Fields contain per-pixel information about the camera view, parameterized as an up vector and a latitude value. This representation has a number of advantages as it makes minimal assumptions about the camera model and is invariant or equivariant to common image editing operations like cropping, warping, and rotation. It is also more interpretable and aligned with huma…
Convolutional neural networks are able to learn realistic image priors from numerous training samples in low-level image generation and restoration. We show that, for high-level image recognition tasks, we can further reconstruct "realistic" images of each category by leveraging intrinsic Batch Normalization (BN) statistics without any training data. Inspired by the popular VAE/GAN methods, we regard the zero-shot optimization process of synthetic images as generative modeling to match the distribution of BN statistics. The generated images serve as a calibration set for the following zero-sho…
Hardware-Friendly Static Quantization Method for Video Diffusion Transformers
Division of the Humanities and Social Sciences（人文与社会科学系）；California Institute of Technology（加州理工学院）；Device Solutions Research America（设备解决方案研究美国）；Samsung Semiconductor Inc.（三星半导体公司）
Imperial College London（伦敦帝国学院）；Arm ML Research Lab（Arm机器学习研究实验室）
Modern convolutional neural networks (CNNs) are known to be overconfident in terms of their calibration on unseen input data. That is to say, they are more confident than they are accurate. This is undesirable if the probabilities predicted are to be used for downstream decision making. When considering accuracy, CNNs are also surprisingly robust to compression techniques, such as quantization, which aim to reduce computational and memory costs. We show that this robustness can be partially explained by the calibration behavior of modern CNNs, and may be improved with overconfidence. This is d…
MergeQuant: Accurate 4-bit Static Quantization of Large Language Models by Channel-wise Calibration
State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications（北京邮电大学网络与交换技术国家重点实验室）；PengCheng Laboratory（鹏城实验室）
Saliency-Aware Regularized Quantization Calibration for Large Language Models
University of Science and Technology of China（中国科学技术大学）；University College London（伦敦大学学院）；Shanghai University of Finance and Economics（上海财经大学）；Academy of Mathematics and Systems Science, Chinese Academy of Sciences（中国科学院数学与系统科学研究院）；University of Copenhagen（哥本哈根大学）；Imperial College London（伦敦帝国学院）；Technical University of Denmark（丹麦技术大学）；Peking University（北京大学）
Habana Labs – An Intel company（Habana Labs——英特尔旗下公司）；Technion（以色列理工学院）
Lately, post-training quantization methods have gained considerable attention, as they are simple to use, and require only a small unlabeled calibration set. This small dataset cannot be used to fine-tune the model without significant over-fitting. Instead, these methods only use the calibration set to set the activations' dynamic ranges. However, such methods always resulted in significant accuracy degradation, when used below 8-bits (except on small datasets). Here we aim to break the 8-bit barrier. To this end, we minimize the quantization errors of each layer separately by optimizing its p…
DartQuant: Efficient Rotational Distribution Calibration for LLM Quantization
Nanjing University of Science and Technology（南京理工大学）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Huawei Technologies Co., Ltd.（华为技术有限公司）
Calibration and Transformation-Free Weight-Only LLMs Quantization via Dynamic Grouping
Division of Artificial Intelligence, School of Data Science, Lingnan University, Hong Kong, China（岭南大学人工智能学院）；Department of Computer Science and Engineering, The Hong Kong University of Science and Technology, Hong Kong, China（香港理工大学计算机科学与工程系）；Department of Computing, The Hong Kong Polytechnic University, Hong Kong, China（香港理工大学计算机系）；Xiaoi Robot Inc., Shanghai, China（小蚁机器人有限公司）
Coverage-Based Calibration for Post-Training Quantization via Weighted Set Cover over Outlier Channels
Department of Computer Science, Iowa State University（伊阿华州立大学计算机科学系）；Department of Civil, Construction & Environmental Engineering, Iowa State University（伊阿华州立大学土木、建设与环境工程系）
AlphaQ: Calibration-Free Bit Allocation for Mixture-of-Experts Quantization
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；ELLIS Institute Tübingen（图宾根ELLIS研究所）；Tübingen AI Center（图宾根人工智能中心）；Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Xiamen University（厦门大学多媒体可信感知与高效计算重点实验室）；International Computer Science Institute（国际计算机科学研究所）；Lawrence Berkeley National Laboratory（伯克利国家实验室）；University of California, Berkeley（加州大学伯克利分校）；Liquid AI
Houmo AI（后摩智能）；Huazhong University of Science & Technology（华中科技大学）；Tencent AI Lab（腾讯人工智能实验室）；Peking University（北京大学）
Post-training quantization (PTQ) is a popular method for compressing deep neural networks (DNNs) without modifying their original architecture or training procedures. Despite its effectiveness and convenience, the reliability of PTQ methods in the presence of some extrem cases such as distribution shift and data noise remains largely unexplored. This paper first investigates this problem on various commonly-used PTQ methods. We aim to answer several research questions related to the influence of calibration set distribution variations, calibration paradigm selection, and data augmentation or s…
Post-Training Quantization of Generative and Discriminative LSTM Text Classifiers: A Study of Calibration, Class Balance, and Robustness
School of Mathematical and Data Sciences（数学与数据科学学院）；West Virginia University（西弗吉尼亚大学）；Department of Industrial and Management Systems Engineering（工业与管理系统工程系）
