Diffusion models have achieved great success in image synthesis through iterative noise estimation using deep neural networks. However, the slow inference, high memory consumption, and computation intensity of the noise estimation model hinder the efficient adoption of diffusion models. Although post-training quantization (PTQ) is considered a go-to compression method for other tasks, it does not work out-of-the-box on diffusion models. We propose a novel PTQ method specifically tailored towards the unique multi-timestep pipeline and model architecture of the diffusion models, which compresses the noise estimation network to accelerate the generation process. We identify the key difficulty of diffusion model quantization as the changing output distributions of noise estimation networks over multiple time steps and the bimodal activation distribution of the shortcut layers within the noise estimation network. We tackle these challenges with timestep-aware calibration and split shortcut quantization in this work. Experimental results show that our proposed method is able to quantize full-precision unconditional diffusion models into 4-bit while maintaining comparable performance (small FID change of at most 2.34 compared to >100 for traditional PTQ) in a training-free manner. Our approach can also be applied to text-guided image generation, where we can run stable diffusion in 4-bit weights with high generation quality for the first time.
FP4DiT: Towards Effective Floating Point Quantization for Diffusion Transformers
Department of Electrical and Computer Engineering, Faculty of Engineering University of Alberta(电气与计算机工程系,工程学院,阿尔伯塔大学);Division of Computer Science and Engineering, College of Engineering Louisiana State University(计算机科学与工程系,工程学院,路易斯安那州立大学)
EDA-DM: Enhanced Distribution Alignment for Post-Training Quantization of Diffusion Models
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)
Q-Drift: Quantization-Aware Drift Correction for Diffusion Model Sampling
Department of Computer Science and Engineering, Seoul National University, Seoul, South Korea(计算机科学与工程系,首尔国立大学,韩国首尔);School of Computer and Communication Sciences, EPFL, Lausanne, Switzerland(计算机与通信科学学院,瑞士洛桑联邦理工学院);Meta Reality Labs, Redmond, USA(Meta现实实验室,美国西雅图)
Post-Training Quantization via Residual Truncation and Zero Suppression for Diffusion Models
Department of Artificial Intelligence(人工智能系);Kyung Hee University(庆尚大学);Department of Electrical Engineering(电气工程系);Department of Computer Science(计算机科学系)
LRQ-DiT: Log-Rotation Post-Training Quantization of Diffusion Transformers for Image and Video Generation
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Department of Electronic Engineering, Tsinghua University(清华大学电子工程系);School of Engineering and Applied Sciences, Harvard University(哈佛大学工程与应用科学学院);Department of Computer Science, City University of Hong Kong(香港城市大学计算机科学系)
Modulated Diffusion: Accelerating Generative Modeling with Modulated Quantization
Department of Computer Science, North Carolina State University(北卡罗来纳州立大学计算机科学系);National Center for Computational Science, Oak Ridge National Lab(橡树岭国家实验室计算科学中心);Department of Mechanical Engineering, Keio University(庆应大学机械工程系)
Temporal Feature Matters: A Framework for Diffusion Model Quantization
Department of Electrical and Computer Engineering, the Hong Kong University of Science and Technology(香港科技大学电子与计算机工程系);State Key Laboratory of Complex & Critical Software Environment, Beihang University(北京航空航天大学复杂与关键软件环境国家重点实验室);Department of Information Technology, Monash University(墨尔本大学信息科技系);Department of Electrical Engineering, Yale University(耶鲁大学电子工程系)
Tsinghua University(清华大学);University of California Santa Barbara(加利福尼亚大学圣巴巴拉分校);Microsoft(微软公司);Infinigence AI(智源人工智能);Shanghai Jiaotong University(上海交通大学)