The recent introduction of Diffusion Transformers (DiTs) has demonstrated exceptional capabilities in image generation by using a different backbone architecture, departing from traditional U-Nets and embracing the scalable nature of transformers. Despite their advanced capabilities, the wide deployment of DiTs, particularly for real-time applications, is currently hampered by considerable computational demands at the inference stage. Post-training Quantization (PTQ) has emerged as a fast and data-efficient solution that can significantly reduce computation and memory footprint by using low-bit weights and activations. However, its applicability to DiTs has not yet been explored and faces non-trivial difficulties due to the unique design of DiTs. In this paper, we propose PTQ4DiT, a specifically designed PTQ method for DiTs. We discover two primary quantization challenges inherent in DiTs, notably the presence of salient channels with extreme magnitudes and the temporal variability in distributions of salient activation over multiple timesteps. To tackle these challenges, we propose Channel-wise Salience Balancing (CSB) and Spearmen's $ρ$-guided Salience Calibration (SSC). CSB leverages the complementarity property of channel magnitudes to redistribute the extremes, alleviating quantization errors for both activations and weights. SSC extends this approach by dynamically adjusting the balanced salience to capture the temporal variations in activation. Additionally, to eliminate extra computational costs caused by PTQ4DiT during inference, we design an offline re-parameterization strategy for DiTs. Experiments demonstrate that our PTQ4DiT successfully quantizes DiTs to 8-bit precision (W8A8) while preserving comparable generation ability and further enables effective quantization to 4-bit weight precision (W4A8) for the first time.
Hardware-Friendly Static Quantization Method for Video Diffusion Transformers
Division of the Humanities and Social Sciences（人文与社会科学系）；California Institute of Technology（加州理工学院）；Device Solutions Research America（设备解决方案研究美国）；Samsung Semiconductor Inc.（三星半导体公司）
FP4DiT: Towards Effective Floating Point Quantization for Diffusion Transformers
Department of Electrical and Computer Engineering, Faculty of Engineering University of Alberta（电气与计算机工程系，工程学院，阿尔伯塔大学）；Division of Computer Science and Engineering, College of Engineering Louisiana State University（计算机科学与工程系，工程学院，路易斯安那州立大学）
QuantSparse: Comprehensively Compressing Video Diffusion Transformer with Model Quantization and Attention Sparsification
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences（人工智能安全国家重点实验室，计算技术研究所，中国科学院）；University of Chinese Academy of Sciences（中国科学院大学）；ETH Zürich（苏黎世联邦理工学院）；City College of New York, City University of New York, USA（纽约城市学院，纽约城市大学，美国）；Shanghai Jiao Tong University（上海交通大学）
LRQ-DiT: Log-Rotation Post-Training Quantization of Diffusion Transformers for Image and Video Generation
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Department of Electronic Engineering, Tsinghua University（清华大学电子工程系）；School of Engineering and Applied Sciences, Harvard University（哈佛大学工程与应用科学学院）；Department of Computer Science, City University of Hong Kong（香港城市大学计算机科学系）
Recently, transformer has achieved remarkable performance on a variety of computer vision applications. Compared with mainstream convolutional neural networks, vision transformers are often of sophisticated architectures for extracting powerful feature representations, which are more difficult to be developed on mobile devices. In this paper, we present an effective post-training quantization algorithm for reducing the memory storage and computational costs of vision transformers. Basically, the quantization task can be regarded as finding the optimal low-bit quantization intervals for weights…
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Megvii Technology（旷视科技）
In this paper, we propose a fully differentiable quantization method for vision transformer (ViT) named as Q-ViT, in which both of the quantization scales and bit-widths are learnable parameters. Specifically, based on our observation that heads in ViT display different quantization robustness, we leverage head-wise bit-width to squeeze the size of Q-ViT while preserving performance. In addition, we propose a novel technique named switchable scale to resolve the convergence problem in the joint training of quantization scales and bit-widths. In this way, Q-ViT pushes the limits of ViT quantiza…
Vision transformer emerges as a potential architecture for vision tasks. However, the intense computation and non-negligible delay hinder its application in the real world. As a widespread model compression technique, existing post-training quantization methods still cause severe performance drops. We find the main reasons lie in (1) the existing calibration metric is inaccurate in measuring the quantization influence for extremely low-bit representation, and (2) the existing quantization paradigm is unfriendly to the power-law distribution of Softmax. Based on these observations, we propose a…
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
Vision transformers have recently gained great success on various computer vision tasks; nevertheless, their high model complexity makes it challenging to deploy on resource-constrained devices. Quantization is an effective approach to reduce model complexity, and data-free quantization, which can address data privacy and security concerns during model deployment, has received widespread interest. Unfortunately, all existing methods, such as BN regularization, were designed for convolutional neural networks and cannot be applied to vision transformers with significantly different model archite…
Network quantization significantly reduces model inference complexity and has been widely used in real-world deployments. However, most existing quantization methods have been developed mainly on Convolutional Neural Networks (CNNs), and suffer severe degradation when applied to fully quantized vision transformers. In this work, we demonstrate that many of these difficulties arise because of serious inter-channel variation in LayerNorm inputs, and present, Power-of-Two Factor (PTF), a systematic method to reduce the performance degradation and inference complexity of fully quantized vision tra…
Patch-wise Mixed-Precision Quantization of Vision Transformer
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
University of Washington（华盛顿大学）；OctoML；Amazon Web Services（亚马逊云服务）；Carnegie Mellon University（卡内基梅隆大学）
Quantization is a key technique to reduce the resource requirement and improve the performance of neural network deployment. However, different hardware backends such as x86 CPU, NVIDIA GPU, ARM CPU, and accelerators may demand different implementations for quantized networks. This diversity calls for specialized post-training quantization pipelines to built for each hardware target, an engineering effort that is often too large for developers to keep up with. We tackle this problem with an automated post-training quantization framework called HAGO. HAGO provides a set of general quantization …
DopQ-ViT: Towards Distribution-Friendly and Outlier-Aware Post-Training Quantization for Vision Transformers
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Department of Computer Science, City University of Hong Kong（香港城市大学计算机科学系）
Quantization using a small number of bits shows promise for reducing latency and memory usage in deep neural networks. However, most quantization methods cannot readily handle complicated functions such as exponential and square root, and prior approaches involve complex training processes that must interact with floating-point values. This paper proposes a robust method for the full integer quantization of vision transformer networks without requiring any intermediate floating-point computations. The quantization techniques can be applied in various hardware or software implementations, inclu…
Semantic Alignment and Reinforcement for Data-Free Quantization of Vision Transformers
Institute of Artificial Intelligence, Xiamen University（厦门大学人工智能研究院）；MAC Lab, Department of Artificial Intelligence, School of Informatics, Xiamen University（厦门大学信息学院人工智能系）；Alibaba（阿里巴巴）
Intel – Artificial Intelligence Products Group (AIPG)（英特尔——人工智能产品集团（AIPG））；Technion – Israel Institute of Technology（以色列理工学院）
Convolutional neural networks require significant memory bandwidth and storage for intermediate computations, apart from substantial computing resources. Neural network quantization has significant benefits in reducing the amount of intermediate results, but it often requires the full datasets and time-consuming fine tuning to recover the accuracy lost after quantization. This paper introduces the first practical 4-bit post training quantization approach: it does not involve training the quantized model (fine-tuning), nor it requires the availability of the full dataset. We target the quantiza…
AdaLog: Post-Training Quantization for Vision Transformers with Adaptive Logarithm Quantizer
State Key Laboratory of Virtual Reality Technology and Systems, Beihang University（北京航空航天大学虚拟现实技术与系统国家重点实验室）；School of Computer Science and Engineering, Beihang University（北京航空航天大学计算机科学与工程学院）
GLF-Q: Global-Local Feature-based Quantization for Vision Transformers
State Key Laboratory of Novel Software Technology, Nanjing University（南京大学计算机软件新技术全国重点实验室）；School of Artificial Intelligence, Nanjing University（南京大学人工智能学院）；Zhongguancun Academy（中关村学院）
Not All Tasks Quantize Equally: Fisher-Guided Quantization for Visual Geometry Transformer
Department of Electronic and Computer Engineering, HKUST（香港科技大学电子与计算机工程系）；State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences（中国科学院人工智能安全国家重点实验室）；University of Chinese Academy of Sciences（中国科学院大学）；School of Data Science and Engineering, South China Normal University（华南师范大学数据科学与工程学院）
MGRQ: Post-Training Quantization For Vision Transformer With Mixed Granularity Reconstruction
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
Sensitivity-Aware Post-Training Quantization for Deep Neural Networks
South China University of Technology（华南理工大学）；Pazhou Laboratory（Pazhou实验室）；Key Laboratory of Big Data and Intelligent Robot, Ministry of Education（大数据与智能机器人重点实验室，教育部）
