Quantization using a small number of bits shows promise for reducing latency and memory usage in deep neural networks. However, most quantization methods cannot readily handle complicated functions such as exponential and square root, and prior approaches involve complex training processes that must interact with floating-point values. This paper proposes a robust method for the full integer quantization of vision transformer networks without requiring any intermediate floating-point computations. The quantization techniques can be applied in various hardware or software implementations, including processor/memory architectures and FPGAs.
We propose the position-based scaled gradient (PSG) that scales the gradient depending on the position of a weight vector to make it more compression-friendly. First, we theoretically show that applying PSG to the standard gradient descent (GD), which is called PSGD, is equivalent to the GD in the warped weight space, a space made by warping the original weight space via an appropriately designed invertible function. Second, we empirically show that PSG acting as a regularizer to a weight vector is favorable for model compression domains such as quantization and pruning. PSG reduces the gap be…
Recently, transformer has achieved remarkable performance on a variety of computer vision applications. Compared with mainstream convolutional neural networks, vision transformers are often of sophisticated architectures for extracting powerful feature representations, which are more difficult to be developed on mobile devices. In this paper, we present an effective post-training quantization algorithm for reducing the memory storage and computational costs of vision transformers. Basically, the quantization task can be regarded as finding the optimal low-bit quantization intervals for weights…
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Megvii Technology（旷视科技）
In this paper, we propose a fully differentiable quantization method for vision transformer (ViT) named as Q-ViT, in which both of the quantization scales and bit-widths are learnable parameters. Specifically, based on our observation that heads in ViT display different quantization robustness, we leverage head-wise bit-width to squeeze the size of Q-ViT while preserving performance. In addition, we propose a novel technique named switchable scale to resolve the convergence problem in the joint training of quantization scales and bit-widths. In this way, Q-ViT pushes the limits of ViT quantiza…
Northeastern University（东北大学）；Simon Fraser University（西蒙弗雷泽大学）；University of California, Irvine（加州大学欧文分校）；ETH Zurich（苏黎世联邦理工学院）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）
Vision transformers (ViTs) are emerging with significantly improved accuracy in computer vision tasks. However, their complex architecture and enormous computation/storage demand impose urgent needs for new hardware accelerator design methodology. This work proposes an FPGA-aware automatic ViT acceleration framework based on the proposed mixed-scheme quantization. To the best of our knowledge, this is the first FPGA-based ViT acceleration framework exploring model quantization. Compared with state-of-the-art ViT quantization work (algorithmic approach only without hardware acceleration), our q…
Vision transformer emerges as a potential architecture for vision tasks. However, the intense computation and non-negligible delay hinder its application in the real world. As a widespread model compression technique, existing post-training quantization methods still cause severe performance drops. We find the main reasons lie in (1) the existing calibration metric is inaccurate in measuring the quantization influence for extremely low-bit representation, and (2) the existing quantization paradigm is unfriendly to the power-law distribution of Softmax. Based on these observations, we propose a…
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
Vision transformers have recently gained great success on various computer vision tasks; nevertheless, their high model complexity makes it challenging to deploy on resource-constrained devices. Quantization is an effective approach to reduce model complexity, and data-free quantization, which can address data privacy and security concerns during model deployment, has received widespread interest. Unfortunately, all existing methods, such as BN regularization, were designed for convolutional neural networks and cannot be applied to vision transformers with significantly different model archite…
Network quantization significantly reduces model inference complexity and has been widely used in real-world deployments. However, most existing quantization methods have been developed mainly on Convolutional Neural Networks (CNNs), and suffer severe degradation when applied to fully quantized vision transformers. In this work, we demonstrate that many of these difficulties arise because of serious inter-channel variation in LayerNorm inputs, and present, Power-of-Two Factor (PTF), a systematic method to reduce the performance degradation and inference complexity of fully quantized vision tra…
Patch-wise Mixed-Precision Quantization of Vision Transformer
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
Mixed Non-linear Quantization for Vision Transformers
Chungnam National University（Chungnam国立大学）；Electronics and Telecommunications Research Institute（电子电信研究所）；Korea Electronics Technology Institute（韩国电子技术研究所）
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；Arm Inc.（Arm公司）
When considering post-training quantization, prior work has typically focused on developing a mixed precision scheme or learning the best way to partition a network for quantization. In our work, CPT-V, we look at a general way to improve the accuracy of networks that have already been quantized, simply by perturbing the quantization scales. Borrowing the idea of contrastive loss from self-supervised learning, we find a robust way to jointly minimize a loss function using just 1,000 calibration images. In order to determine the best performing quantization scale, CPT-V contrasts the features o…
DopQ-ViT: Towards Distribution-Friendly and Outlier-Aware Post-Training Quantization for Vision Transformers
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Department of Computer Science, City University of Hong Kong（香港城市大学计算机科学系）
RepQ-ViT: Scale Reparameterization for Post-Training Quantization of Vision Transformers
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
FIMA-Q: Post-Training Quantization for Vision Transformers by Fisher Information Matrix Approximation
State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, China（虚拟现实技术与系统国家重点实验室，北京航空航天大学）；School of Computer Science and Engineering, Beihang University, Beijing, China（计算机科学与工程学院，北京航空航天大学）
Semantic Alignment and Reinforcement for Data-Free Quantization of Vision Transformers
Institute of Artificial Intelligence, Xiamen University（厦门大学人工智能研究院）；MAC Lab, Department of Artificial Intelligence, School of Informatics, Xiamen University（厦门大学信息学院人工智能系）；Alibaba（阿里巴巴）
GLF-Q: Global-Local Feature-based Quantization for Vision Transformers
State Key Laboratory of Novel Software Technology, Nanjing University（南京大学计算机软件新技术全国重点实验室）；School of Artificial Intelligence, Nanjing University（南京大学人工智能学院）；Zhongguancun Academy（中关村学院）
ScalePredictor: Instance-aware Scale Learning for Accurate Quantization of Vision Transformers
School of Electronics and Communication Engineering, Shenzhen Campus, Sun Yat-sen University（中山大学深圳校区电子与通信工程学院）；Computer Science and Software Engineering, The University of Western Australia（西澳大学计算机科学与软件工程学院）；University of Oxford（牛津大学）
Scaled dot-product attention applies a softmax function on the scaled dot-product of queries and keys to calculate weights and then multiplies the weights and values. In this work, we study how to improve the learning of scaled dot-product attention to improve the accuracy of DETR. Our method is based on the following observations: using ground truth foreground-background mask (GT Fg-Bg Mask) as additional cues in the weights/values learning enables learning much better weights/values; with better weights/values, better values/weights can be learned. We propose a triple-attention module in whi…
MGRQ: Post-Training Quantization For Vision Transformer With Mixed Granularity Reconstruction
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
TWEO: Transformers Without Extreme Outliers Enables FP8 Training And Quantization For Dummies
Nationa Key Laboratory for Novel Software Technology（新型软件技术国家实验室）；School of Artificial Intelligence（人工智能学院）；Zhongguancun Academy（中关村学院）；University of Science and Technology of China（中国科学技术大学）
DFQ-ViT: Data-Free Quantization for Vision Transformers without Fine-tuning
School of Computer Science and Artificial Intelligence, Wuhan University of Technology, Hubei Key Laboratory of Transportation Internet of Things（计算机科学与人工智能学院，武汉理工大学，交通物联网重点实验室）；NEC Corporation（日本日电株式会社）；State Key Laboratory of Internet of Things for Smart City, University of Macau（智慧城市物联网国家重点实验室，澳门大学）
Boost Vision Transformer with GPU-Friendly Sparsity and Quantization
Academy for Engineering and Technology, Fudan University（复旦大学工程与应用技术研究院）；NVIDIA Corporation（英伟达公司）；School for Information Science and Technology, Fudan University（复旦大学信息科学与工程学院）
I-ViT: Integer-only Quantization for Efficient Vision Transformer Inference
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
Quasar-ViT: Hardware-Oriented Quantization-Aware Architecture Search for Vision Transformers
Northeastern University（东北大学）；Simon Fraser University（西蒙菲莎大学）；ETH Zurich（苏黎世联邦理工学院）；University of Connecticut（康涅狄格大学）；Beijing University of Technology（北京工业大学）
