Transformer-based architectures have become the de-facto standard models for a wide range of Natural Language Processing tasks. However, their memory footprint and high latency are prohibitive for efficient deployment and inference on resource-limited devices. In this work, we explore quantization for transformers. We show that transformers have unique quantization challenges -- namely, high dynamic activation ranges that are difficult to represent with a low bit fixed-point format. We establish that these activations contain structured outliers in the residual connections that encourage specific attention patterns, such as attending to the special separator token. To combat these challenges, we present three solutions based on post-training quantization and quantization-aware training, each with a different set of compromises for accuracy, model size, and ease of use. In particular, we introduce a novel quantization scheme -- per-embedding-group quantization. We demonstrate the effectiveness of our methods on the GLUE benchmark using BERT, establishing state-of-the-art results for post-training quantization. Finally, we show that transformer weights and embeddings can be quantized to ultra-low bit-widths, leading to significant memory savings with a minimum accuracy loss. Our source code is available at~\url{https://github.com/qualcomm-ai-research/transformer-quantization}.
RepQuant: Towards Accurate Post-Training Quantization of Large Transformer Models via Scale Reparameterization
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
GoQuant: Geometric Orthogonal Residual Projection for Multiplier-Free Power-of-Two Transformer Quantization
Information Systems Technology and Design（信息系统技术与设计）；Singapore University of Technology and Design（新加坡科技设计大学）；Institute of High Performance Computing (IHPC)（高性能计算研究所）；Agency for Science, Technology and Research (A*STAR)（科技研究局）
Auto-ViT-Acc: An FPGA-Aware Automatic Acceleration Framework for Vision Transformer with Mixed-Scheme Quantization
Northeastern University（东北大学）；Simon Fraser University（西蒙弗雷泽大学）；University of California, Irvine（加州大学欧文分校）；ETH Zurich（苏黎世联邦理工学院）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）
视觉Transformer（ViT）正在出现，在计算机视觉任务中具有显著提高的准确性。然而，其复杂的体系结构和巨大的计算/存储需求迫切需要新的硬件加速器的设计方法。这项工作提出了一个FPGA感知的自动ViT加速框架的基础上提出的混合方案量化。据我们所知，这是第一个基于FPGA的ViT加速框架，探索模型量化。与最先进的ViT量化工作（算法方法，只有没有硬件加速）相比，我们的量化实现了0.47%至1.36%的高的Top-1精度在相同的位宽。与32位浮点基线FPGA加速器相比，我们的加速器的帧速率提高了约5.6倍（即，56.8 FPS vs. 10.0 FPS），DeiT-base在ImageNet数据集上的准确率下降了0.71%。
RepQ-ViT: Scale Reparameterization for Post-Training Quantization of Vision Transformers
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
Boost Vision Transformer with GPU-Friendly Sparsity and Quantization
Academy for Engineering and Technology, Fudan University（复旦大学工程与应用技术研究院）；NVIDIA Corporation（英伟达公司）；School for Information Science and Technology, Fudan University（复旦大学信息科学与工程学院）
Optimising TinyML with Quantization and Distillation of Transformer and Mamba Models for Indoor Localisation on Edge Devices
School of Engineering Mathematics and Technology, University of Bristol（布里斯托大学工程数学与技术学院）；School of Natural and Computing Sciences, University of Aberdeen（阿伯丁大学自然与计算科学学院）
Apollo-Forecast: Overcoming Aliasing and Inference Speed Challenges in Language Models for Time Series Forecasting
Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University（同济大学上海智能自主系统研究院）；College of Electronics and Information Engineering, Tongji University（同济大学电子与信息工程学院）；Ant Group（蚂蚁集团）；Huazhong University of Science and Technology（华中科技大学）；Macau University of Science and Technology（澳门科技大学）
QFlash: Bridging Quantization and Memory Efficiency in Vision Transformer Attention
University of Science and Technology（科学技术大学）；Electronics and Telecommunications Research Institute（电信研究所）；Pusan National University（釜山国立大学）；Jeonbuk National University（全州国立大学）
FIMA-Q: Post-Training Quantization for Vision Transformers by Fisher Information Matrix Approximation
State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, China（虚拟现实技术与系统国家重点实验室，北京航空航天大学）；School of Computer Science and Engineering, Beihang University, Beijing, China（计算机科学与工程学院，北京航空航天大学）
TWEO: Transformers Without Extreme Outliers Enables FP8 Training And Quantization For Dummies
Nationa Key Laboratory for Novel Software Technology（新型软件技术国家实验室）；School of Artificial Intelligence（人工智能学院）；Zhongguancun Academy（中关村学院）；University of Science and Technology of China（中国科学技术大学）
AQ4SViT: An Automated Quantization Framework with Search Gating Policy for Compressing Spiking Vision Transformers
eBRAIN Lab, Division of Engineering, New York University (NYU) Abu Dhabi（eBRAIN实验室，工程学院，纽约大学（NYU）阿布扎克分校）；New York University (NYU) Abu Dhabi, United Arab Emirates (UAE)（纽约大学（NYU）阿布扎克分校，阿拉伯联合酋长国（UAE））
Quasar-ViT: Hardware-Oriented Quantization-Aware Architecture Search for Vision Transformers
Northeastern University（东北大学）；Simon Fraser University（西蒙菲莎大学）；ETH Zurich（苏黎世联邦理工学院）；University of Connecticut（康涅狄格大学）；Beijing University of Technology（北京工业大学）
