QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Models
ZIP Lab, Monash University（莫纳什大学ZIP实验室）；SenseTime Research（商汤科技研究院）；Beihang University（北京航空航天大学）；School of Computer and Communication Sciences, EPFL（洛桑联邦理工学院计算机与通信科学学院）；University of Science and Technology Beijing（北京科技大学）
Large Language Models (LLMs) excel in NLP, but their demands hinder their widespread deployment. While Quantization-Aware Training (QAT) offers a solution, its extensive training costs make Post-Training Quantization (PTQ) a more practical approach for LLMs. In existing studies, activation outliers in particular channels are identified as the bottleneck to PTQ accuracy. They propose to transform the magnitudes from activations to weights, which however offers limited alleviation or suffers from unstable gradients, resulting in a severe performance drop at low-bitwidth. In this paper, we propose QLLM, an accurate and efficient low-bitwidth PTQ method designed for LLMs. QLLM introduces an adaptive channel reassembly technique that reallocates the magnitude of outliers to other channels, thereby mitigating their impact on the quantization range. This is achieved by channel disassembly and channel assembly, which first breaks down the outlier channels into several sub-channels to ensure a more balanced distribution of activation magnitudes. Then similar channels are merged to maintain the original channel number for efficiency. Additionally, an adaptive strategy is designed to autonomously determine the optimal number of sub-channels for channel disassembly. To further compensate for the performance loss caused by quantization, we propose an efficient tuning method that only learns a small number of low-rank weights while freezing the pre-trained quantized model. After training, these low-rank parameters can be fused into the frozen weights without affecting inference. Extensive experiments on LLaMA-1 and LLaMA-2 show that QLLM can obtain accurate quantized models efficiently. For example, QLLM quantizes the 4-bit LLaMA-2-70B within 10 hours on a single A100-80G GPU, outperforming the previous state-of-the-art method by 7.89% on the average accuracy across five zero-shot tasks.
Hong Kong University of Science and Technology（香港科技大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Jockey Club Institute for Advanced Study, HKUST（香港科技大学赛马会高等研究院）；Reality Labs, Meta Inc（Meta公司现实实验室）；ACCESS - AI Chip Center for Emerging Smart Systems（新兴智能系统AI芯片中心（ACCESS））；Carnegie Mellon University（卡内基梅隆大学）
In order to deploy deep models in a computationally efficient manner, model quantization approaches have been frequently used. In addition, as new hardware that supports mixed bitwidth arithmetic operations, recent research on mixed precision quantization (MPQ) begins to fully leverage the capacity of representation by searching optimized bitwidths for different layers and modules in a network. However, previous studies mainly search the MPQ strategy in a costly scheme using reinforcement learning, neural architecture search, etc., or simply utilize partial prior knowledge for bitwidth assignm…
Meta Reality Lab Research（Meta现实实验室研究部门）；AGH UST（AGH科技大学）
Deploying Deep Neural Networks in low-power embedded devices for real time-constrained applications requires optimization of memory and computational complexity of the networks, usually by quantizing the weights. Most of the existing works employ linear quantization which causes considerable degradation in accuracy for weight bit widths lower than 8. Since the distribution of weights is usually non-uniform (with most weights concentrated around zero), other methods, such as logarithmic quantization, are more suitable as they are able to preserve the shape of the weight distribution more precis…
Fully quantized training (FQT), which uses low-bitwidth hardware by quantizing the activations, weights, and gradients of a neural network model, is a promising approach to accelerate the training of deep neural networks. One major challenge with FQT is the lack of theoretical understanding, in particular of how gradient quantization impacts convergence properties. In this paper, we address this problem by presenting a statistical framework for analyzing FQT algorithms. We view the quantized gradient of FQT as a stochastic estimator of its full precision counterpart, a procedure known as quant…
Mitigating the Impact of Outlier Channels for Language Model Quantization with Activation Regularization
Massachusetts Institute of Technology（麻省理工学院）；MIT-IBM Watson AI Lab（MIT-IBM沃森人工智能实验室）；IBM Research（IBM研究院）；IST Austria（奥地利科学技术研究所）；Neural Magic, Inc.（Neural Magic公司）
The Chinese University of Hong Kong（香港中文大学）；SenseBrain, Ltd（SenseBrain有限公司）；Shanghai AI Lab（上海人工智能实验室）；The University of Hong Kong（香港大学）
Model quantization is challenging due to many tedious hyper-parameters such as precision (bitwidth), dynamic range (minimum and maximum discrete values) and stepsize (interval between discrete values). Unlike prior arts that carefully tune these values, we present a fully differentiable approach to learn all of them, named Differentiable Dynamic Quantization (DDQ), which has several benefits. (1) DDQ is able to quantize challenging lightweight architectures like MobileNets, where different layers prefer different quantization parameters. (2) DDQ is hardware-friendly and can be easily implement…
APTQ: Attention-aware Post-Training Mixed-Precision Quantization for Large Language Models
School of Microelectronics, Southern University of Science and Technology（南方科技大学微电子学院）；Department of Electrical and Electronic Engineering, University of Hong Kong（香港大学电机电子工程学系）
SenseTime Research（商汤科技研究院）；University of Electronic Science and Technology of China（电子科技大学）
We study the challenging task of neural network quantization without end-to-end retraining, called Post-training Quantization (PTQ). PTQ usually requires a small subset of training data but produces less powerful quantized models than Quantization-Aware Training (QAT). In this work, we propose a novel PTQ framework, dubbed BRECQ, which pushes the limits of bitwidth in PTQ down to INT2 for the first time. BRECQ leverages the basic building blocks in neural networks and reconstructs them one-by-one. In a comprehensive theoretical study of the second-order error, we show that BRECQ achieves a goo…
Why Should the Server Do It All?: A Scalable, Versatile, and Model-Agnostic Framework for Server-Light DNN Inference over Massively Distributed Clients via Training-Free Intermediate Feature Compression
Department of Artificial Intelligence, Kyungpook National University（人工智能系，庆北国立大学）；Department of Electrical and Computer Engineering, Queen’s University（电气与计算机工程系，皇后大学）；Department of Information and Communications Engineering, Pukyong National University（信息与通信工程系，浦原国立大学）
