QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving
MIT（麻省理工学院）
Yujun Lin, Haotian Tang, Shang Yang, Zhekai Zhang, Guangxuan Xiao, Chuang Gan, Song Han
备注：
The first three authors contribute equally to this project and are listed in the alphabetical order. Yujun Lin leads the quantization algorithm, Haotian Tang and Shang Yang lead the GPU kernels and the serving system. Code is available at https://github.com/mit-han-lab/omniserve
Quantization can accelerate large language model (LLM) inference. Going beyond INT8 quantization, the research community is actively exploring even lower precision, such as INT4. Nonetheless, state-of-the-art INT4 quantization techniques only accelerate low-batch, edge LLM inference, failing to deliver performance gains in large-batch, cloud-based LLM serving. We uncover a critical issue: existing INT4 quantization methods suffer from significant runtime overhead (20-90%) when dequantizing either weights or partial sums on GPUs. To address this challenge, we introduce QoQ, a W4A8KV4 quantization algorithm with 4-bit weight, 8-bit activation, and 4-bit KV cache. QoQ stands for quattuor-octo-quattuor, which represents 4-8-4 in Latin. QoQ is implemented by the QServe inference library that achieves measured speedup. The key insight driving QServe is that the efficiency of LLM serving on GPUs is critically influenced by operations on low-throughput CUDA cores. Building upon this insight, in QoQ algorithm, we introduce progressive quantization that can allow low dequantization overhead in W4A8 GEMM. Additionally, we develop SmoothAttention to effectively mitigate the accuracy degradation incurred by 4-bit KV quantization. In the QServe system, we perform compute-aware weight reordering and take advantage of register-level parallelism to reduce dequantization latency. We also make fused attention memory-bound, harnessing the performance gain brought by KV4 quantization. As a result, QServe improves the maximum achievable serving throughput of Llama-3-8B by 1.2x on A100, 1.4x on L40S; and Qwen1.5-72B by 2.4x on A100, 3.5x on L40S, compared to TensorRT-LLM. Remarkably, QServe on L40S GPU can achieve even higher throughput than TensorRT-LLM on A100. Thus, QServe effectively reduces the dollar cost of LLM serving by 3x. Code is available at https://github.com/mit-han-lab/omniserve.
Reasoning Language Model Inference Serving Unveiled: An Empirical Study
The Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））；Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；HKBU（香港 Baptist University）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；The Hong Kong University of Science and Technology（香港科学与技术大学）
Doctoral School of Applied Informatics and Applied Mathematics, Obuda University（应用信息与应用数学博士学院，奥布达大学）；John von Neumann Faculty of Informatics, Obuda University（约翰·冯·诺伊曼信息学院，奥布达大学）
Trainable Smooth-Rotation Transforms with Learned Channel Scales for LLM Quantization
Doctoral School of Applied Informatics and Applied Mathematics, Obuda University（应用信息学与应用数学博士学校，奥布达大学）；John von Neumann Faculty of Informatics, Obuda University（约翰·冯·诺伊曼信息学系，奥布达大学）
STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantization
City University of Hong Kong（香港城市大学）；Harvard University（哈佛大学）；Zhejiang University（浙江大学）；NLPR & MAIS, Institute of Automation, CAS（中国科学院自动化研究所模式识别国家重点实验室与多模态人工智能系统全国重点实验室）；The Chinese University of Hong Kong（香港中文大学）；Tsinghua University（清华大学）
SWAN: Sparse Winnowed Attention for Reduced Inference Memory via Decompression-Free KV-Cache Compression
Centre for Responsible AI（负责任人工智能中心）；Indian Institute of Technology Madras（印度理工学院马德拉斯分校）；Department of Electrical Engineering（电气工程系）；Wadhwani School of Data Science and Artificial Intelligence（Wadhwani数据科学与人工智能学院）
