The demand for inference on extremely large scale LLMs has seen enormous growth in the recent months. It made evident the colossal shortage of dedicated hardware capable of efficient and fast processing of the involved compute and memory movement. The problem is aggravated by the exploding raise in the lengths of the sequences being processed, since those require efficient on-chip storage of the KV-cache of size proportional to the sequence length. To make the required compute feasible and fit the involved data into available memory, numerous quantization techniques have been proposed that allow accurate quantization for both weights and activations. One of the main recent breakthroughs in this direction was introduction of the family of Block Floating Point (BFP) formats characterized by a block of mantissas with a shared scale factor. These enable memory- power-, and compute- efficient hardware support of the tensor operations and provide extremely good quantization accuracy. The main issues preventing widespread application of block formats is caused by the presence of outliers in weights and activations since those affect the accuracy of the other values in the same block. In this paper, we focus on the most critical problem of limited KV-cache storage. We propose a novel approach enabling usage of low precision BFP formats without compromising the resulting model accuracy. We exploit the common channel-wise patterns exhibited by the outliers to rearrange them in such a way, that their quantization quality is significantly improved. The methodology yields 2x savings in the memory footprint without significant degradation of the model's accuracy. Importantly, the rearrangement of channels happens at the compile time and thus has no impact on the inference latency.
OffQ: Taming Structured Outliers in LLM Quantization by Offsetting
School of Computer and Communication Sciences, EPFL, Switzerland（瑞士联邦理工学院计算机与通信科学学院）；Huawei, Switzerland（华为公司）；Swiss Data Science Center, ETHZ & EPFL, Switzerland（瑞士数据科学中心，苏黎世联邦理工学院与联邦理工学院）
The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences（认知与决策智能复杂系统重点实验室，自动化研究所，中国科学院）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）；AMD；Ritzz-AI
Astro: Activation-guided Structured Regularization for Outlier-Robust LLM Post-Training Quantization
Beijing Institute of Technology（北京理工大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Hebei Province Key Laboratory of Big Data Science and Intelligent Technology（河北省大数据科学与智能技术重点实验室）
MorphoQuant: Modality-Aware Quantization for Omni-modal Large Language Models
institutetext: MorphoQuant: Modality-Aware Quantization for Omni-modal Large Language Models Yue Wu Changyuan Wang Zixuan Wang Shilin Ma Yansong Tang（机构文本：MorphoQuant：多模态大语言模型的模态感知量化 Yue Wu 王昌元 王梓轩 马世林 唐彦松）
Breaking Modality Heterogeneity in Low-Bit Quantization for Large Vision-Language Models
VCIP, College of Computer Science, Nankai University（南开大学计算机科学学院VCIP）；D-ITET, ETH Zürich（苏黎世联邦理工学院D-ITET）；OPPO Research Institute（OPPO研究院）；Department of Computing, Hong Kong Polytechnic University（香港理工大学计算机系）
Doctoral School of Applied Informatics and Applied Mathematics, Obuda University（应用信息与应用数学博士学院，奥布达大学）；John von Neumann Faculty of Informatics, Obuda University（约翰·冯·诺伊曼信息学院，奥布达大学）
SmoothRot: Combining Channel-Wise Scaling and Rotation for Quantization-Friendly LLMs
Doctoral School of Applied Informatics and Applied Mathematics, Obuda University（应用信息学与应用数学博士学院，奥布达大学）；John von Neumann Faculty of Informatics, Obuda University（约翰·冯·诺依曼信息学院，奥布达大学）
IntactKV: Improving Large Language Model Quantization by Keeping Pivot Tokens Intact
Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；The Chinese University of Hong Kong（香港中文大学）
MoBiQuant: Mixture-of-Bits Quantization for Token-Adaptive Any-Precision LLM
University of Arizona（亚利桑那大学）；Duke University（杜克大学）；Sungkyunkwan University（成均馆大学）；Panasonic AI Lab（松下人工智能实验室）；Korea Advanced Institute of Science and Technology（韩国科学技术院）
LFQ: Logit-aware Final-block Quantization for Boosting the Generation Quality of Low-Bit Quantized LLMs
Kim Jaechul Graduate School of AI, KAIST, Daejeon, South Korea（韩国科学技术院人工智能研究生院）；LG AI Research, Seoul, South Korea（LG人工智能研究）；Hanyang University, Seoul, South Korea（翰阳大学）
InfoQuant: Shaping Activation Distributions for Low-Bit LLM Quantization
School of Software Technology, Zhejiang University（浙江大学软件学院）；Ant Group（蚂蚁集团）；College of Computer Science and Technology, Zhejiang University of Technology（浙江工业大学计算机科学与技术学院）；China Mobile (Zhejiang) Research & Innovation Institute（中国移动（浙江）研究院）；Alibaba Cloud Computing（阿里云计算）；State Key Lab of CAD&CG, Zhejiang University（浙江大学CAD&CG国家重点实验室）
Assigning Distinct Roles to Quantized and Low-Rank Matrices Toward Optimal Weight Decomposition
Department of Computer Science, Yonsei University（yonsei大学计算机科学系）；Department of Artificial Intelligence, Yonsei University（yonsei大学人工智能系）；Department of Electronic & Electrical Convergence Engineering, Hongik University（hongik大学电子与电气融合工程系）
Pushing the Limits of Block Rotations in Post-Training Quantization
Advanced Micro Devices, Inc. (AMD)（Advanced Micro Devices公司）；State University of New York at Buffalo（纽约州立大学布法罗分校）；Norwegian University of Science（挪威科学与技术大学）
