Large language models (LLMs) have proven to be very superior to conventional methods in various tasks. However, their expensive computations and high memory requirements are prohibitive for deployment. Model quantization is an effective method for reducing this overhead. The problem is that in most previous works, the quantized model was calibrated using few samples from the training data, which might affect the generalization of the quantized LLMs to unknown cases and tasks. Hence in this work, we explore an important question: Can we design a data-independent quantization method for LLMs to guarantee its generalization performance? In this work, we propose EasyQuant, a training-free and data-independent weight-only quantization algorithm for LLMs. Our observation indicates that two factors: outliers in the weight and quantization ranges, are essential for reducing the quantization error. Therefore, in EasyQuant, we leave the outliers (less than 1%) unchanged and optimize the quantization range to reduce the reconstruction error. With these methods, we surprisingly find that EasyQuant achieves comparable performance to the original model. Since EasyQuant does not depend on any training data, the generalization performance of quantized LLMs is safely guaranteed. Moreover, EasyQuant can be implemented in parallel so that the quantized model could be attained in a few minutes even for LLMs over 100B. To our best knowledge, we are the first work that achieves almost lossless quantization performance for LLMs under a data-independent setting and our algorithm runs over 10 times faster than the data-dependent methods.
Achieving binary weight and activation for LLMs using Post-Training Quantization
MAIS, Institute of Automation, Chinese Academy of Sciences(自动化研究所,中国科学院);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Central Media Technology Institute, Huawei Technologies Ltd.(华为技术有限公司媒体技术研究所)
Q-resafe: Assessing Safety Risks and Quantization-aware Safety Patching for Quantized Large Language Models
The State Key Laboratory of Blockchain and Data Security, Zhejiang University(区块链与数据安全国家重点实验室,浙江大学);Sun Yat-sen University(中山大学);Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security(杭州高新技术区(滨江)区块链与数据安全研究院)
HESTIA: A Hessian-Guided Differentiable Quantization-Aware Training Framework for Extremely Low-Bit LLMs
School of Software and Microelectronics, Peking University, Beijing, China(软件与微电子学院,北京大学,北京,中国);School of Computer Science, Peking University, Beijing, China(计算机科学学院,北京大学,北京,中国)
SmoothRot: Combining Channel-Wise Scaling and Rotation for Quantization-Friendly LLMs
Doctoral School of Applied Informatics and Applied Mathematics, Obuda University(应用信息学与应用数学博士学院,奥布达大学);John von Neumann Faculty of Informatics, Obuda University(约翰·冯·诺依曼信息学院,奥布达大学)
End-to-End On-Device Quantization-Aware Training for LLMs at Inference Cost
University of Georgia(佐治亚大学);University of North Texas(北卡罗来纳州立大学);Northeastern University(东北大学);University of Minnesota(明尼苏达大学);University of Virginia(弗吉尼亚大学);Stevens Institute of Technology(史蒂文斯理工学院)
SDQ-LLM: Sigma-Delta Quantization for 1-bit LLMs of any size
Dept of Electronic Engineering, Tsinghua University, Beijing, China(电子工程系,清华大学,北京,中国);Beijing National Research Center for Information Science and Technology, Beijing, China(北京信息科学与技术国家研究中心,北京,中国)
Calibration and Transformation-Free Weight-Only LLMs Quantization via Dynamic Grouping
Division of Artificial Intelligence, School of Data Science, Lingnan University, Hong Kong, China(岭南大学人工智能学院);Department of Computer Science and Engineering, The Hong Kong University of Science and Technology, Hong Kong, China(香港理工大学计算机科学与工程系);Department of Computing, The Hong Kong Polytechnic University, Hong Kong, China(香港理工大学计算机系);Xiaoi Robot Inc., Shanghai, China(小蚁机器人有限公司)
MODE: Modality-Decomposed Expert-Level Mixed-Precision Quantization for MoE Multimodal LLMs
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Zhongguancun Academy(中关村学院)