Methods based on weight compensation, which iteratively apply quantization and weight compensation to minimize the output error, have recently demonstrated remarkable success in quantizing Large Language Models (LLMs). The representative work, GPTQ, introduces several key techniques that make such iterative methods practical for LLMs with billions of parameters. GPTAQ extends this approach by introducing an asymmetric calibration process that aligns the output of each quantized layer with its full-precision counterpart, incorporating a residual error into the weight compensation framework. In this work, we revisit the formulation of the residual error. We identify a sub-optimal calibration objective in existing methods: during the intra-layer calibration process, they align the quantized output with the output from compensated weights, rather than the true output from the original full-precision model. Therefore, we redefine the objective to precisely align the quantized model's output with the original output of the full-precision model at each step. We then reveal that the residual error originates not only from the output difference of the preceding layer but also from the discrepancy between the compensated and original weights within each layer, which we name the 'compensation-aware error'. By inheriting the neuron decomposition technique from GPTAQ, we can efficiently incorporate this compensation-aware error into the weight update process. Extensive experiments on various LLMs and quantization settings demonstrate that our proposed enhancements integrate seamlessly with both GPTQ and GPTAQ, significantly improving their quantization performance. Our code is publicly available at https://github.com/list0830/ResComp.
Tango: rethinking quantization for graph neural network training on GPUs
Rutgers, The State University of New Jersey(罗格斯,新泽西州立大学);Amazon(亚马逊);University of Connecticut(康涅狄格大学);Institution of Software, Chinese Academy of Sciences(中国科学院软件研究所);University of North Texas(北德克萨斯大学)
Beyond local reconstruction: Geometry-preserving output alignment for binary LLM quantization
Department of Data Science and AI(数据科学与人工智能系);Monash University(墨尔本大学);Centre for Vision, Speech and Signal Processing(视觉、语音和信号处理中心);University of Surrey(萨里大学)
GPTQ-intrinsic LoRA: A Near-optimal Algorithm for Low-precision Quantization with Low-rank Adaptation
Department of Mathematics, University of California San Diego(数学系,加州大学圣地亚哥分校);Department of Mathematics and Halıcıoğlu Data Science Institute, University of California San Diego(数学系和Halıcıoğlu数据科学研究所,加州大学圣地亚哥分校)
Quantitative Error Feedback for Quantization Noise Reduction of Filtering over Graphs
Division of Computer, Electrical and Mathematical Sciences and Engineering (CEMSE), King Abdullah University Of Science And Technology (KAUST)(计算机、电子和数学科学与工程系,国王阿卜杜勒-阿齐兹大学科学与技术(KAUST));School of Information Science and Technology, ShanghaiTech University(信息科学与技术学院,上海科技大学);Department of Electrical and Electronic Engineering, Imperial College London(电子与电气工程系,帝国理工学院伦敦分校)
A Comprehensive Evaluation on Quantization Techniques for Large Language Models
School of Computer Science and Technology, Tongji University(同济大学计算机科学与技术学院);School of Engineering Math and Technology, University of Bristol(布里斯托大学工程数学与技术学院)
RPIQ: Residual-Projected Multi-Collaboration Closed-Loop and Single Instance Quantization for Visually Impaired Assistance
School of Information;Software Engineering, University of Electronic Science;School of Mathematical Sciences, Sichuan Normal University, Chengdu, Sichuan, China;Sichuan Provincial Key Laboratory for Human Disease Gene Study, Sichuan Academy of Medical Sciences \& Sichuan Provincial People's Hospital, University of Electronic Science;Research Unit for Blindness Prevention, Chinese Academy of Medical Sciences, Sichuan Academy of Medical Sciences;Sichuan Provincial People’s Hospital, Chengdu, Sichuan, China;School of Medicine, University of Electronic Science;Tibetan Language Intelligence National Key Laboratory, Qinghai Normal University, Xining, Qinghai, China
From Signal Degradation to Computation Collapse: Uncovering the Two Failure Modes of LLM Quantization
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences(中国科学院大学先进交叉学科学院);The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所认知与决策智能复杂系统重点实验室);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);College of Computer Science, Inner Mongolia University(内蒙古大学计算机学院)
Replay the Curvature: Accurate and Scalable NVFP4 Quantization for Large Language Model Inference
KlingAI Research(可灵AI研究院);Fudan University(复旦大学);Shanghai AI Incubation and Innovation Center(上海人工智能孵化与创新中心);Shanghai Academy of AI for Science(上海人工智能科学研究院);University of Science and Technology of China(中国科学技术大学)