Compensate Quantization Errors: Make Weights Hierarchical to Compensate Each Other
University of Electronic Science and Technology of China(电子科技大学);Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);Beijing Normal University(北京师范大学);Tianjin University(天津大学)
Yifei Gao, Jie Ou, Lei Wang, Yuting Xiao, Zhiyuan Xiang, Ruiting Dai, Jun Cheng
新兴的大语言模型(Large Language Models, LLMs)凭借其卓越的性能和强大的推理能力,与传统语言模型形成了鲜明区分。然而,这类大语言模型的计算资源和存储成本高得惊人,量化技术也因此成为了热门研究方向。为解决量化带来的精度下降问题,训练后量化方法中涌现出两类主流方案:一类利用其他权重补偿已有的量化误差,另一类则将量化难度转移到模型的其他部分。结合两者的优势,我们提出了可学习奇异值增量(Learnable Singular value Increment, LSI)作为进阶解决方案。LSI 利用奇异值分解(Singular Value Decomposition)提取权重的奇异值,并使其具备可学习性,从而在激活值的条件下帮助权重相互补偿。将 LSI 与现有技术结合后,我们在多种量化设置下都取得了当前最优性能,无论是仅权重量化、权重-激活量化还是极低比特场景均是如此。通过释放 LSI 的潜力,在量化模型上进行高效微调不再是一个难以实现的问题。
英文摘要:
Emergent Large Language Models (LLMs) use their extraordinary performance and powerful deduction capacity to discern from traditional language models. However, the expenses of computational resources and storage for these LLMs are stunning, quantization then arises as a trending conversation. To address accuracy decay caused by quantization, two streams of works in post-training quantization methods stand out. One uses other weights to compensate existing quantization error, while the other transfers the quantization difficulty to other parts in the model. Combining both merits, we introduce Learnable Singular value Increment (LSI) as an advanced solution. LSI uses Singular Value Decomposition to extract singular values of the weights and make them learnable to help weights compensate each other conditioned on activation. Incorporating LSI with existing techniques, we achieve state-of-the-art performance in diverse quantization settings, no matter in weight-only, weight-activation or extremely low bit scenarios. By unleashing the potential of LSI, efficient finetuning on quantized model is no longer a prohibitive problem.
Quant Experts: Token-aware Adaptive Error Reconstruction with Mixture of Experts for Large Vision-Language Models Quantization
State Key Laboratory of Human-Machine Hybrid Augmented Intelligence(人机混合增强智能国家重点实验室);Institute of Artificial Intelligence and Robotics(人工智能与机器人研究院);Xi’an Jiaotong University(西安交通大学)
D-QRELO: Training- and Data-Free Delta Compression for Large Language Models via Quantization and Residual Low-Rank Approximation
Harbin Institute of Technology, Shenzhen, China(哈尔滨工业大学(深圳));TeleAI of China Telecom(中国电信TeleAI);The Hong Kong Polytechnic University(香港理工大学);The University of Hong Kong(香港大学)