arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2312.07950cs.LGcs.CL
收藏

CBQ:大型语言模型的跨块量化

CBQ: Cross-Block Quantization for Large Language Models

University of Science and Technology of China(中国科学技术大学);Hong Kong University of Science and Technology (GZ)(香港科技大学(广州))

Xin Ding, Xiaoyu Liu, Zhijun Tu, Yun Zhang, Wei Li, Jie Hu, Hanting Chen, Yehui Tang, Zhiwei Xiong, Baoqun Yin, Yunhe Wang

AI总结:

提出跨块重建量化方法CBQ,通过同源重建和粗到细预处理及自适应LoRA-Rounding,在低位量化中优于现有方法,并高效量化65B模型。

摘要:

训练后量化(PTQ)在超低成本的压缩大型语言模型(LLMs)方面发挥了关键作用。然而,现有的PTQ方法仅关注处理单个层或单个块内的离群值,忽略了块之间的依赖关系,导致在低位设置下性能严重下降。在本文中,我们提出了CBQ,一种基于跨块重建的PTQ方法,用于大型语言模型。CBQ采用同源重建方案实现跨块依赖,建立跨多个块的长距离依赖关系,以最小化误差累积。此外,CBQ结合了一种从粗到细的预处理(CFP)策略,用于抑制权重和激活离群值,并配合自适应LoRA-Rounding技术进行精确的权重量化。这些创新使CBQ不仅能够有效处理极端离群值,还能提高整体量化精度。大量实验表明,CBQ在低位量化(W4A4、W4A8、W2A16)方面取得了优越性能,并在各种LLMs和数据集上优于现有的最先进方法。值得注意的是,CBQ在单个GPU上仅用4.3小时就量化了4位LLAMA1-65B模型,实现了性能与量化效率之间的良好平衡。

英文摘要:

Post-training quantization (PTQ) has played a key role in compressing large language models (LLMs) with ultra-low costs. However, existing PTQ methods only focus on handling the outliers within one layer or one block, which ignores the dependency of blocks and leads to severe performance degradation in low-bit settings. In this paper, we propose CBQ, a cross-block reconstruction-based PTQ method for LLMs. CBQ employs a cross-block dependency using a homologous reconstruction scheme, establishing long-range dependencies across multiple blocks to minimize error accumulation. Furthermore, CBQ incorporates a coarse-to-fine preprocessing (CFP) strategy for suppressing weight and activation outliers, coupled with an adaptive LoRA-Rounding technique for precise weight quantization. These innovations enable CBQ to not only handle extreme outliers effectively but also improve overall quantization accuracy. Extensive experiments show that CBQ achieves superior low-bit quantization (W4A4, W4A8, W2A16) and outperforms existing state-of-the-art methods across various LLMs and datasets. Notably, CBQ quantizes the 4-bit LLAMA1-65B model within only 4.3 hours on a single GPU, achieving a commendable tradeoff between performance and quantization efficiency.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑