arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.11534cs.LGcs.AI
收藏

LRQ:通过学习低秩权重缩放矩阵优化大语言模型的训练后量化

LRQ: Optimizing Post-Training Quantization for Large Language Models by Learning Low-Rank Weight-Scaling Matrices

NAVER Cloud(NAVER云);KAIST AI(韩国科学技术院人工智能学院);AITRICS;SNU AI Center(首尔大学人工智能中心)

Jung Hyun Lee, Jeonghoon Kim, June Yong Yang, Se Jung Kwon, Eunho Yang, Kang Min Yoo, Dongsoo Lee

AI总结:

本文提出低秩量化方法LRQ,通过用低秩权重缩放矩阵替代全权重缩放矩阵来重建Transformer块输出,在显著减少可学习参数的同时提升量化LLMs的泛化能力,并在多种位宽量化设置下优于现有PTQ方法。

摘要:

随着大语言模型(LLMs)的商业化,权重-激活量化已被用于压缩和加速LLMs,在降低推理成本的同时实现高吞吐量。然而,现有的用于量化LLMs权重和激活的训练后量化(PTQ)技术仍然遭受不可忽视的精度下降,尤其是在大规模多任务语言理解方面。为了解决这一问题,我们提出了低秩量化(LRQ)——一种简单而有效的LLM训练后权重量化方法,该方法通过利用低秩权重缩放矩阵来重建中间Transformer块的输出,取代了传统的全权重缩放矩阵,后者需要与其关联权重同样多的可学习缩放参数。得益于低秩结构带来的参数共享,LRQ只需学习显著更少的参数,同时能够对权重进行逐项缩放,从而提升了量化后LLMs的泛化能力。我们展示了LRQ在以下设置中优于先前的LLM PTQ工作:(i)8位权重和逐张量激活量化,(ii)4位权重和8位逐令牌激活量化,以及(iii)低位仅权重量化方案。我们的代码可在Software获取。

英文摘要:

With the commercialization of large language models (LLMs), weight-activation quantization has emerged to compress and accelerate LLMs, achieving high throughput while reducing inference costs. However, existing post-training quantization (PTQ) techniques for quantizing weights and activations of LLMs still suffer from non-negligible accuracy drops, especially on massive multitask language understanding. To address this issue, we propose Low-Rank Quantization (LRQ) - a simple yet effective post-training weight quantization method for LLMs that reconstructs the outputs of an intermediate Transformer block by leveraging low-rank weight-scaling matrices, replacing the conventional full weight-scaling matrices that entail as many learnable scales as their associated weights. Thanks to parameter sharing via low-rank structure, LRQ only needs to learn significantly fewer parameters while enabling the individual scaling of weights, thus boosting the generalization capability of quantized LLMs. We show the superiority of LRQ over prior LLM PTQ works under (i) 8-bit weight and per-tensor activation quantization, (ii) 4-bit weight and 8-bit per-token activation quantization, and (iii) low-bit weight-only quantization schemes. Our code is available at Software.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑