arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2606.04620cs.LGcs.AI
收藏

QuBLAST: 一种采用块级压缩方法和激活缩放策略量化大语言模型的框架

QuBLAST: A Framework for Quantizing Large Language Models with Block-Level Compression Approach and Activation Scaling Strategy

eBRAIN Lab, Division of Engineering, New York University (NYU) Abu Dhabi(eBRAIN实验室,工程系,纽约大学(NYU)阿布扎赫德分校)

Pasindu Wickramasinghe, Achyuta Muthuvelan, Rachmad Vidya Wicaksana Putra, Minghao Shao, Muhammad Shafique

AI总结:

针对大语言模型部署困难,提出QuBLAST框架,通过块级混合精度量化和激活缩放策略,在降低模型大小40%-45.2%的同时保持困惑度增加不超过5%。

摘要:

大语言模型已成为解决NLP任务的最先进算法。然而,它们通常伴随着巨大的计算和内存成本,因此难以部署在嵌入式系统上。为此,最先进的方法通常在网络的所有注意力块上采用统一的训练后量化,从而忽略了在同一网络中应用不同量化级别的潜力。它们还采用复杂操作来减轻激活异常值的负面影响,从而产生高计算开销。此外,它们没有考虑使用具有非传统注意力架构(例如状态空间模型)的新兴大语言模型进行评估,这些模型在应用量化时提出了不同的挑战。为了解决这些局限性,我们提出了QuBLAST,一种新颖的训练后量化方法,该方法采用块级压缩方法和激活缩放策略用于大语言模型。块级压缩方法实现了网络各块之间的混合精度量化,而激活缩放策略有效减轻了激活异常值的负面影响。具体来说,QuBLAST首先通过交叉熵损失分析预训练模型中不同注意力块的敏感性。QuBLAST利用这种敏感性分析来确定模型中每个注意力块的权重量化级别。此外,QuBLAST为每个块采用激活缩放图来控制激活值的范围并减轻激活异常值的负面影响,从而实现更好的量化结果。实验结果表明,QuBLAST在不同模型架构(即Qwen3-8B、Llama3-8B、Mistral v0.1-8B和Falcon H1R-7B)上将模型大小减少了40%-45.2%,同时在WikiText-2和WikiText-103数据集上保持性能在5%的困惑度增加之内。

英文摘要:

LLMs have become the state-of-the-art algorithms for solving NLP tasks. However, they typically come at huge computational and memory costs, thus making them difficult to deploy on embedded systems. Toward this, state-of-the-art methods typically employ uniform post-training quantization (PTQ) across attention blocks of the network, hence overlooking the potential of applying different quantization levels in the same network. They also employ complex operations to mitigate the negative impact of activation outliers, hence incurring high computational overheads. Moreover, they have not considered evaluation using emerging LLMs with non-conventional attention architectures (e.g., state-space models), which pose different challenges in applying quantization. To address these limitations, we propose QuBLAST, a novel PTQ methodology that employs block-level compression approach with activation scaling strategy for LLMs. Block-level compression approach enables mixed-precision quantization across blocks of the network, while activation scaling strategy efficiently mitigates the negative impact of activation outliers. Specifically, QuBLAST first analyzes the sensitivity of different attention blocks in the pre-trained model through the cross-entropy loss analysis. QuBLAST leverages this sensitivity analysis to determine the weight quantization level for each attention block in the model. Furthermore, QuBLAST employs the activation scaling map for each block to control the range of activation values and mitigate the negative impact of activation outliers, thereby enabling better quantization results. Experimental results show that, QuBLAST reduces model sizes by 40%-45.2% across different model architectures (i.e., Qwen3-8B, Llama3-8B, Mistral v0.1-8B, and Falcon H1R-7B), while maintaining the performance within 5% perplexity increase for the WikiText-2 and WikiText-103 datasets.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑