arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2405.00632cs.CLcs.AI
收藏

量化何时影响大型语言模型的置信度?

When Quantization Affects Confidence of Large Language Models?

Université de Lyon(里昂大学)

Irina Proskurina, Luc Brun, Guillaume Metzler, Julien Velcin

AI总结:

本文研究量化对LLMs置信度的影响,发现GPTQ 4比特量化会降低模型对真实标签的置信度,且影响随模型类型和规模变化,并指出量化损失主要集中在全精度模型原本低置信的样本上。

摘要:

近期研究通过训练后量化或低比特权重表示,为大型语言模型(LLMs)引入了有效的压缩技术。尽管量化权重能够带来存储效率并支持更快的推理,但已有工作表明量化可能会损害性能并加剧LLMs中的偏差。本研究考察了量化模型的置信度与校准情况,并将语言模型类型和规模等因素视为量化损失的来源。首先,我们揭示使用GPTQ量化到4比特会导致模型对真实标签的置信度下降,且在不同语言模型中观察到的影响各不相同。其次,我们观察到不同规模下量化对置信度的影响存在波动。最后,我们基于置信水平提出了对量化损失的一种解释,表明量化对全精度模型原本就表现出低置信水平的样本影响更为显著。

英文摘要:

Recent studies introduced effective compression techniques for Large Language Models (LLMs) via post-training quantization or low-bit weight representation. Although quantized weights offer storage efficiency and allow for faster inference, existing works have indicated that quantization might compromise performance and exacerbate biases in LLMs. This study investigates the confidence and calibration of quantized models, considering factors such as language model type and scale as contributors to quantization loss. Firstly, we reveal that quantization with GPTQ to 4-bit results in a decrease in confidence regarding true labels, with varying impacts observed among different language models. Secondly, we observe fluctuations in the impact on confidence across different scales. Finally, we propose an explanation for quantization loss based on confidence levels, indicating that quantization disproportionately affects samples where the full model exhibited low confidence levels in the first place.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑