Recent studies introduced effective compression techniques for Large Language Models (LLMs) via post-training quantization or low-bit weight representation. Although quantized weights offer storage efficiency and allow for faster inference, existing works have indicated that quantization might compromise performance and exacerbate biases in LLMs. This study investigates the confidence and calibration of quantized models, considering factors such as language model type and scale as contributors to quantization loss. Firstly, we reveal that quantization with GPTQ to 4-bit results in a decrease in confidence regarding true labels, with varying impacts observed among different language models. Secondly, we observe fluctuations in the impact on confidence across different scales. Finally, we propose an explanation for quantization loss based on confidence levels, indicating that quantization disproportionately affects samples where the full model exhibited low confidence levels in the first place.
Target-Aware Calibration Data Selection for Preserving Uncertainty in Quantized Language Models
The Hong Kong University of Science and Technology(香港科技大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));City University of Hong Kong(香港城市大学);Shanghai Institute of Optics and Fine Mechanics(上海光学精密机械研究所);University of Oxford(牛津大学);Yale University(耶鲁大学)
Gated Relational Alignment via Confidence-based Distillation for Efficient VLMs
Department of Information Technology(信息科技系);Electrical Engineering, ETH Zurich, Zurich, Switzerland(电气工程,苏黎世联邦理工学院,苏黎世,瑞士);Qualcomm AI Research, Amsterdam, the Netherlands(高通人工智能研究,阿姆斯特丹,荷兰);Department of Electrical, Electronic and Information Engineering(电气、电子与信息工程系);University of Bologna, Bologna, Italy(博洛尼亚大学,博洛尼亚,意大利);School of Electrical and Electronic Engineering(电气与电子工程学院)
The Impact of Quantization on Retrieval-Augmented Generation: An Analysis of Small LLMs
Amsterdam University of Applied Sciences(阿姆斯特丹应用科学大学);Leiden University(莱顿大学);Peoples' Friendship University of Russia (RUDN University)(俄罗斯人民友谊大学(鲁登大学));Joint Institute for Nuclear Research(联合核子研究所);Vrije Universiteit Amsterdam(阿姆斯特丹自由大学);University of Skövde(舍夫德大学)
Latent Confidence Alignment for LLM Self-Assessment
Department of Information Management(信息管理系);National Sun Yat-Sen University(国立中山大学);Kaohsiung Medical University Hospital(高雄医学大学附设医院);Kaohsiung Medical University(高雄医学大学)
How Quantization Shapes Bias in Large Language Models
Tsinghua University(清华大学);The Hebrew University of Jerusalem(耶路撒冷希伯来大学);TU Darmstadt(达姆施塔特工业大学);National Research Center for Applied Cybersecurity ATHENE(雅典娜国家应用网络安全研究中心)