Post-Training Quantization (PTQ) is pivotal for deploying large language models (LLMs) within resource-limited settings by significantly reducing resource demands. However, existing PTQ strategies underperform at low bit levels < 3 bits due to the significant difference between the quantized and original weights. To enhance the quantization performance at low bit widths, we introduce a Mixed-precision Graph Neural PTQ (MG-PTQ) approach, employing a graph neural network (GNN) module to capture dependencies among weights and adaptively assign quantization bit-widths. Through the information propagation of the GNN module, our method more effectively captures dependencies among target weights, leading to a more accurate assessment of weight importance and optimized allocation of quantization strategies. Extensive experiments on the WikiText2 and C4 datasets demonstrate that our MG-PTQ method outperforms previous state-of-the-art PTQ method GPTQ, setting new benchmarks for quantization performance under low-bit conditions.
AnTKV: Anchor Token-Aware Sub-Bit Vector Quantization for KV Cache in Large Language Models
DSA Thrust, The Hong Kong University of Science and Technology (Guangzhou)(DSA团队,香港科学与技术大学(广州));School of Mathematics and Computational Science, Xiangtan University(数学与计算科学学院,湘潭大学);Microsoft(微软)
Efficient Speech Representation Learning with Low-Bit Quantization
Meta AI
随着机器学习硬件的发展,更新的模型通常以增加的尺寸和计算复杂性为代价。在努力提高这些模型的效率,我们应用和研究最近的量化技术的语音表示学习模型。在SUPERB基准上对量化技术进行了评估。在ASR任务中,通过积极量化到1位,我们实现了86.32%的存储减少(184.42 -> 25.23),88%的估计运行时间减少(1.00 -> 0.12),同时增加了字错误率(7.06 -> 15.96)。与同样致力于模型压缩的DistillHuBERT相比,2位配置产生了更小的存储(35.84 vs. 46.98),更好的字错误率(12.68 vs. 13.37)和更有效的估计运行时间(0.15 vs. 0.73)。