arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2310.05079cs.LG
收藏

重新审视块量化:对于低于8比特的LLM推理,什么才是重要的?

Revisiting Block-based Quantisation: What is Important for Sub-8-bit LLM Inference?

Imperial College London(帝国理工学院);University of Oxford(牛津大学)

Cheng Zhang, Jianyi Cheng, Ilia Shumailov, George A. Constantinides, Yiren Zhao

AI总结:

针对LLM推理的高计算与内存成本,本文探索层统计与学习特性,将量化瓶颈归因于数值缩放偏移,并适配块量化方法,实现无需校准或重训的近乎无损6比特量化,算术密度较float32提升19倍,并分享子8比特量化见解,使4比特LLM在下游任务近乎无损。

摘要:

大型语言模型(LLM)的推理需要巨大的计算和内存资源。为降低这些成本,量化已成为一种有前景的解决方案,但现有的LLM量化主要聚焦于8比特。在这项工作中,我们探讨了LLM层的统计与学习特性,并将LLM量化的瓶颈归因于数值缩放偏移。为解决此问题,我们将块量化(一种在打包数字间共享缩放因子的方法族)适配于LLM。块量化仅从算术角度有效减少数值缩放偏移,而无需在计算路径中增加额外处理。我们近乎无损的6比特量化LLM,其算术密度比float32基线提高了19倍,内存密度提高了5倍,在算术密度上比现有最优的8比特量化高出2.5倍,内存密度高出1.2倍,且无需任何数据校准或重新训练。我们还分享了对低于8比特LLM量化的见解,包括激活与权重分布之间的不匹配、最优微调策略,以及LLM统计特性中固有的更低量化粒度。后两个技巧使得在下游任务上实现近乎无损的4比特LLM成为可能。我们的代码已开源。

英文摘要:

The inference of Large language models (LLMs) requires immense computation and memory resources. To curtail these costs, quantisation has merged as a promising solution, but existing LLM quantisation mainly focuses on 8-bit. In this work, we explore the statistical and learning properties of the LLM layer and attribute the bottleneck of LLM quantisation to numerical scaling offsets. To address this, we adapt block quantisations for LLMs, a family of methods that share scaling factors across packed numbers. Block quantisations efficiently reduce the numerical scaling offsets solely from an arithmetic perspective, without additional treatments in the computational path. Our nearly-lossless quantised 6-bit LLMs achieve a $19\times$ higher arithmetic density and $5\times$ memory density than the float32 baseline, surpassing the prior art 8-bit quantisation by $2.5\times$ in arithmetic density and $1.2\times$ in memory density, without requiring any data calibration or re-training. We also share our insights into sub-8-bit LLM quantisation, including the mismatch between activation and weight distributions, optimal fine-tuning strategies, and a lower quantisation granularity inherent in the statistical properties of LLMs. The latter two tricks enable nearly-lossless 4-bit LLMs on downstream tasks. Our code is open-sourced.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑