There has been significant interest in "extreme" compression of large language models (LLMs), i.e., to 1-2 bits per parameter, which allows such models to be executed efficiently on resource-constrained devices. Existing work focused on improved one-shot quantization techniques and weight representations; yet, purely post-training approaches are reaching diminishing returns in terms of the accuracy-vs-bit-width trade-off. State-of-the-art quantization methods such as QuIP# and AQLM include fine-tuning (part of) the compressed parameters over a limited amount of calibration data; however, such fine-tuning techniques over compressed weights often make exclusive use of straight-through estimators (STE), whose performance is not well-understood in this setting. In this work, we question the use of STE for extreme LLM compression, showing that it can be sub-optimal, and perform a systematic study of quantization-aware fine-tuning strategies for LLMs. We propose PV-Tuning - a representation-agnostic framework that generalizes and improves upon existing fine-tuning strategies, and provides convergence guarantees in restricted cases. On the practical side, when used for 1-2 bit vector quantization, PV-Tuning outperforms prior techniques for highly-performant models such as Llama and Mistral. Using PV-Tuning, we achieve the first Pareto-optimal quantization for Llama 2 family models at 2 bits per parameter.
GraphComp: Extreme Error-bounded Compression of Scientific Data via Temporal Graph Autoencoders
Computer, Electrical and Mathematical Sciences and Engineering Division, KAUST(卡奥斯特大学计算机、电气与数学科学与工程分校);Department of Informatics and Telecommunications, University of Peloponnese(希腊皮埃尼亚斯大学信息与电信系);Physical Science and Engineering Division, KAUST(卡奥斯特大学物理科学与工程分校)
Training with Quantization Noise for Extreme Model Compression
Facebook AI Research(Facebook人工智能研究院);LORIA(洛里亚计算机科学实验室);Univ Lyon(里昂大学);Inria(法国国家信息与自动化研究所);CNRS(法国国家科学研究中心);ENS de Lyon(里昂高等师范学校);UCB Lyon 1(里昂第一大学);LIP UMR 5668(里昂第一大学计算机科学实验室)
UltraSketchLLM: Sub-1-Bit LLM Compression via Sketch and Hardware-Friendly Operators
National Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University(国家多媒体信息处理重点实验室,计算机科学学院,北京大学);School of Electronic Engineering and Computer Science, Peking University(电子工程与计算机科学学院,北京大学);Center for Energy-efficient Computing and Applications, Peking University(能效计算与应用中心,北京大学)
CompactifAI: Extreme Compression of Large Language Models using Quantum-Inspired Tensor Networks
Multiverse Computing(多维计算公司);Catalan Institute of Nanoscience and Nanotechnology (ICN2)(加泰罗尼亚纳米科学与纳米技术研究所);The Barcelona Institute of Science and Technology(巴塞罗那科学技术研究院);Donostia International Physics Center(多斯蒂亚国际物理中心);Centre for Social Innovation(社会创新中心)