Pretrained large language models (LLMs) exhibit exceptional general language processing capabilities but come with significant demands on memory and computational resources. As a powerful compression technology, binarization can extremely reduce model weights to a mere 1 bit, lowering the expensive computation and memory requirements. However, existing quantization techniques fall short of maintaining LLM performance under ultra-low bit-widths. In response to this challenge, we present BiLLM, a groundbreaking 1-bit post-training quantization scheme tailored for pretrained LLMs. Based on the weight distribution of LLMs, BiLLM first identifies and structurally selects salient weights, and minimizes the compression loss through an effective binary residual approximation strategy. Moreover, considering the bell-shaped distribution of the non-salient weights, we propose an optimal splitting search to group and binarize them accurately. BiLLM achieving for the first time high-accuracy inference (e.g. 8.41 perplexity on LLaMA2-70B) with only 1.08-bit weights across various LLMs families and evaluation metrics, outperforms SOTA quantization methods of LLM by significant margins. Moreover, BiLLM enables the binarization process of the LLM with 7 billion weights within 0.5 hours on a single GPU, demonstrating satisfactory time efficiency. Our code is available at https://github.com/Aaronhuang-778/BiLLM.
An Empirical Study of openPangu Quantization on Ascend NPUs
State Key Laboratory of Complex & Critical Software Environment, Beihang University（北京航空航天大学复杂关键软件环境国家重点实验室）；School of Computer Science and Engineering, Beihang University（北京航空航天大学计算机科学与工程学院）；School of Artificial Intelligence, Beihang University（北京航空航天大学人工智能学院）
Achieving binary weight and activation for LLMs using Post-Training Quantization
MAIS, Institute of Automation, Chinese Academy of Sciences（自动化研究所，中国科学院）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Central Media Technology Institute, Huawei Technologies Ltd.（华为技术有限公司媒体技术研究所）
Pushing Large Language Models to the 6G Edge: Vision, Challenges, and Opportunities
Department of Electrical and Electronic Engineering, University of Hong Kong（香港大学电子与电气工程系）；HKU Musketeers Foundation Institute of Data Science, University of Hong Kong（香港大学穆斯奎特基金会数据科学研究所）；School of Computer Science, Fudan University（复旦大学计算机学院）
Breaking Modality Heterogeneity in Low-Bit Quantization for Large Vision-Language Models
VCIP, College of Computer Science, Nankai University（南开大学计算机科学学院VCIP）；D-ITET, ETH Zürich（苏黎世联邦理工学院D-ITET）；OPPO Research Institute（OPPO研究院）；Department of Computing, Hong Kong Polytechnic University（香港理工大学计算机系）
Pushing the Limits of Block Rotations in Post-Training Quantization
Advanced Micro Devices, Inc. (AMD)（Advanced Micro Devices公司）；State University of New York at Buffalo（纽约州立大学布法罗分校）；Norwegian University of Science（挪威科学与技术大学）
Shedding the Bits: Pushing the Boundaries of Quantization with Minifloats on FPGAs
School of Computing, National University of Singapore（新加坡国立大学计算机学院）；AMD Research and Advanced Development（AMD研究与高级开发部）；Tampere University（坦佩雷大学）；AMD（AMD公司）
