QUIK: Towards End-to-End 4-Bit Inference on Generative Large Language Models
ETH Zurich(苏黎世联邦理工学院);Institute of Science and Technology Austria(奥地利科学技术研究所);Xidian University(西安电子科技大学);KAUST(阿卜杜拉国王科技大学);Neural Magic, Inc.(Neural Magic公司)
Saleh Ashkboos, Ilia Markov, Elias Frantar, Tingxuan Zhong, Xincheng Wang, Jie Ren, Torsten Hoefler, Dan Alistarh
Large Language Models (LLMs) from the GPT family have become extremely popular, leading to a race towards reducing their inference costs to allow for efficient local computation. Yet, the vast majority of existing work focuses on weight-only quantization, which can reduce runtime costs in the memory-bound one-token-at-a-time generative setting, but does not address them in compute-bound scenarios, such as batched inference or prompt processing. In this paper, we address the general quantization problem, where both weights and activations should be quantized. We show, for the first time, that the majority of inference computations for large generative models such as LLaMA, OPT, and Falcon can be performed with both weights and activations being cast to 4 bits, in a way that leads to practical speedups, while at the same time maintaining good accuracy. We achieve this via a hybrid quantization strategy called QUIK, which compresses most of the weights and activations to 4-bit, while keeping some outlier weights and activations in higher-precision. The key feature of our scheme is that it is designed with computational efficiency in mind: we provide GPU kernels matching the QUIK format with highly-efficient layer-wise runtimes, which lead to practical end-to-end throughput improvements of up to 3.4x relative to FP16 execution. We provide detailed studies for models from the OPT, LLaMA-2 and Falcon families, as well as a first instance of accurate inference using quantization plus 2:4 sparsity. Code is available at: https://github.com/IST-DASLab/QUIK.
Bidirectional Generative Pre-training for Improving Healthcare Time-series Representation Learning
School of Computer Science, McGill University(麦吉尔大学计算机学院);School of Population and Global Health, McGill University(麦吉尔大学人口与全球健康学院);Mila - Quebec AI Institute(米拉-魁北克人工智能研究所)
Department of Computer Science and Technology, Tsinghua University(清华大学计算机科学与技术系);Department of Electrical and Computer Engineering, University of California, Santa Barbara(加利福尼亚大学圣巴巴拉分校电气与计算机工程系)
Quantization of Generative Adversarial Networks for Efficient Inference: a Methodological Study
Higher School of Economics(高等经济学院);Skolkovo Institute of Science and Technology(斯科尔科沃科学技术学院);Samsung AI Center Moscow(三星莫斯科人工智能中心);Yandex;Samsung-HSE Laboratory(三星-高等经济学院实验室)
Post-Training Quantization of Generative and Discriminative LSTM Text Classifiers: A Study of Calibration, Class Balance, and Robustness
School of Mathematical and Data Sciences(数学与数据科学学院);West Virginia University(西弗吉尼亚大学);Department of Industrial and Management Systems Engineering(工业与管理系统工程系)
A Bit of Freedom Goes a Long Way: Classical and Quantum Algorithms for Reinforcement Learning under a Generative Model
Center for Quantum Computer Science, Faculty of Computing(量子计算机科学中心,计算学院);University of Latvia(拉脱维亚大学);HUN-REN Alfréd Rényi Institute of Mathematics(匈牙利-中国阿尔弗雷德·雷尼数学研究所)
Modulated Diffusion: Accelerating Generative Modeling with Modulated Quantization
Department of Computer Science, North Carolina State University(北卡罗来纳州立大学计算机科学系);National Center for Computational Science, Oak Ridge National Lab(橡树岭国家实验室计算科学中心);Department of Mechanical Engineering, Keio University(庆应大学机械工程系)
Bit-Level Discrete Diffusion with Markov Probabilistic Models: An Improved Framework with Sharp Convergence Bounds under Minimal Assumptions
Department of XXX, University of YYY, Location, Country;School of ZZZ, Institute of WWW, Location, Country;Centre de Math\'ematiques Appliqu\'ees (CMAP), \'Ecole Polytechnique, Institut Polytechnique de Paris, 91120 Palaiseau, France;Department of Mathematics, University of Padova, Italy;Sierra lab, Inria, Paris, France
Computer Systems Lab, Cornell University(康奈尔大学计算机系统实验室);Systems and Networking Research Group, Microsoft Research(微软研究院系统与网络研究组);Department of Electrical and Electronic Engineering, Imperial College London(伦敦帝国理工学院电子与电气工程系)