arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2310.09259cs.LG
收藏

QUIK:迈向生成式大型语言模型的端到端4比特推理

QUIK: Towards End-to-End 4-Bit Inference on Generative Large Language Models

ETH Zurich(苏黎世联邦理工学院);Institute of Science and Technology Austria(奥地利科学技术研究所);Xidian University(西安电子科技大学);KAUST(阿卜杜拉国王科技大学);Neural Magic, Inc.(Neural Magic公司)

Saleh Ashkboos, Ilia Markov, Elias Frantar, Tingxuan Zhong, Xincheng Wang, Jie Ren, Torsten Hoefler, Dan Alistarh

AI总结:

本文提出QUIK混合4比特量化方法及匹配GPU内核,实现LLaMA、OPT、Falcon权重与激活的端到端高效推理,最高较FP16提速3.4倍。

摘要:

GPT系列大型语言模型(LLM)已变得极为流行,从而引发了一场降低其推理成本以实现高效本地计算的竞赛。然而,现有绝大多数工作都集中在仅权重量化上,这种方法可以在内存受限的逐词生成场景中降低运行时成本,但无法解决批量推理或提示处理等计算受限场景中的成本问题。在本文中,我们解决了权重和激活都应被量化的通用量化问题。我们首次表明,LLaMA、OPT和Falcon等大型生成模型的大多数推理计算都可以在权重和激活均转换为4比特的情况下执行,从而带来实际的加速,同时保持良好的精度。我们通过一种名为QUIK的混合量化策略实现这一点,该策略将大多数权重和激活压缩到4比特,同时将部分离群权重和激活保持在更高精度。我们方案的关键特征是在设计时考虑了计算效率:我们提供了与QUIK格式匹配的GPU内核,具有高效的逐层运行时,相对于FP16执行,端到端吞吐量最高可提升3.4倍。我们对OPT、LLaMA-2和Falcon系列模型进行了详细研究,并首次给出了使用量化加2:4稀疏性实现准确推理的实例。代码可在:https://github.com/IST-DASLab/QUIK 获取。

英文摘要:

Large Language Models (LLMs) from the GPT family have become extremely popular, leading to a race towards reducing their inference costs to allow for efficient local computation. Yet, the vast majority of existing work focuses on weight-only quantization, which can reduce runtime costs in the memory-bound one-token-at-a-time generative setting, but does not address them in compute-bound scenarios, such as batched inference or prompt processing. In this paper, we address the general quantization problem, where both weights and activations should be quantized. We show, for the first time, that the majority of inference computations for large generative models such as LLaMA, OPT, and Falcon can be performed with both weights and activations being cast to 4 bits, in a way that leads to practical speedups, while at the same time maintaining good accuracy. We achieve this via a hybrid quantization strategy called QUIK, which compresses most of the weights and activations to 4-bit, while keeping some outlier weights and activations in higher-precision. The key feature of our scheme is that it is designed with computational efficiency in mind: we provide GPU kernels matching the QUIK format with highly-efficient layer-wise runtimes, which lead to practical end-to-end throughput improvements of up to 3.4x relative to FP16 execution. We provide detailed studies for models from the OPT, LLaMA-2 and Falcon families, as well as a first instance of accurate inference using quantization plus 2:4 sparsity. Code is available at: https://github.com/IST-DASLab/QUIK.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑