FlattenQuant：通过逐张量量化突破大型语言模型的推理计算瓶颈
FlattenQuant: Breaking Through the Inference Compute-bound for Large Language Models with Per-tensor Quantization
AI总结：
针对LLM大批量、长序列推理的计算瓶颈，提出FlattenQuant逐张量低比特量化方法，通过展平大通道降低张量最大值，实现最高2倍加速与2.3倍显存降低。
摘要：
大型语言模型（LLM）已在各类任务中展现出最先进的性能。然而，LLM的推理延迟和巨大的GPU显存消耗限制了其部署性能。近年来，已有一些针对LLM量化的高效尝试，但在大批量或长序列推理时仍存在计算受限的问题。细粒度量化方法已证明其能够熟练实现LLM的低比特量化，却要求线性层计算使用FP16数据类型，这在处理大批量或长序列时非常耗时。本文提出一种名为FlattenQuant的方法，通过将张量中的大通道展平来显著降低张量的最大值，从而以最小的精度损失实现低比特逐张量量化。实验表明，FlattenQuant可直接使用4比特完成LLM中48.29%的线性层计算，其余层使用8比特。FlattenQuant方法引入的4比特矩阵乘法能够有效解决由大矩阵计算导致的计算受限问题。本工作为LLM实现了最高2倍的加速和2.3倍的显存降低，同时精度损失可忽略不计。
英文摘要：
Large language models (LLMs) have demonstrated state-of-the-art performance across various tasks. However, the latency of inference and the large GPU memory consumption of LLMs restrict their deployment performance. Recently, there have been some efficient attempts to quantize LLMs, yet inference with large batch size or long sequence still has the issue of being compute-bound. Fine-grained quantization methods have showcased their proficiency in achieving low-bit quantization for LLMs, while requiring FP16 data type for linear layer computations, which is time-consuming when dealing with large batch size or long sequence. In this paper, we introduce a method called FlattenQuant, which significantly reduces the maximum value of the tensor by flattening the large channels in the tensor, to achieve low bit per-tensor quantization with minimal accuracy loss. Our experiments show that FlattenQuant can directly use 4 bits to achieve 48.29% of the linear layer calculation in LLMs, with the remaining layers using 8 bits. The 4-bit matrix multiplication introduced in the FlattenQuant method can effectively address the compute-bound caused by large matrix calculation. Our work achieves up to 2$\times$ speedup and 2.3$\times$ memory reduction for LLMs with negligible loss in accuracy.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
