arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.16997cs.LGcs.AI
收藏

INT-FlashAttention:为INT8量化启用Flash Attention

INT-FlashAttention: Enabling Flash Attention for INT8 Quantization

Peking University(北京大学);Baichuan Inc(百川智能);Beihang University(北京航空航天大学)

Shimao Chen, Zirui Liu, Zhiying Wu, Ce Zheng, Peizhuang Cong, Zihan Jiang, Yuhan Wu, Lei Su, Tong Yang

AI总结:

本文提出INT-FlashAttention,首个与FlashAttention前向工作流兼容的INT8量化架构,通过全INT8激活和GEMM内核实现,在Ampere GPU上显著提升推理速度,同时支持INT4等数据格式,实验显示推理速度提升72%,量化误差减少82%。

摘要:

作为大语言模型(LLMs)的基础,自注意力模块面临序列长度带来的二次时间和内存复杂度挑战。FlashAttention通过利用GPU内存层次结构加速注意力计算并降低内存占用。一个有前景的研究方向是将FlashAttention与量化方法相结合。本文提出INT-FlashAttention,这是首个与FlashAttention前向工作流兼容的INT8量化架构,显著提升了FlashAttention在Ampere GPU上的推理速度。我们实现了全INT8激活和通用矩阵乘法(GEMM)内核的INT-FlashAttention原型,使其成为首个具有全INT8输入的注意力算子。作为通用的token级训练后量化框架,INT-FlashAttention还兼容INT4等其他数据格式。实验结果表明,与使用FP16和FP8数据格式的标准FlashAttention相比,INT-FlashAttention实现了72%的推理速度提升和82%的量化误差减少。

英文摘要:

As the foundation of large language models (LLMs), self-attention module faces the challenge of quadratic time and memory complexity with respect to sequence length. FlashAttention accelerates attention computation and reduces its memory usage by leveraging the GPU memory hierarchy. A promising research direction is to integrate FlashAttention with quantization methods. This paper introduces INT-FlashAttention, the first INT8 quantization architecture compatible with the forward workflow of FlashAttention, which significantly improves the inference speed of FlashAttention on Ampere GPUs. We implement our INT-FlashAttention prototype with fully INT8 activations and general matrix-multiplication (GEMM) kernels, making it the first attention operator with fully INT8 input. As a general token-level post-training quantization framework, INT-FlashAttention is also compatible with other data formats like INT4, etc. Experimental results show INT-FlashAttention achieves 72% faster inference speed and 82% smaller quantization error compared to standard FlashAttention with FP16 and FP8 data format.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑