MInference 1.0：通过动态稀疏注意力加速长上下文大语言模型的预填充阶段
MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention
AI总结：
针对长上下文大语言模型预填充阶段计算效率低的问题，提出MInference动态稀疏注意力方法，利用三种注意力模式优化GPU计算，无需微调即可将预填充延迟最高降低10倍并维持准确率。
摘要：
大语言模型（LLM）推理的计算挑战仍是其广泛部署的重大阻碍，尤其是在提示词长度持续增长的背景下。由于注意力计算的二次复杂度，单张A100 GPU上，一个8B参数的LLM处理100万token的提示词（即预填充阶段）需要30分钟。现有加速预填充的方法应用于长上下文LLM时，往往无法维持可接受的准确率或效率。为填补这一空白，我们提出MInference（百万token推理），这是一种专为加速长序列处理的预填充阶段设计的稀疏计算方法。具体而言，我们识别出长上下文注意力矩阵中的三种独特模式——A形、竖斜形和块稀疏模式，可用于在GPU上实现高效稀疏计算。我们离线确定每个注意力头的最优模式，并在推理过程中根据分配的模式动态构建稀疏索引。借助该模式和稀疏索引，我们通过优化的GPU内核执行高效的稀疏注意力计算，显著降低长上下文LLM预填充阶段的延迟。我们提出的技术可直接应用于现有LLM，无需修改预训练设置或额外微调。通过在InfiniteBench、RULER、PG-19和大海捞针（Needle In A Haystack）等广泛下游任务，以及LLaMA-3-1M、GLM4-1M、Yi-200K、Phi-3-128K和Qwen2-128K等模型上进行评估，我们证明MInference在A100上可将预填充推理延迟最高降低10倍，同时保持准确率。我们的代码可在https://aka.ms/MInference获取。
英文摘要：
The computational challenges of Large Language Model (LLM) inference remain a significant barrier to their widespread deployment, especially as prompt lengths continue to increase. Due to the quadratic complexity of the attention computation, it takes 30 minutes for an 8B LLM to process a prompt of 1M tokens (i.e., the pre-filling stage) on a single A100 GPU. Existing methods for speeding up prefilling often fail to maintain acceptable accuracy or efficiency when applied to long-context LLMs. To address this gap, we introduce MInference (Milliontokens Inference), a sparse calculation method designed to accelerate pre-filling of long-sequence processing. Specifically, we identify three unique patterns in long-context attention matrices-the A-shape, Vertical-Slash, and Block-Sparsethat can be leveraged for efficient sparse computation on GPUs. We determine the optimal pattern for each attention head offline and dynamically build sparse indices based on the assigned pattern during inference. With the pattern and sparse indices, we perform efficient sparse attention calculations via our optimized GPU kernels to significantly reduce the latency in the pre-filling stage of long-context LLMs. Our proposed technique can be directly applied to existing LLMs without any modifications to the pre-training setup or additional fine-tuning. By evaluating on a wide range of downstream tasks, including InfiniteBench, RULER, PG-19, and Needle In A Haystack, and models including LLaMA-3-1M, GLM4-1M, Yi-200K, Phi-3-128K, and Qwen2-128K, we demonstrate that MInference effectively reduces inference latency by up to 10x for pre-filling on an A100, while maintaining accuracy. Our code is available at https://aka.ms/MInference.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
