Flash-LLM：利用非结构化稀疏性实现低成本且高效的大规模生成模型推理
Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsity
AI总结：
针对大规模生成模型推理中非结构化稀疏在Tensor Core上利用不足的问题，提出Flash-LLM框架，采用“按稀疏加载、按稠密计算”方法设计高效SpMM内核，显著提升推理速度并降低成本。
摘要：
随着参数规模的快速增长，部署大规模生成模型变得越来越具有挑战性，因为它们通常需要大量的GPU内存消耗和海量计算。非结构化模型剪枝一直是一种常见的方法，可以在保持良好模型精度的同时减少GPU内存占用和整体计算量。然而，现有解决方案未能在现代GPU上提供对非结构化稀疏性的高效支持，尤其是在高度结构化的Tensor Core硬件上。因此，我们提出Flash-LLM，旨在通过在高性能但高度受限的Tensor Core上对非结构化稀疏性提供精细支持，实现低成本和高效的大规模生成模型推理。基于我们的关键观察，即生成模型推理的主要瓶颈是若干细长矩阵乘法，由于计算强度低，Tensor Core在这些运算中会显著利用不足，我们提出了一种通用的“按稀疏加载、按稠密计算”（Load-as-Sparse and Compute-as-Dense）方法，用于非结构化稀疏矩阵乘法。其基本思想是解决显著的显存带宽瓶颈，同时容忍对Tensor Core端到端性能并非关键的多余计算。在此基础上，我们设计了一个有效的基于Tensor Core的非结构化SpMM软件框架，利用片上资源实现高效的稀疏数据提取以及计算与内存访问的重叠。在SpMM内核层面，Flash-LLM显著优于最先进的库，即Sputnik和SparTA，平均分别提升2.9倍和1.5倍。在OPT-30B/66B/175B模型的端到端框架层面，就每GPU每秒生成的token数而言，Flash-LLM相比DeepSpeed和FasterTransformer分别实现了最高3.8倍和3.6倍的提升，同时显著降低了推理成本。
英文摘要：
With the fast growth of parameter size, it becomes increasingly challenging to deploy large generative models as they typically require large GPU memory consumption and massive computation. Unstructured model pruning has been a common approach to reduce both GPU memory footprint and the overall computation while retaining good model accuracy. However, the existing solutions do not provide a highly-efficient support for handling unstructured sparsity on modern GPUs, especially on the highly-structured Tensor Core hardware. Therefore, we propose Flash-LLM for enabling low-cost and highly-efficient large generative model inference with the sophisticated support of unstructured sparsity on high-performance but highly restrictive Tensor Cores. Based on our key observation that the main bottleneck of generative model inference is the several skinny matrix multiplications for which Tensor Cores would be significantly under-utilized due to low computational intensity, we propose a general Load-as-Sparse and Compute-as-Dense methodology for unstructured sparse matrix multiplication. The basic insight is to address the significant memory bandwidth bottleneck while tolerating redundant computations that are not critical for end-to-end performance on Tensor Cores. Based on this, we design an effective software framework for Tensor Core based unstructured SpMM, leveraging on-chip resources for efficient sparse data extraction and computation/memory-access overlapping. At SpMM kernel level, Flash-LLM significantly outperforms the state-of-the-art library, i.e., Sputnik and SparTA by an average of 2.9x and 1.5x, respectively. At end-to-end framework level on OPT-30B/66B/175B models, for tokens per GPU-second, Flash-LLM achieves up to 3.8x and 3.6x improvement over DeepSpeed and FasterTransformer, respectively, with significantly lower inference cost.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
