在量化LLM推理中实现动态稀疏性
Enabling Dynamic Sparsity in Quantized LLM Inference
AI总结：
本文提出一套在低比特量化下实现动态稀疏推理的技术，包括锯齿形量化布局、专用GEMV内核和紧凑的稀疏索引收集机制，在多个模型规模和硬件配置上实现高达1.55倍的解码吞吐量提升，同时保持与稠密量化推理相当的准确率。
摘要：
在终端用户设备上部署大语言模型（LLMs）由于在响应速度、隐私和运营成本方面的优势而日益重要。然而，移动和桌面GPU有限的内存与计算能力使得高效执行变得困难。近期观察表明，LLMs的内部激活通常是动态稀疏的，这意味着对于每个输入，只有网络的一部分对输出有显著贡献。这种稀疏性可以减少计算量，但它与分组量化（group-wise quantization）的交互效果不佳，而分组量化仍然是将LLMs适配到资源受限硬件上的主流方法。为了协调这两个特性，本研究提出了一套在低比特量化下实现动态稀疏推理的技术。该方法的特点包括：（1）一种锯齿形（zigzag-patterned）量化布局，以与激活稀疏性一致的方式组织权重，并改善GPU内存局部性；（2）一个为此布局专门设计的GEMV内核，以充分利用并行计算单元；（3）一种紧凑的运行时机制，以最小的开销收集稀疏索引。在多个模型规模和硬件配置上，该方法实现了高达1.55倍的解码吞吐量提升，同时保持了与稠密量化推理相当的准确率，表明结构化稀疏性和量化可以在商用GPU上有效共存。
英文摘要：
Deploying large language models (LLMs) on end-user devices is gaining importance due to benefits in responsiveness, privacy, and operational cost. Yet the limited memory and compute capability of mobile and desktop GPUs make efficient execution difficult. Recent observations suggest that the internal activations of LLMs are often dynamically sparse, meaning that for each input, only part of the network contributes significantly to the output. Such sparsity could reduce computation, but it interacts poorly with group-wise quantization, which remains the dominant approach for fitting LLMs onto resource-constrained hardware. To reconcile these two properties, this study proposes a set of techniques that realize dynamic sparse inference under low-bit quantization. The method features: (1) a zigzag-patterned quantization layout that organizes weights in a way consistent with activation sparsity and improves GPU memory locality; (2) a specialized GEMV kernel designed for this layout to fully utilize parallel compute units; and (3) a compact runtime mechanism that gathers sparse indices with minimal overhead. Across several model scales and hardware configurations, the approach achieves up to 1.55x faster decoding throughput while maintaining accuracy comparable to dense quantized inference, showing that structured sparsity and quantization can effectively coexist on commodity GPUs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
