QUICK:用于高效LLM推理的量化感知交错与无冲突内核
QUICK: Quantization-aware Interleaving and Conflict-free Kernel for efficient LLM inference
AI总结:
本文提出QUICK,一组新型优化CUDA内核,通过离线交错量化权重矩阵以跳过反量化后的共享内存写回,解决了混合精度矩阵乘法的共享内存库冲突问题,在多种NVIDIA GPU上实现了最高1.91倍加速和1.94倍吞吐量提升。
摘要:
我们推出了QUICK,一组用于量化大型语言模型(LLM)高效推理的新型优化CUDA内核。QUICK解决了最先进的混合精度矩阵乘法内核的共享内存库冲突问题。该方法在离线阶段交错LLM的量化权重矩阵,以跳过反量化后的共享内存写回操作。实验表明,在较大批次下,相比AutoAWQ的现有内核实现了高达1.91倍加速,在多种NVIDIA GPU设备上的代表性LLM模型上实现了高达1.94倍的吞吐量提升。
英文摘要:
We introduce QUICK, a group of novel optimized CUDA kernels for the efficient inference of quantized Large Language Models (LLMs). QUICK addresses the shared memory bank-conflict problem of state-of-the-art mixed precision matrix multiplication kernels. Our method interleaves the quantized weight matrices of LLMs offline to skip the shared memory write-back after the dequantization. We demonstrate up to 1.91x speedup over existing kernels of AutoAWQ on larger batches and up to 1.94x throughput gain on representative LLM models on various NVIDIA GPU devices.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。