arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2410.13461cs.LGcs.CL
收藏

渐进式混合精度解码用于高效LLM推理

Progressive Mixed-Precision Decoding for Efficient LLM Inference

Samsung AI Center(三星人工智能中心);Imperial College London(帝国理工学院)

Hao Mark Chen, Fuwen Tan, Alexandros Kouris, Royson Lee, Hongxiang Fan, Stylianos I. Venieris

AI总结:

针对LLM推理中统一量化策略忽视不同阶段特性导致性能下降的问题,提出相位感知的渐进式混合精度解码(PMPD),在预填充和解码阶段动态分配精度,实现显著加速并保持输出质量。

摘要:

尽管大型语言模型(LLM)在各种任务中展现出巨大潜力,但由于其过高的计算和内存需求,在资源受限设备上的部署仍然具有挑战性。量化通过以降低的精度存储权重,已成为一种有效的解决方案。然而,利用低精度(即2/3位)来大幅缓解LLM解码的内存受限性,仍然面临严重的性能下降。在这项工作中,我们认为现有方法未能探索LLM推理不同阶段的计算模式、冗余性和对近似敏感性的多样性,而是全程采用统一的量化策略。相反,我们提出了一种新颖的相位感知方法,在LLM推理的不同阶段选择性地分配精度,从而在预填充阶段实现强大的上下文提取,并在解码阶段实现高效的内存带宽利用。为了进一步解决解码阶段的内存受限性,我们引入了渐进式混合精度解码(PMPD),该技术能够在生成的序列中逐渐降低精度,并提供一系列精度切换调度器,以任务自适应或提示自适应的方式动态驱动精度降低决策。在多种语言任务上的广泛评估表明,当面向Nvidia GPU时,PMPD在矩阵-向量乘法上相比fp16模型实现了1.4至12.2倍的加速;而当面向LLM优化的NPU时,我们的方法相比fp16模型实现了3.8至8.0倍的吞吐量提升,相比统一量化方法最高提升1.54倍,同时保持了输出质量。

英文摘要:

In spite of the great potential of large language models (LLMs) across various tasks, their deployment on resource-constrained devices remains challenging due to their excessive computational and memory demands. Quantization has emerged as an effective solution by storing weights in reduced precision. However, utilizing low precisions (i.e.~2/3-bit) to substantially alleviate the memory-boundedness of LLM decoding, still suffers from prohibitive performance drop. In this work, we argue that existing approaches fail to explore the diversity in computational patterns, redundancy, and sensitivity to approximations of the different phases of LLM inference, resorting to a uniform quantization policy throughout. Instead, we propose a novel phase-aware method that selectively allocates precision during different phases of LLM inference, achieving both strong context extraction during prefill and efficient memory bandwidth utilization during decoding. To further address the memory-boundedness of the decoding phase, we introduce Progressive Mixed-Precision Decoding (PMPD), a technique that enables the gradual lowering of precision deeper in the generated sequence, together with a spectrum of precision-switching schedulers that dynamically drive the precision-lowering decisions in either task-adaptive or prompt-adaptive manner. Extensive evaluation across diverse language tasks shows that when targeting Nvidia GPUs, PMPD achieves 1.4$-$12.2$\times$ speedup in matrix-vector multiplications over fp16 models, while when targeting an LLM-optimized NPU, our approach delivers a throughput gain of 3.8$-$8.0$\times$ over fp16 models and up to 1.54$\times$ over uniform quantization approaches while preserving the output quality.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑