arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.06003cs.ARcs.LG
收藏

LUT Tensor Core:面向基于 LUT 的低位 LLM 推理的软硬件协同设计

LUT Tensor Core: A Software-Hardware Co-Design for LUT-Based Low-Bit LLM Inference

Imperial College London(帝国理工学院伦敦分校);Microsoft Research London, UK(微软研究院伦敦, UK);Peking University(北京大学);Microsoft Research Beijing, China(微软研究院北京, China);University of Science(科学大学);University of Washington(华盛顿大学);Microsoft Research Seattle, USA(微软研究院西雅图, USA);Shanghai Jiao Tong University Shanghai, China(上海交通大学上海, China);Microsoft Research(微软研究院);Shanghai Jiao Tong University(上海交通大学)

Zhiwen Mo, Lei Wang, Jianyu Wei, Zhichen Zeng, Shijie Cao, Lingxiao Ma, Naifeng Jing, Ting Cao, Jilong Xue, Fan Yang, Mao Yang

AI总结:

针对低位 LLM 中混合精度矩阵乘法的低效问题,本文提出软硬件协同设计 LUT Tensor Core,通过软件优化、专用硬件、指令集与编译优化实现 1.44 倍计算密度和能效提升。

摘要:

大语言模型(LLM)推理变得高度消耗资源,这促使人们转向低位模型权重,以减少内存占用并提高效率。此类低位 LLM 需要混合精度矩阵乘法(mpGEMM),这是一项重要但尚未得到充分研究的操作,涉及较低精度权重与较高精度激活值的乘法。现成硬件并不原生支持该操作,因此产生了间接的、因而低效的基于反量化的实现。本文研究了用于 mpGEMM 的基于查找表(LUT)的方法,并发现传统 LUT 实现无法实现预期收益。为释放基于 LUT 的 mpGEMM 的全部潜力,我们提出 LUT Tensor Core,这是一种面向低位 LLM 推理的软硬件协同设计。LUT Tensor Core 与传统 LUT 设计的区别在于:1)通过基于软件的优化最小化表预计算开销,并通过权重重解释减少表存储;2)基于 LUT 的 Tensor Core 硬件设计,采用拉长的分块形状以最大化表复用,并采用位串行设计支持 mpGEMM 中多种精度组合;3)面向基于 LUT 的 mpGEMM 的新指令集和编译优化。LUT Tensor Core 显著优于现有纯软件 LUT 实现,并且与此前最先进的基于 LUT 的加速器相比,在计算密度和能源效率方面实现了 1.44 倍提升。

英文摘要:

Large Language Model (LLM) inference becomes resource-intensive, prompting a shift toward low-bit model weights to reduce the memory footprint and improve efficiency. Such low-bit LLMs necessitate the mixed-precision matrix multiplication (mpGEMM), an important yet underexplored operation involving the multiplication of lower-precision weights with higher-precision activations. Off-the-shelf hardware does not support this operation natively, leading to indirect, thus inefficient, dequantization-based implementations. In this paper, we study the lookup table (LUT)-based approach for mpGEMM and find that a conventional LUT implementation fails to achieve the promised gains. To unlock the full potential of LUT-based mpGEMM, we propose LUT Tensor Core, a software-hardware co-design for low-bit LLM inference. LUT Tensor Core differentiates itself from conventional LUT designs through: 1) software-based optimizations to minimize table precompute overhead and weight reinterpretation to reduce table storage; 2) a LUT-based Tensor Core hardware design with an elongated tiling shape to maximize table reuse and a bit-serial design to support diverse precision combinations in mpGEMM; 3) a new instruction set and compilation optimizations for LUT-based mpGEMM. LUT Tensor Core significantly outperforms existing pure software LUT implementations and achieves a 1.44$\times$ improvement in compute density and energy efficiency compared to previous state-of-the-art LUT-based accelerators.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑