T-MAC：通过查表实现边缘低比特大语言模型部署的CPU复兴
T-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edge
AI总结：
T-MAC通过查表法直接支持混合精度矩阵乘法，无需反量化，实现CPU上低比特LLM高效推理，吞吐量提升4倍、能耗降低70%，推动边缘部署。
摘要：
在边缘设备上部署大语言模型（LLMs）对于增强设备端智能越来越重要。权重量化对于减少设备上LLMs的内存占用至关重要。然而，低比特LLMs在推理过程中需要低精度权重与高精度激活的混合精度矩阵乘法（mpGEMM）。现有系统缺乏对mpGEMM的原生支持，只能通过反量化权重进行高精度计算。这种间接方式可能导致显著的推理开销。在本文中，我们介绍了T-MAC，一种基于查找表（LUT）的创新方法，旨在CPU上实现高效的低比特LLM（即权重量化LLM）推理。T-MAC直接支持mpGEMM而无需反量化，同时消除了乘法并减少了所需的加法。具体而言，T-MAC将传统的数据类型为中心的乘法转换为逐位的查找表操作，并提供统一且可扩展的mpGEMM解决方案。我们基于LUT的内核随权重位宽线性扩展。在低比特Llama和BitNet模型上的评估表明，与llama.cpp相比，T-MAC的吞吐量提升高达4倍，能耗降低70%。对于BitNet-b1.58-3B，T-MAC在M2-Ultra上使用单核实现每秒30个token的生成吞吐量，使用八核实现每秒71个token，在树莓派5等低端设备上达到每秒11个token，显著超过成人平均阅读速度。T-MAC采用基于LUT的计算范式，为在资源受限的边缘设备上实际部署低比特LLMs铺平了道路，同时不损害计算效率。该系统已在https://github.com/microsoft/T-MAC开源。
英文摘要：
The deployment of Large Language Models (LLMs) on edge devices is increasingly important to enhance on-device intelligence. Weight quantization is crucial for reducing the memory footprint of LLMs on devices. However, low-bit LLMs necessitate mixed precision matrix multiplication (mpGEMM) of low precision weights and high precision activations during inference. Existing systems, lacking native support for mpGEMM, resort to dequantize weights for high precision computation. Such an indirect way can lead to a significant inference overhead. In this paper, we introduce T-MAC, an innovative lookup table(LUT)-based method designed for efficient low-bit LLM (i.e., weight-quantized LLM) inference on CPUs. T-MAC directly supports mpGEMM without dequantization, while simultaneously eliminating multiplications and reducing additions required. Specifically, T-MAC transforms the traditional data-type-centric multiplication to bit-wise table lookup, and enables a unified and scalable mpGEMM solution. Our LUT-based kernels scale linearly to the weight bit-width. Evaluated on low-bit Llama and BitNet models, T-MAC demonstrates up to 4x increase in throughput and 70% reduction in energy consumption compared to llama.cpp. For BitNet-b1.58-3B, T-MAC delivers a token generation throughput of 30 tokens/s with a single core and 71 tokens/s with eight cores on M2-Ultra, and 11 tokens/s on lower-end devices like Raspberry Pi 5, which significantly exceeds the adult average reading speed. T-MAC with LUT-based computing paradigm, paves the way for the practical deployment of low-bit LLMs on resource-constrained edge devices without compromising computational efficiency. The system is open-sourced at https://github.com/microsoft/T-MAC .
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
