利用DRAM与SSD实现可持续且可访问的LLM推理:混合精度与多级缓存
Harnessing Your DRAM and SSD for Sustainable and Accessible LLM Inference with Mixed-Precision and Multi-level Caching
AI总结:
本文提出M2Cache方法,通过神经元模块化、动态稀疏混合精度量化以及HBM-DRAM-SSD三级缓存管理,在老旧GPU上实现LLM高效推理,从而显著降低碳排放。
摘要:
尽管大语言模型(LLMs)已展现出卓越的能力,但其庞大的参数量及随之而来的大量计算使得LLM的部署成为当今AI应用碳排放的主要来源。与现代GPU(如H100)相比,如果我们能够利用老式GPU(如M40)进行LLM服务,将显著更具碳可持续性(如图1所示,M40的碳排放仅为H100的三分之一)。然而,这类GPU上有限的高带宽内存(HBM)往往无法支持LLM的加载,原因在于巨大的模型规模和中间激活数据,这使得其服务面临挑战。例如,一个具有70B参数的LLaMA2模型通常需要128GB内存进行推理,这大幅超过了3090 GPU中24GB的HBM,即使考虑额外的64GB DRAM也仍然不可行。为应对这一挑战,本文提出了一种混合精度与模型模块化算法,以在资源受限的过时硬件上实现LLM推理。(精度指FP16、INT8、INT4等数值精度)以及多级缓存(M2Cache)。具体而言,我们的M2Cache首先对LLM中的神经元进行模块化,并建立其重要性排序。然后,它在权重空间中采用动态稀疏混合精度量化机制,以降低每个解码步骤的计算需求和通信开销。它共同降低了与LLM推理相关的运营碳排放。此外,M2Cache引入了一个由HBM、DRAM和SSD组成的三级缓存管理系统,以补充动态稀疏混合精度推理。为了提高通信效率,M2Cache在HBM中维护一个神经元级混合精度LRU缓存,在DRAM中维护一个更大的层感知缓存,并在SSD中保存完整模型。
英文摘要:
Although Large Language Models (LLMs) have demonstrated remarkable capabilities, their massive parameter counts and associated extensive computing make LLMs' deployment the main part of carbon emission from nowadays AI applications. Compared to modern GPUs like H$100$, it would be significantly carbon-sustainable if we could leverage old-fashioned GPUs such as M$40$ (as shown in Figure 1, M$40$ only has one third carbon emission of H$100$'s) for LLM servings. However, the limited High Bandwidth Memory (HBM) available on such GPU often cannot support the loading of LLMs due to the gigantic model size and intermediate activation data, making their serving challenging. For instance, a LLaMA2 model with $70$B parameters typically requires $128$GB for inference, which substantially surpasses $24$GB HBM in a $3090$ GPU and remains infeasible even considering the additional $64$GB DRAM. To address this challenge, this paper proposes a mixed-precision with a model modularization algorithm to enable LLM inference on outdated hardware with resource constraints. (The precision denotes the numerical precision like FP16, INT8, INT4) and multi-level caching (M2Cache).) Specifically, our M2Cache first modulizes neurons in LLM and creates their importance ranking. Then, it adopts a dynamic sparse mixed-precision quantization mechanism in weight space to reduce computational demands and communication overhead at each decoding step. It collectively lowers the operational carbon emissions associated with LLM inference. Moreover, M2Cache introduces a three-level cache management system with HBM, DRAM, and SSDs that complements the dynamic sparse mixed-precision inference. To enhance communication efficiency, M2Cache maintains a neuron-level mixed-precision LRU cache in HBM, a larger layer-aware cache in DRAM, and a full model in SSD.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。