arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2501.01792cs.DC
收藏

高效LLM推理与混合缓存激活检查点

Throughput-Oriented LLM Inference via KV-Activation Hybrid Caching with a Single GPU

School of Computing, KAIST（韩国科学技术院计算机学院）

Sanghyeon Lee, Hongbeen Kim, Soojin Hwang, Guseul Heo, Minwoo Noh, Jaehyuk Huh

备注：
8 pages, 10 figures
发表信息：
The 43rd IEEE International Conference on Computer Design, 2025
阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

HybridServe通过激活检查点和混合缓存方案，实现高效LLM推理，提升吞吐量2.19倍。

摘要：

近年来，具有巨大模型规模的大型语言模型（LLMs）使用许多GPU来满足内存容量要求，这导致在生成token时产生显著的成本。为了提供成本效益的LLM推理并放宽延迟约束，广泛研究集中在通过利用主机内存扩展GPU内存。然而，利用主机内存的LLM推理引擎常常面临GPU计算单元的利用率不足，因为相当一部分的推理时间用于通过主机-GPU互连加载模型到GPU。为了解决LLM主机内存卸载中的这些挑战，我们引入了HybridServe，一种基于激活缓存的LLM推理系统，采用激活检查点。激活缓存存储在中间推理阶段生成的激活检查点，允许在模型参数从主机内存传输到GPU时快速重新计算KV缓存。与传统方法重新计算KV缓存从头使用token ID不同，激活缓存允许跳过投影和FFN操作。为了在激活重新计算和参数加载开销之间取得平衡，本研究提出了一种KV-激活混合缓存方案，该方案找到最佳的键值和激活缓存比例以调整重新计算时间。我们的系统在卸载模型权重和KV缓存方面比最先进的先前工作实现了2.19倍的吞吐量提升。

英文摘要：

Large language models (LLMs) with growing model sizes use many GPUs to meet memory capacity requirements, incurring substantial costs for token generation. To provide cost-effective LLM inference with relaxed latency constraints, recent studies proposed to expand GPU memory by leveraging the host memory. However, in such host memory offloading, the host-GPU interconnect becomes a performance bottleneck for the transfer of weights and key-value (KV) cache entries, causing the underutilization of GPUs. To address the challenge, we introduce CAPTURE, an LLM inference system with KV-activation hybrid caching. The activation cache stores activation checkpoints instead of keys and values during intermediate inference stages, requiring half of the memory capacity compared to the conventional KV cache. While model parameters are transferred to GPU from host memory, idle GPU resources can be utilized for the partial recomputation. To balance the latency of activation recomputation and parameter loading, our KV-activation hybrid caching scheme determines the optimal ratio between KV and activation caches to manage both recomputation time and data transfer times. CAPTURE achieves 2.19x throughput improvement over the state-of-the-art prior work for offloading both model weights and KV cache.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑