Beluga：基于CXL的内存架构用于可扩展和高效的LLM KVCache管理
Beluga: A CXL-Based Memory Architecture for Scalable and Efficient LLM KVCache Management
AI总结：
Beluga通过CXL交换机实现GPU与CPU共享大规模内存池，显著提升LLM推断中的KVCache管理效率。
摘要：
随着LLM模型规模的迅速增长以及对长上下文推断需求的增加，内存已成为GPU加速服务系统中的关键瓶颈。尽管GPU上的高带宽内存（HBM）提供快速访问，但其容量有限，需要依赖主机内存（CPU DRAM）来支持更大的工作集，如KVCache。然而，DRAM的最大容量受限于每个CPU插槽的内存通道数量有限。为克服这一限制，当前系统常常采用基于RDMA的离散内存池，这引入了显著的挑战，包括高访问延迟、复杂的通信协议以及同步开销。幸运的是，新兴的CXL技术在KVCache设计中引入了新的机会。在本文中，我们提出了Beluga，一种新的内存架构，使GPU和CPU能够通过CXL交换机访问共享的大规模内存池。通过支持CXL织物上的原生加载/存储访问语义，我们的设计实现了接近本地内存延迟，同时减少了编程复杂性并最小化了同步开销。我们系统地表征了一个商业CXL交换机基于的内存池，并提出了一组设计指南。基于Beluga，我们设计并实现了Beluga-KVCache，一个专门用于管理LLM推断中的大规模KVCache的系统。Beluga-KVCache在vLLM推断引擎中相比基于RDMA的解决方案，实现了TTFT的89.6%减少和7.35倍的吞吐量提升。据我们所知，Beluga是首个使GPU能够通过CXL交换机直接访问大规模内存池的系统，标志着向通过GPU实现低延迟、共享访问大量内存资源的重要一步。
英文摘要：
The rapid increase in LLM model sizes and the growing demand for long-context inference have made memory a critical bottleneck in GPU-accelerated serving systems. Although high-bandwidth memory (HBM) on GPUs offers fast access, its limited capacity necessitates reliance on host memory (CPU DRAM) to support larger working sets such as the KVCache. However, the maximum DRAM capacity is constrained by the limited number of memory channels per CPU socket. To overcome this limitation, current systems often adopt RDMA-based disaggregated memory pools, which introduce significant challenges including high access latency, complex communication protocols, and synchronization overhead. Fortunately, the emerging CXL technology introduces new opportunities in KVCache design. In this paper, we propose Beluga, a novel memory architecture that enables GPUs and CPUs to access a shared, large-scale memory pool through CXL switches. By supporting native load/store access semantics over the CXL fabric, our design delivers near-local memory latency, while reducing programming complexity and minimizing synchronization overhead. We conduct a systematic characterization of a commercial CXL switch-based memory pool and propose a set of design guidelines. Based on Beluga, we design and implement Beluga-KVCache, a system tailored for managing the large-scale KVCache in LLM inference. Beluga-KVCache achieves an 89.6% reduction in Time-To-First-Token (TTFT) and 7.35x throughput improvement in the vLLM inference engine compared to RDMA-based solutions. To the best of our knowledge, Beluga is the first system that enables GPUs to directly access large-scale memory pools through CXL switches, marking a significant step toward low-latency, shared access to vast memory resources by GPUs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
