arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2405.04437cs.LGcs.OS
收藏

vAttention:无需PagedAttention的LLM服务动态内存管理

vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention

Microsoft Research(微软研究院);Indian Institute of Science(印度科学学院)

Ramya Prabhu, Ajay Nayak, Jayashree Mohan, Ramachandran Ramjee, Ashish Panwar

AI总结:

提出vAttention,通过CUDA虚拟内存管理API解耦虚拟与物理内存分配,在保持KV缓存虚拟内存连续性的同时缓解物理内存碎片化,作为PagedAttention的更优替代方案,提升LLM服务吞吐量达1.23倍。

摘要:

PagedAttention是LLM服务系统中用于动态内存分配的流行方法。它支持按需分配GPU内存以缓解KV缓存碎片化——这一现象在早期系统中严重限制了批处理大小(进而影响吞吐量)。然而,在尝试于运行时分配物理内存时,PagedAttention最终将KV缓存的虚拟内存布局从连续变为非连续。这种设计带来了不可忽视的编程和性能开销。我们提出了vAttention——一种在保持虚拟内存中KV缓存连续性的同时缓解物理内存碎片化的方法。我们通过使用CUDA虚拟内存管理API解耦虚拟和物理内存的分配来实现这一点。我们还引入了各种LLM特定的优化,以解决CUDA虚拟内存支持的局限性。总体而言,vAttention是PagedAttention更简单、更便携且更高效的替代方案:它开箱即用地支持各种注意力内核,与使用基于PagedAttention的FlashAttention和FlashInfer内核相比,将LLM服务吞吐量提高了1.23倍。

英文摘要:

PagedAttention is a popular approach for dynamic memory allocation in LLM serving systems. It enables on-demand allocation of GPU memory to mitigate KV cache fragmentation -- a phenomenon that crippled the batch size (and consequently throughput) in prior systems. However, in trying to allocate physical memory at runtime, PagedAttention ends up changing the virtual memory layout of the KV cache from contiguous to non-contiguous. Such a design leads to non-trivial programming and performance overheads. We present vAttention -- an approach that mitigates fragmentation in physical memory while retaining the contiguity of KV cache in virtual memory. We achieve this by decoupling the allocation of virtual and physical memory using CUDA virtual memory management APIs. We also introduce various LLM-specific optimizations to address the limitations of CUDA virtual memory support. Overall, vAttention is a simpler, portable, and performant alternative to PagedAttention: it supports various attention kernels out-of-the-box and improves LLM serving throughput by up to 1.23x compared to the use of PagedAttention-based kernels of FlashAttention and FlashInfer.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑