ContiguousKV: 通过粒度对齐的KV缓存管理加速LLM预填
ContiguousKV: Accelerating LLM Prefill with Granularity-Aligned KV Cache Management
AI总结:
ContiguousKV通过粒度对齐的KV缓存管理,提升LLM预填效率,实现3.85倍加速并保持输出质量。
摘要:
高效地为大型语言模型(LLM)提供持久化的前缀键值(KV)缓存对于应用如对话搜索和多轮对话至关重要。处理一个请求需要加载预计算的前缀KV缓存并生成第一个词,这被称为Re-Prefill阶段。将共享前缀缓存卸载到二级存储是内存扩展的关键。Re-Prefill卸载在两个方面面临严重的I/O瓶颈。首先,语义感知的KV缓存修剪算法在细粒度上选择重要的词,而系统在粗粒度、固定大小的块上管理I/O,导致严重的读取放大。其次,识别重要词与加载KV缓存之间的顺序依赖性会创建空闲I/O和计算气泡,浪费系统资源。本文提出了ContiguousKV,一个高性能的前缀KV缓存卸载系统,通过将算法语义与I/O效率相结合,以加速Re-Prefill阶段。我们首先引入ContiguousChunk,一种统一的数据管理粒度,使KV缓存修剪与I/O操作对齐。所有对I/O性能至关重要的机制都在ContiguousChunk的粒度上执行,从而消除读取放大。通过利用重要ContiguousChunk索引在各层之间的高相似性,我们提出内周期和外周期异步预取,打破I/O和计算之间的顺序依赖性,有效消除空闲气泡。最后,我们提出注意力引导的缓存管理,以在内存中保留语义关键的前缀数据。在Qwen2.5系列模型上的评估表明,ContiguousKV在Re-Prefill阶段相对于最先进的卸载系统IMPRESS实现了3.85倍的加速,同时保持高质量的输出。
英文摘要:
Efficiently serving Large Language Models (LLMs) with persistent Prefix Key-Value (KV) Cache is critical for applications like conversational search and multi-turn dialogue. Serving a request requires loading the pre-computed prefix KV cache and generating the first token, defined as the Re-Prefill Phase. Offloading this shared prefix cache to secondary storage is essential for memory scalability. Re-Prefill with offloading suffers from severe I/O bottlenecks in two aspects. First, semantic-aware KV cache pruning algorithms select important tokens in fine granularity, while systems manage I/O in coarse, fixed-size blocks, causing severe read amplification. Second, the sequential dependency between identifying important tokens and loading KV cache creates idle I/O and compute bubbles, under-utilizing system resources. This paper proposes \textit{ContiguousKV}, a high-performance prefix KV cache offloading system that bridges algorithmic semantics with I/O efficiency to accelerate the Re-Prefill phase. We first introduce \textit{ContiguousChunk}, a unified data management granularity that aligns KV cache pruning with I/O operations. All the mechanisms critical for I/O performance are performed at the granularity of ContiguousChunk, thereby eliminating read amplification. By exploiting the high similarity in important ContiguousChunk indices across layers, we propose intra- and inter-period asynchronous prefetching to break the sequential dependency between I/O and compute, effectively eliminating idle bubbles. Finally, we propose attention-guided cache management to retain semantically critical prefix data in memory. Evaluations on Qwen2.5 series models show that ContiguousKV achieves a 3.85x speedup in the Re-Prefill phase over the state-of-the-art offloading system IMPRESS, while maintaining high output quality.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。