arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2606.01751cs.PF
收藏

SparseX: 面向交错式LLM服务的高效段级KV缓存共享

SparseX: Efficient Segment-Level KV Cache Sharing for Interleaved LLM Serving

MemTensor (Shanghai) Technology Co., Ltd.(MemTensor(上海)科技有限公司)

Quqing Zhang, Kai Chen, Ning Liao, Zehao Lin, Bo Tang, Feiyu Xiong, Zhiyu Li, Xiaoxing Wang

AI总结:

针对长上下文LLM服务中非前缀重复内容导致的缓存效率低问题,提出SparseX,一种基于稀疏Q索引的段级KV缓存共享方法,通过单次前向传播中的稀疏KV重计算恢复跨段上下文交互,实现无额外模型或预处理的缓存复用。

摘要:

在长上下文LLM服务中,预填充阶段通常主导首字生成时间和计算成本。尽管vLLM/PagedAttention中的前缀缓存已被广泛用于重用相同的提示前缀,但实际应用中的重复内容经常以非前缀、跨请求、跨轮次和跨代理片段的形式出现,这使得传统缓存机制不足。本文提出SparseX,一种针对常见服务场景的段级KV缓存共享方法。SparseX使用连续令牌段作为重用单元,并利用KV缓存重用工作负载中自然出现的稀疏Q索引来估计需要校正的关键令牌。基于此估计,SparseX在单次前向传播内执行稀疏KV重计算,从而在复杂的交错重用模式下恢复跨段上下文交互,同时避免额外的模型或单独的令牌选择预处理阶段。SparseX进一步基于层特定阈值实现全+稀疏混合注意力模式:早期层保留全注意力以获得更稳定的令牌重要性信号,后期层切换到稀疏重计算以提高复杂长上下文任务的重用质量。我们在vLLM之上实现SparseX-vLLM,将段级缓存查找、PagedAttention管理、RoPE对齐、稀疏Q令牌选择和FlashAttention后端集成到统一执行路径中。SparseX是模型无关、无需训练且与前缀缓存兼容的,并为常见的在线服务场景(包括多轮聊天、检索增强生成(RAG)和代理工作流)提供统一支持。

英文摘要:

In long-context LLM serving, the prefill stage often dominates time-to-first-token and computational cost. Although Prefix Cache in vLLM/PagedAttention has been widely used to reuse identical prompt prefixes, repeated content in practical applications frequently appears as non-prefix, cross-request, cross-turn, and cross-agent segments, which makes conventional cache mechanisms insufficient. This paper presents SparseX, a segment-level KV Cache sharing method for common serving scenarios. SparseX uses contiguous token segments as reuse units and exploits Sparse-Q indices that naturally arise in KV Cache reuse workloads to estimate the key tokens that require correction. Based on this estimate, SparseX performs Sparse-KV Recomputation within a single forward pass, thereby restoring cross-segment contextual interactions under complex interleaved reuse patterns while avoiding additional models or separate preprocessing stages for token selection. SparseX further implements a full+sparse hybrid attention mode based on a layer-specific threshold: early layers retain full attention to obtain a more stable token-importance signal, and later layers switch to sparse recomputation to improve reuse quality on complex long-context tasks. We implement SparseX-vLLM on top of vLLM, integrating segment-level cache lookup, PagedAttention management, RoPE alignment, Sparse-Q token selection, and FlashAttention backends into a unified execution path. SparseX is model-agnostic, training-free, and compatible with Prefix Cache, and it provides unified support for common online serving scenarios including multi-round chat, retrieval-augmented generation (RAG), and agent workflows.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑