arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2507.08045cs.CLcs.AI
收藏

Krul:基于动态跨层KV共享的多轮对话高效状态恢复

Krul: Efficient State Restoration for Multi-turn Conversations with Dynamic Cross-layer KV Sharing

Sun Yat-sen University(中山大学);Hong Kong University of Science and Technology(香港理工大学);Peng Cheng Laboratory(鹏城实验室);Hubei University of Technology(湖北工业大学)

Junyi Wen, Junyuan Liang, Zicong Hong, Wuhui Chen, Ting Cai, Zibin Zheng

AI总结:

针对多轮对话中LLM状态恢复因KV缓存开销大的问题,提出Krul系统,通过动态跨层KV共享、定制压缩策略等方法,在不降低精度的前提下提升恢复效率并减少存储开销。

摘要:

大语言模型(LLMs)在多轮对话中的高效状态恢复仍是一项关键挑战,主要源于为所有历史token重新计算或加载完整键值(KV)缓存的开销。为此,现有方法针对相邻层中高度相似的注意力模式压缩KV缓存。但这些方法常对所有对话应用固定压缩方案,选择相同层对进行压缩,未考虑对话特定的注意力动态。这种静态策略忽略了不同对话间注意力模式相似性的差异,可能导致明显的精度下降。我们提出Krul,一种多轮LLM推理系统,可实现准确高效的KV缓存恢复。Krul基于层对间的注意力相似性动态选择压缩策略,并使用重计算-加载流水线恢复KV缓存。它引入三大关键创新:1)一种抢占式压缩策略选择器,为未来对话轮次保留关键上下文,并为对话选择定制策略;2)一种token级异构注意力相似性估计器,以减轻模型生成期间注意力相似性计算和存储的开销;3)一种无气泡恢复调度器,减少因压缩KV缓存导致的重计算和加载流不平衡带来的潜在气泡。对真实世界任务的实证评估表明,与最先进方法相比,Krul在不影响生成质量的情况下,将首token时间(TTFT)减少1.5倍至2.68倍,KV缓存存储减少1.33倍至2.35倍。

英文摘要:

Efficient state restoration in multi-turn conversations with large language models (LLMs) remains a critical challenge, primarily due to the overhead of recomputing or loading full key-value (KV) caches for all historical tokens. To address this, existing approaches compress KV caches across adjacent layers with highly similar attention patterns. However, these methods often apply a fixed compression scheme across all conversations, selecting the same layer pairs for compression without considering conversation-specific attention dynamics. This static strategy overlooks variability in attention pattern similarity across different conversations, which can lead to noticeable accuracy degradation. We present Krul, a multi-turn LLM inference system that enables accurate and efficient KV cache restoration. Krul dynamically selects compression strategies based on attention similarity across layer pairs and uses a recomputation-loading pipeline to restore the KV cache. It introduces three key innovations: 1) a preemptive compression strategy selector to preserve critical context for future conversation turns and selects a customized strategy for the conversation; 2) a token-wise heterogeneous attention similarity estimator to mitigate the attention similarity computation and storage overhead during model generation; 3) a bubble-free restoration scheduler to reduce potential bubbles brought by the imbalance of recomputing and loading stream due to compressed KV caches. Empirical evaluations on real-world tasks demonstrate that Krul achieves a 1.5x-2.68x reduction in time-to-first-token (TTFT) and a 1.33x-2.35x reduction in KV cache storage compared to state-of-the-art methods without compromising generation quality.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑