S$^4$R：用于压缩长上下文KV缓存的选择性采样、子空间与稀疏重构
S$^4$R: Selective Sampling, Subspaces, and Sparse Reconstruction for Compressed Long-Context KV Caching
浏览论文内容
中文总结 AI 辅助
S$^4$R通过选择性采样token构建低秩子空间、稀疏重构KV表示，在LongBench等数据集上实现最高5倍KV压缩且接近全缓存准确率，兼顾压缩效率与提示词适应性。
中文摘要 AI 辅助
大型语言模型（LLMs）上下文窗口长度的增长显著提升了其长上下文能力，但因键值（KV）缓存产生了过高的内存成本。尽管KV缓存的低秩压缩是有前景的解决方案，现有方法却面临两难：离线方法依赖外部校准数据，在线方法则因完整提示词分解与重构产生大量计算开销。本文提出S$^4$R，它从选择性采样的token构建低秩子空间，并在稀疏重构的KV表示上计算注意力。S$^4$R采用提示词感知初始化，从代表性提示词子集构建初始键/值基，权衡校准数据依赖与预填充成本。由于在每个解码步骤完全重构缓存成本过高且会降低吞吐量，我们进一步采用稀疏重构，在解码期间仅保留信息性位置。在LongBench和RULER数据集上针对Llama和Qwen模型系列开展的大量实验表明，S$^4$R实现了高达5倍的KV压缩，同时达到接近全缓存的准确率，结合了固定压缩的效率与依赖提示词方法的适应性。
英文摘要
The growth of context window lengths in Large Language Models (LLMs) significantly enhances their long-context capabilities but incurs prohibitive memory costs due to the Key-Value (KV) cache. Although low-rank compression of KV cache is a promising remedy, existing methods face a dilemma: offline approaches depend on external calibration data, whereas online approaches incur substantial compute for full-prompt decomposition and reconstruction. In this paper, we propose S$^4$R, which builds low-rank subspaces from selectively sampled tokens and computes attention over a sparsely reconstructed KV representation. S$^4$R uses prompt-aware initialization to build initial key/value bases from a representative prompt subset, trading off calibration-data dependence against prefilling cost. Because fully reconstructing the cache at every decoding step is prohibitively expensive and hurts throughput, we further adopt sparse reconstruction to retain only informative positions during decoding. Extensive experiments on LongBench and RULER with Llama and Qwen model families show that S$^4$R achieves up to 5$\times$ KV compression with near full-cache accuracy, combining the efficiency of fixed compression with the adaptability of prompt-dependent methods.
发表机构
- School of Information Science and Technology, ShanghaiTech University（上海科技大学信息科学与技术学院）
- Shanghai Engineering Research Center of Intelligent Vision and Imaging（上海智能视觉与成像工程研究中心）
机构由 AI 辅助整理，请以论文原文为准。
