EchoKV: 通过基于相似性的重建实现高效的KV缓存压缩
EchoKV: Efficient KV Cache Compression via Similarity-Based Reconstruction
AI总结：
本文提出EchoKV框架，支持按需切换全缓存与压缩缓存，利用轻量网络重建丢弃的KV组件，通过层内和层间相似性提升压缩效率，实验显示其在多种压缩比和模型上均优于现有方法，保持短上下文场景的吞吐量。
摘要：
随着键值（KV）缓存内存需求的增加，这对大型语言模型（LLMs）在长上下文应用中构成了显著瓶颈。现有低秩KV压缩方法通过修改模型投影来减少内存占用，限制了在内存充足时切换回标准全缓存推理的灵活性。本文提出EchoKV，一种灵活的KV缓存压缩框架，支持按需从全KV缓存切换到压缩缓存。与传统压缩-解压缩范式不同，EchoKV利用轻量网络从部分子集重建被丢弃的KV组件，利用注意力头间的内在层内和层间相似性。我们进一步引入了一种轻量的两阶段微调策略，仅需在单个A100 GPU上几分钟即可对7B模型进行微调。在LongBench和RULER上的实验结果表明，EchoKV在多种压缩比和基础模型上均优于现有方法，同时在短上下文场景中保持全缓存推理的吞吐量。
英文摘要：
The increasing memory demand of the Key-Value (KV) cache poses a significant bottleneck for Large Language Models (LLMs) in long-context applications. Existing low-rank KV compression methods reduce this footprint by modifying model projections, limiting the flexibility to switch back to standard full-cache inference when sufficient memory is available. In this paper, we propose EchoKV, a flexible KV cache compression framework that supports on-demand transitions from full KV caching to compressed caching. Unlike traditional compression-decompression paradigms, EchoKV utilizes a lightweight network to reconstruct the discarded KV components from a partial subset, exploiting intrinsic inter-layer and intra-layer similarities among attention heads. We further introduce a lightweight two-stage fine-tuning strategy, requiring only a few minutes on a single A100 GPU for a 7B model. Experimental results on LongBench and RULER demonstrate that EchoKV consistently outperforms existing methods across multiple compression ratios and backbone models while preserving the throughput of full-cache inference in short-context scenarios.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
