DynSplit-KV: 动态语义分割用于高效长上下文LLM推理中的KV缓存压缩
DynSplit-KV: Dynamic Semantic Splitting for KVCache Compression in Efficient Long-Context LLM Inference
AI总结:
DynSplit-KV通过动态语义分割和统一映射策略,提升长上下文LLM推理的准确性与效率,减少内存占用。
摘要:
尽管键值(KV)缓存对于高效的大语言模型(LLM)推理至关重要,但在长上下文场景中其增长的内存足迹成为一个显著瓶颈,使KV缓存压缩变得至关重要。当前的压缩方法依赖于刚性的分割策略,如固定间隔或预定义的分隔符。我们观察到,刚性分割在不同场景中会显著降低准确性(范围从5.5%到55.1%),这是因为语义边界具有场景依赖性。这突显了动态语义分割以匹配语义的必要性。为此,我们面临两个挑战。(1)不恰当的分隔符选择会导致语义与KV缓存不一致,造成28.6%的准确性损失。(2)分割后的可变长度块引入了超过73.1%的额外推理开销。为了解决上述挑战,我们提出了DynSplit-KV,一种KV缓存压缩方法,能够动态地识别分割的分隔符。我们提出:(1)一种动态重要性感知的分隔符选择策略,提高了49.9%的准确性。(2)一种统一的映射策略,将可变长度的语义块转换为固定长度格式,减少了推理开销4.9倍。实验表明,DynSplit-KV在长上下文场景中实现了最高准确性,比FlashAttention快2.2倍,峰值内存减少2.6倍。
英文摘要:
Although Key-Value (KV) Cache is essential for efficient large language models (LLMs) inference, its growing memory footprint in long-context scenarios poses a significant bottleneck, making KVCache compression crucial. Current compression methods rely on rigid splitting strategies, such as fixed intervals or pre-defined delimiters. We observe that rigid splitting suffers from significant accuracy degradation (ranging from 5.5% to 55.1%) across different scenarios, owing to the scenario-dependent nature of the semantic boundaries. This highlights the necessity of dynamic semantic splitting to match semantics. To achieve this, we face two challenges. (1) Improper delimiter selection misaligns semantics with the KVCache, resulting in 28.6% accuracy loss. (2) Variable-length blocks after splitting introduce over 73.1% additional inference overhead. To address the above challenges, we propose DynSplit-KV, a KVCache compression method that dynamically identifies delimiters for splitting. We propose: (1) a dynamic importance-aware delimiter selection strategy, improving accuracy by 49.9%. (2) A uniform mapping strategy that transforms variable-length semantic blocks into a fixed-length format, reducing inference overhead by 4.9x. Experiments show that DynSplit-KV achieves the highest accuracy, 2.2x speedup compared with FlashAttention and 2.6x peak memory reduction in long-context scenarios.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。