Meta-Soft: 利用可组合元标记实现上下文保持的KV缓存压缩
Meta-Soft: Leveraging Composable Meta-Tokens for Context-Preserving KV Cache Compression
AI总结:
提出Meta-Soft动态压缩框架,通过可学习正交基矩阵和Gumbel-Softmax选择网络合成元标记,结合注意力流整合机制保留丢弃上下文信息,解决KV缓存压缩中的信息丢失和上下文断裂问题。
摘要:
大型语言模型中使用的KV缓存具有线性增长的时间复杂度,因此当处理长上下文时,LLMs面临内存爆炸和解码效率降低的问题。当前的KV缓存驱逐已成为重要的研究方向;然而,基于固定软标记(例如Judge Q)的现有方法依赖静态参数集作为查询来评估KV对的重要性,因此无法动态适应不同的输入提示,也无法精确捕捉复杂且变化的任务相关性。此外,被驱逐的KV对被永久丢弃,导致不可逆的信息丢失和上下文断裂。为了解决这个问题,我们提出了Meta-Soft,一种基于探针驱动上下文整合的动态压缩框架。具体来说,我们构建了一个带有可学习正交基矩阵$\mathcal{L}$的元库,并使用带有Gumbel-Softmax的选择器网络生成可微分的稀疏组合权重,从而从输入提示特征中动态合成最具针对性的$k$个软标记。我们将这些软标记附加到输入序列的末尾以探针关键信息。我们还引入了一种基于注意力流的整合机制,该机制将移除标记的语义信息重新分配到保留标记中,从而有效保持被丢弃的上下文信息。在多个数据集上的实验表明,我们的方法优于现有的最先进驱逐方法,并为KV缓存压缩提供了新的解决方案。
英文摘要:
The KV cache used in large language models has linearly growing time complexity, so LLMs face memory blow-up and reduced decoding efficiency when they process long contexts. Current KV Cache eviction has become an important research direction; however, existing methods based on fixed Soft Tokens (e.g., Judge Q) rely on a static parameter set as the query to evaluate the importance of KV pairs, so they cannot adapt dynamically to different input prompts, and they cannot precisely capture complex and changing task relevance. Also, evicted KV pairs are discarded permanently, so this causes irreversible information loss and context breaks. To address this problem, we propose Meta-Soft, a dynamic compression framework based on probe-driven context integration. Specifically, we build a meta-library with a learnable orthogonal basis matrix $\mathcal{L}$, and we use a selector network with Gumbel-Softmax to produce differentiable sparse combination weights, so we dynamically synthesize the most targeted $k$ Soft Tokens from the input prompt features. We append these Soft Tokens to the end of the input sequence to probe key information. We also introduce an attention-flow based integration mechanism, which redistributes the semantic information of removed tokens into retained tokens, and this keeps the dropped context information effectively. Experiments on multiple datasets show that our method outperforms existing state-of-the-art eviction methods and provides a new solution for KV Cache compression.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。