子令牌路由用于KV缓存压缩
Sub-Token Routing for KV Cache Compression
AI总结：
提出子令牌路由方法，在保留令牌内对值向量分组并选择性保留，与令牌级压缩互补，在LLM和VLM中提升压缩性能。
摘要：
Transformer推理通常需要大型KV缓存，尤其是在长上下文语言建模和多模态生成中。现有的压缩方法通常通过选择、驱逐、量化或压缩缓存令牌，或在语言模型推理前减少视觉令牌序列来降低缓存成本。我们引入子令牌路由，一种KV压缩方法，它在保留令牌内部添加了更精细的控制轴。它将每个保留的值向量分成组，并仅保留选定的组，同时保持查询和键状态不变。该方法设计在令牌级缩减之后工作。首先，令牌缩减方法确定保留哪些令牌。然后，子令牌路由压缩这些保留令牌内部的值状态。在匹配KV预算下的实验表明，添加子令牌路由提高了令牌级缩减在LLM和VLM设置中的性能，包括LLaMA-2-7B和Qwen2.5-7B上的Quest，以及LLaVA和Qwen-VL模型上的FastV/VisionZip。在较小的KV预算下增益更大，表明当进一步移除令牌成本高昂时，值组路由特别有用。总体而言，令牌级缩减和子令牌路由提供了互补的降低KV成本的方式。
英文摘要：
Transformer inference often requires a large KV cache, especially for long-context language modeling and multimodal generation. Existing compression methods usually reduce cache cost by selecting, evicting, quantizing, or compressing cached tokens, or by reducing the visual-token sequence before language-model inference. We introduce sub-token routing, a KV-compression method that adds a finer control axis inside retained tokens. It splits each retained value vector into groups and keeps only selected groups, while leaving query and key states unchanged. The method is designed to work after token-level reduction. First, a token-reduction method determines which tokens are retained. Then, sub-token routing compresses the value states inside those retained tokens. Experiments under matched KV budgets show that adding sub-token routing improves token-level reduction performance in both LLM and VLM settings, including Quest on LLaMA-2-7B and Qwen2.5-7B, and FastV/VisionZip across LLaVA and Qwen-VL models. The gains are larger at smaller KV budgets, suggesting that value-group routing is especially useful when further token removal becomes costly. Overall, token-level reduction and sub-token routing provide complementary ways to reduce KV cost.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
