通过注意力动态理解LLMs中键值缓存压缩的物理原理
Understanding the Physics of Key-Value Cache Compression for LLMs through Attention Dynamics
AI总结：
通过分析LLMs中键值缓存压缩的物理机制，揭示了压缩对内部表示和语义可达性的影响，以及不同架构在路由动态上的差异。
摘要：
随着LLMs的上下文窗口扩展到100K+个token，键值（KV）缓存成为主要的内存瓶颈，最近的方法声称可实现80-90%的节省，并且对基准测试的降影响很小。我们认为这些评估忽略了结构问题：注意力不仅仅是存储，而是路由，保留KV对并不保证语义可达性。我们提出了一种受物理启发的KV压缩观点，将其视为对token级路由的受控扰动，区分保留、可达性和利用率。使用合成任务探测多实体跟踪、歧义消除、指代消解和多跳推理，我们发现适度压缩会损害内部表示，但对准确性影响很小，揭示出冗余；所有模型在90%压缩附近都表现出明显的幻觉安全悬崖，与全局驱逐率（GER）的激增相关，表明语义可达性存在相变；架构在路由动态上存在差异，LLaMA在早期达成共识，后期多样化，而Qwen则表现出类似漏斗的后期收敛，导致不同的抗压特性。除了擦除之外，我们还识别出表示刚性，即过度的头部一致性会压缩路由灵活性，即使token仍然存活。这些结果表明稀疏token-路由结构决定了压缩容忍度，将KV压缩重新定义为注意力几何结构的探测器，并将长上下文可扩展性与自注意力中的稀疏性和彩票彩票假说联系起来。
英文摘要：
As context windows in LLMs scale to 100K+ tokens, the key-value (KV) cache becomes the dominant memory bottleneck, with recent methods claiming 80-90% savings and minimal benchmark degradation. We argue these evaluations miss a structural issue: attention is not just storage but routing, and retaining KV pairs does not guarantee semantic accessibility. We propose a physics-inspired view of KV compression as a controlled perturbation of token-level routing, distinguishing retention, accessibility, and utilization. Using synthetic tasks probing multi-entity tracking, disambiguation, coreference, and multi-hop reasoning, we find that moderate compression degrades internal representations with little accuracy loss, revealing redundancy; all models exhibit a sharp hallucination safety cliff near 90% compression, correlated with spikes in Global Eviction Ratio (GER), suggesting a phase transition in semantic reachability; and architectures differ in routing dynamics, with LLaMA showing early consensus and late diversification, and Qwen showing funnel-like late convergence, leading to distinct resilience profiles. Beyond erasure, we identify representational rigidity, where excessive head-level consensus collapses routing flexibility despite token survival. These results suggest sparse token-route structures govern compression tolerance, reframing KV compression as a structural probe of attention geometry and linking long-context scalability to sparsity and the lottery ticket hypothesis in self-attention.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
