Lynx: 渐进式推测量化加速长上下文推理中的KV传输
Lynx: Progressive Speculative Quantization for accelerating KV Transfer in Long-Context Inference
AI总结：
针对长上下文推理中KV缓存传输延迟问题，提出Lynx系统，通过将KV缓存分为高优先级Anchor流和低优先级Residual流，实现渐进式传输和推测解码，在降低首令牌延迟的同时保持高精度。
摘要：
长上下文推理在大语言模型服务中日益常见，这得益于检索增强生成和智能体系统。在分离式推理中，这些工作负载需要通过网络传输大型键值（KV）缓存，解码必须等待传输完成才能开始。最近的KV量化技术减少了数据量并缓解了这一瓶颈，但现有方案无法同时实现低网络暴露延迟和高推理精度。我们挑战了KV缓存必须完全接收后才能使用的不可分割单元假设。我们利用观察：KV缓存中的不同比特对注意力计算和推理精度的贡献不同——最高有效位捕捉注意力的粗略结构，最低有效位细化精度。这一特性使得解码过程中可以部分使用KV缓存。我们提出了Lynx系统，该系统通过将KV缓存划分为携带最高有效位的高优先级Anchor流和携带剩余精度的低优先级Residual流，实现了渐进式、分流KV传输。解码在接收到Anchor流后开始，并在Residual流并发传输时进行推测性解码，随后进行验证以确保与更高精度解码等价。在多个模型和服务工作负载上，Lynx实现了与激进4位KV量化相当的首令牌延迟（TTFT），同时匹配高精度（BF16）推理的准确性，相比标准8位KV量化，TTFT提升高达1.43倍，相比最先进方法，准确性提升高达5.1%。
英文摘要：
Long-context inference is increasingly common in large language model (LLM) serving, driven by retrieval-augmented generation and agentic systems. In disaggregated inference, these workloads require transferring large Key-Value (KV) caches across the network, where decoding cannot begin until the transfer completes. Recent KV quantization techniques reduce data volume and alleviate this bottleneck, but existing schemes fail to achieve both low network-exposed latency and high inference accuracy. We challenge the assumption that the KV cache is an indivisible unit that must be fully received before use. We leverage the observation that different bits in the KV cache contribute unequally to attention computation and inference precision: the most significant bits capture the coarse structure of attention and the least significant bits refine precision. This property enables partial use of the KV cache during decoding. We present Lynx, a system that enables progressive, split-stream KV transfer by partitioning the KV cache into a high-priority Anchor stream carrying the most significant bits and a low-priority Residual stream carrying remaining precision. Decoding begins upon receipt of the Anchor stream and proceeds speculatively while the Residual stream is transferred concurrently, followed by verification that ensures equivalence to higher-precision decoding. Across multiple models and serving workloads, Lynx achieves Time-to-First-Token (TTFT) comparable to aggressive 4-bit KV quantization, while matching the accuracy of high-precision (BF16) inference, improving TTFT over standard 8-bit KV quantization by up to $1.43\times$ and improving accuracy over state-of-the-art by up to $5.1\%$.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
