arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2511.18643cs.LGcs.AI
收藏

Kitty: 一种准确且高效的2位KV缓存量化方法,结合动态通道精度提升

Kitty: Accurate and Efficient 2-bit KV Cache Quantization with Dynamic Channel-wise Precision Boost

Haojun Xia, Xiaoxia Wu, Jisen Li, Robert Wu, Junxiong Wang, Jue Wang, Chenxi Li, Aman Singhal, Alay Dilipbhai Shah, Alpay Ariyak, Donglin Zhuang, Zhongzhu Zhou,… 展开作者

Haojun Xia, Xiaoxia Wu, Jisen Li, Robert Wu, Junxiong Wang, Jue Wang, Chenxi Li, Aman Singhal, Alay Dilipbhai Shah, Alpay Ariyak, Donglin Zhuang, Zhongzhu Zhou, Ben Athiwaratkun, Zhen Zheng, Shuaiwen Leon Song

AI总结:

Kitty通过混合精度KV缓存的算法-系统协同设计,实现2位量化在保持高精度的同时显著降低内存占用,提升推理效率。

摘要:

KV缓存是大语言模型(LLM)推理中的主要内存瓶颈。尽管4位KV量化能保持准确性,但2位量化通常会降低准确性,尤其是在长上下文推理任务中。我们通过混合精度KV缓存的算法-系统协同设计方法Kitty来填补这一空白。在算法层面,大量实验表明,动态通道精度提升——通过按敏感性对Key缓存通道进行排序并只保留少量高精度通道——在接近2位内存时能保持几乎零的精度损失。主要挑战是处理动态4位通道提升,同时保持页面布局的紧凑性和去量化的一致性,避免散列读取或硬编码掩码。Kitty通过将每个混合精度的Key页面分解为两个具有统一2位精度的张量来解决这些问题。基于此,Kitty提供了以页面为中心的KV布局、与Triton兼容的页面去量化内核以及一个轻量级的运行时流水线,以保持紧凑性和避免发散。在七个任务和两个模型家族(Qwen3、LLaMA3)上,Kitty将KV内存减少了近8倍,精度损失极小,使在相同内存预算下能够支持高达8倍更大的批次和2.1倍至4.1倍更高的吞吐量。我们在此发布Kitty的完整实现,网址为https://github.com/Summer-Summer/Kitty。

英文摘要:

The KV cache is a dominant memory bottleneck for LLM inference. While 4-bit KV quantization preserves accuracy, 2-bit often degrades it, especially on long-context reasoning. We close this gap via an algorithm-system co-design for mixed-precision KV caching: Kitty. On the algorithm side, extensive experiments show that Dynamic Channel-wise Precision Boost -- which ranks Key-cache channels by sensitivity and keeps only a small fraction at higher precision -- maintains near-zero loss in accuracy drop while approaching 2-bit memory. The main challenge is handling dynamic 4-bit channel boosts while keeping the page layout coalesced and the dequantization uniform, with no scattered reads or hard-coded masks. Kitty addresses these issues by decompose each mixed-precision Key page into two tensors with unified 2-bit precision. Based on this, Kitty provides a page-centric KV layout, Triton-compatible page dequantization kernels, and a lightweight runtime pipeline that preserves coalescing and avoids divergence. Across seven tasks and two model families (Qwen3, LLaMA3), Kitty cuts KV memory by nearly 8x with negligible accuracy loss, enabling up to 8x larger batches and 2.1x-4.1x higher throughput under the same memory budget. We release the full implementation of Kitty at https://github.com/Summer-Summer/Kitty.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑