The KV cache is a dominant memory bottleneck for LLM inference. While 4-bit KV quantization preserves accuracy, 2-bit often degrades it, especially on long-context reasoning. We close this gap via an algorithm-system co-design for mixed-precision KV caching: Kitty. On the algorithm side, extensive experiments show that Dynamic Channel-wise Precision Boost -- which ranks Key-cache channels by sensitivity and keeps only a small fraction at higher precision -- maintains near-zero loss in accuracy drop while approaching 2-bit memory. The main challenge is handling dynamic 4-bit channel boosts while keeping the page layout coalesced and the dequantization uniform, with no scattered reads or hard-coded masks. Kitty addresses these issues by decompose each mixed-precision Key page into two tensors with unified 2-bit precision. Based on this, Kitty provides a page-centric KV layout, Triton-compatible page dequantization kernels, and a lightweight runtime pipeline that preserves coalescing and avoids divergence. Across seven tasks and two model families (Qwen3, LLaMA3), Kitty cuts KV memory by nearly 8x with negligible accuracy loss, enabling up to 8x larger batches and 2.1x-4.1x higher throughput under the same memory budget. We release the full implementation of Kitty at https://github.com/Summer-Summer/Kitty.
Boosting Long-Context Management via Query-Guided Activation Refilling
Beijing Academy of Artificial Intelligence(北京人工智能研究院);Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);University of Science and Technology of China(中国科学技术大学)
IntactKV: Improving Large Language Model Quantization by Keeping Pivot Tokens Intact
Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);The Chinese University of Hong Kong(香港中文大学)
P3-LLM: An Integrated NPU-PIM Accelerator for Edge LLM Inference Using Hybrid Numerical Formats
Department of Electrical and Computer Engineering, Cornell University(康奈尔大学电气与计算机工程系);EAST-MICAS, KU Leuven(KU莱顿大学EAST-MICAS);Department of Electrical Engineering, Stanford University(斯坦福大学电气工程系)