XQuant：通过跨层压缩实现超低比特 KV 缓存量化
XQuant: Achieving Ultra-Low Bit KV Cache Quantization with Cross-Layer Compression
AI总结：
本文提出无需训练的即插即用框架XQuant，通过免数据校准与跨层压缩实现超低比特KV缓存量化，在保持模型精度的同时显著降低内存消耗。
摘要：
大型语言模型（LLMs）在各类自然语言处理任务中展现出卓越能力。然而，其庞大的内存需求，尤其是长文本理解和生成过程中 KV 缓存的不断增长，给在资源受限环境中的部署带来了巨大挑战。量化已成为一种极具前景的解决方案，能在保留历史信息的同时降低内存消耗。我们提出了 XQuant，这是一个无需训练且即插即用的框架，可实现超低等效比特宽度的 KV 缓存量化。XQuant 引入了两项关键创新：计算开销可忽略的免数据校准方法以及跨层 KV 缓存压缩，使量化能够降至 1.4 比特以下。在 TruthfulQA 和 LongBench 上的大量实验表明，XQuant 通过在保持卓越性能的同时实现更低比特宽度，超越了现有最先进方法（例如 KIVI-2bit 和 AsymKV-1.5bit），在内存效率与模型精度之间建立了更优的平衡。
英文摘要：
Large Language Models (LLMs) have demonstrated remarkable capabilities across diverse natural language processing tasks. However, their extensive memory requirements, particularly due to KV cache growth during long-text understanding and generation, present significant challenges for deployment in resource-constrained environments. Quantization has emerged as a promising solution to reduce memory consumption while preserving historical information. We propose XQuant, a training-free and plug-and-play framework that achieves ultra-low equivalent bit-width KV cache quantization. XQuant introduces two key innovations: a computationally negligible data-free calibration method and cross-layer KV cache compression, enabling quantization to sub-1.4 bits. Extensive experiments on TruthfulQA and LongBench demonstrate that XQuant outperforms state-of-the-art methods (e.g., KIVI-2bit and AsymKV-1.5bit) by achieving lower bit-width while maintaining superior performance, establishing a better trade-off between memory efficiency and model accuracy.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
