arXiv 2610.186422026-10-14cs.LG

MultiAlloc：面向KV缓存量化的预算感知精度分配

MultiAlloc: Budget-Aware Precision Allocation for KV Cache Quantization

Nanjing University（南京大学）；The Hong Kong University of Science and Technology（香港科技大学）；Geely Goertek（吉利歌尔）

Yangchengyu Zhou, Zhiping Wu, Runchang Jiang, Dongdong Ren, Youfang Han, Wei Zhao, Dongqi Cai, Wenbin Li

备注：: Code: https://github.com/RL-MIND/MultiAlloc

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

MultiAlloc是一种KV缓存量化的预算感知精度分配框架，通过离线分析实现层级与令牌级精度资源的自适应分配，在固定内存预算下优于多种现有KV缓存量化方法。

摘要：

键值（KV）缓存是大语言模型（LLM）推理过程中内存消耗的主要来源，尤其在长上下文和复杂推理场景下更为突出。现有的KV缓存量化方法已探索了不同的压缩粒度，包括层级精度调整和令牌级信息保留。然而在实际部署中，核心挑战不仅是压缩，而是如何在固定KV缓存内存预算下，将有限的量化资源分配给异构的精度保留策略。我们提出MultiAlloc，这是一种用于KV缓存量化的、由分析引导的精度分配框架，可自适应地在层级精度增强和令牌级保留之间分配有限的量化资源。MultiAlloc采用轻量级离线分析来表征层敏感性和令牌重要性，从而在固定内存约束下实现自适应精度分配。在分析得到的内存-精度权衡的引导下，它无需在线分配搜索即可自动推导有效的精度分配策略，并在推理过程中应用该策略。在推理和长上下文基准上进行的大量实验表明，MultiAlloc在匹配的内存预算下，相较于层级、令牌级以及近期的多粒度KV缓存量化方法，在Mistral、Llama、DeepSeek-Qwen和QwQ等多种模型上均实现了持续提升。

英文摘要：

The Key-Value (KV) cache constitutes a dominant source of memory consumption during large language model (LLM) inference, particularly in long-context and complex reasoning scenarios. Existing KV cache quantization methods have explored different compression granularities, including layer-wise precision adjustment and token-level information preservation. In practical deployment, however, the key challenge is not merely compression, but how to allocate limited quantization resources among heterogeneous precision preservation strategies under a fixed KV cache memory budget. We propose MultiAlloc, a profiling-guided precision allocation framework for KV cache quantization that adaptively distributes limited quantization resources between layer-wise precision enhancement and token-wise preservation. MultiAlloc employs lightweight offline profiling to characterize layer sensitivity and token importance, enabling adaptive precision allocation under a fixed memory constraint. Guided by the profiled memory--accuracy trade-off, it automatically derives an effective precision allocation strategy without online allocation search, and applies the resulting policy during inference. Extensive experiments on reasoning and long-context benchmarks demonstrate that MultiAlloc consistently improves over layer-wise, token-wise, and recent multi-granularity KV cache quantization methods under matched memory budgets across diverse models, including Mistral, Llama, DeepSeek-Qwen, and QwQ.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。