arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.05929cs.CL
收藏

KV-CoRE: 对LLMs中KV缓存数据依赖性低秩压缩性的基准测试

KV-CoRE: Benchmarking Data-Dependent Low-Rank Compressibility of KV-Caches in LLMs

University at Buffalo(布法罗大学);Institute of Artificial Intelligence (TeleAI), China Telecom(人工智能研究所(TeleAI),中国电信);Delft University of Technology(代尔夫特理工大学);University of Maryland(马里兰大学);ByteDance(字节跳动)

Jian Chen, Zhuoran Wang, Jiayu Qin, Ming Li, Meng Wang, Changyou Chen, Yin Chen, Qizhen Weng, Yirui Liu

AI总结:

KV-CoRE提出了一种基于SVD的KV缓存压缩性评估方法,通过分析模型架构、训练数据和语言覆盖等因素,揭示了压缩性与性能下降之间的关系,为数据驱动的压缩和模型开发提供了新的基准。

摘要:

大型语言模型依赖于KV缓存来避免在自回归解码过程中出现冗余计算,但随着上下文长度的增长,读取和写入缓存会迅速耗尽GPU内存带宽。最近的研究探索了KV缓存压缩,但大多数方法忽略了KV缓存的数据依赖性及其在不同层之间的变化。我们介绍了KV-CoRE(KV缓存压缩性由秩评估),这是一种基于SVD的方法,用于量化KV缓存的数据依赖性低秩压缩性。KV-CoRE在Frobenius范数下计算最优低秩近似,并且由于其无梯度和增量特性,能够实现数据层面和层层面的高效评估。使用这种方法,我们分析了涵盖五个英语领域和十六种语言的多个模型和数据集,揭示了压缩性与模型架构、训练数据和语言覆盖之间的系统性模式。作为这一分析的一部分,我们采用了归一化有效秩作为压缩性的度量,并展示了其与压缩下的性能下降之间存在强相关性。我们的研究建立了一个原则性的评估框架,并提出了LLMs中首个大规模的KV缓存压缩性基准,为动态、数据感知的压缩和数据导向的模型开发提供了见解。

英文摘要:

Large language models rely on kv-caches to avoid redundant computation during autoregressive decoding, but as context length grows, reading and writing the cache can quickly saturate GPU memory bandwidth. Recent work has explored KV-cache compression, yet most approaches neglect the data-dependent nature of kv-caches and their variation across layers. We introduce KV-CoRE KV-cache Compressibility by Rank Evaluation), an SVD-based method for quantifying the data-dependent low-rank compressibility of kv-caches. KV-CoRE computes the optimal low-rank approximation under the Frobenius norm and, being gradient-free and incremental, enables efficient dataset-level, layer-wise evaluation. Using this method, we analyze multiple models and datasets spanning five English domains and sixteen languages, uncovering systematic patterns that link compressibility to model architecture, training data, and language coverage. As part of this analysis, we employ the Normalized Effective Rank as a metric of compressibility and show that it correlates strongly with performance degradation under compression. Our study establishes a principled evaluation framework and the first large-scale benchmark of kv-cache compressibility in LLMs, offering insights for dynamic, data-aware compression and data-centric model development.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑