模型告诉你何处合并:面向长上下文任务的LLM自适应KV缓存合并
Model Tells You Where to Merge: Adaptive KV Cache Merging for LLMs on Long-Context Tasks
AI总结:
提出KVMerger,一种基于键状态相似性和高斯核加权合并的自适应KV缓存压缩方法,在长上下文任务中实现内存高效且性能不降。
摘要:
如何高效地服务大语言模型(LLMs)已成为一个紧迫问题,因为它们在自回归生成过程中计算成本巨大。为降低计算成本,LLMs常采用KV缓存技术以提高生成速度。在提升计算效率的同时,KV缓存的存储需求巨大,尤其在长上下文场景中,导致显著的内存消耗。现有的KV缓存驱逐方法常因驱逐引入的信息丢失而降低LLMs在长上下文场景中的性能。本文提出一种新颖的KV缓存合并方法,称为KVMerger,旨在实现长上下文任务的自适应KV缓存压缩,在受限内存预算下无明显性能下降。我们的方法受到一个有趣观察的启发:在单个序列内,键状态在令牌级别表现出高度相似性。为促进合并,我们开发了一个有效且直接的合并集识别算法,以识别适合合并的KV状态。我们的合并集识别算法激发了第二个观察:从相似性角度看,KV缓存稀疏性与数据集无关,并在模型级别上保持持久性。随后,我们提出一种高斯核加权合并算法,以选择性地合并每个合并集中的所有状态。我们进行了大量实验,以证明KVMerger在受限内存预算下对长上下文任务的有效性,将其应用于包括Llama2-7B-chat和Llama2-13B-chat在内的模型。使用LongBench和ZeroScroll基准,我们将我们的方法与其他KV缓存压缩技术(包括H2O和CaM)进行比较,结果表明在50%和35%的KV缓存预算下,我们的方法在各项任务中均取得了优越性能。
英文摘要:
How to efficiently serve Large Language Models (LLMs) has become a pressing issue because of their huge computational cost in their autoregressive generation process. To mitigate computational costs, LLMs often employ the KV Cache technique to improve the generation speed. While improving the computational efficiency, the storage requirements of the KV cache are substantial, particularly in long-context scenarios, leading to significant memory consumption. Existing KV cache eviction methods often degrade the performance of LLMs in long-context scenarios due to the information loss introduced by eviction. In this paper, we propose a novel KV cache merging approach, called KVMerger, to achieve adaptive KV cache compression for long-context tasks without significant performance degradation under constrained memory budgets. Our approach is inspired by the intriguing observation that key states exhibit high similarity at the token level within a single sequence. To facilitate merging, we develop an effective yet straightforward merging set identification algorithm to identify suitable KV states for merging. Our merging set identification algorithm stimulates the second observation that KV cache sparsity, from similarity perspective, is independent of the dataset and remains persistent at the model level. Subsequently, we propose a Gaussian kernel weighted merging algorithm to selectively merge all states within each merging set. We conduct extensive experiments to demonstrate the effectiveness of KVMerger for long-context tasks under constrained memory budgets, applying it to models including Llama2-7B-chat and Llama2-13B-chat. Using the LongBench and ZeroScroll benchmarks, we compare our method with other KV cache compression techniques, including H2O and CaM, showing that our method achieves superior performance across tasks with both 50% and 35% KV cache budgets.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。