arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.07056cs.CL
收藏

有效压缩大语言模型的 KV 头

Effectively Compress KV Heads for LLM

Nanjing University(南京大学);School of Artificial Intelligence, Nanjing University(南京大学人工智能学院);Alibaba Inc.(阿里巴巴集团)

Hao Yu, Zelan Yang, Shen Li, Yong Li, Jianxin Wu

AI总结:

针对 LLM 推理中 KV 缓存内存瓶颈,本文利用其低秩特性优化 MHA 到 GQA 的转换,并设计兼容 RoPE 的键缓存策略,可压缩一半至四分之三 KV 头且性能基本保持。

摘要:

预训练大语言模型(LLM)的出现彻底改变了各类自然语言处理任务。这些模型主要采用自回归解码机制,利用键值(KV)缓存来消除对先前 token 的重复计算。然而,随着上下文长度和批量大小增加,KV 缓存内存占用的线性扩展成为 LLM 部署的关键瓶颈,并显著降低生成速度。为缓解该问题,multi-query attention(MQA,多查询注意力)和 grouped-query attention(GQA,分组查询注意力)等已有技术被提出,用于减少 KV 头,以在精度与 multi-head attention(MHA,多头注意力)相当的情况下加速推理。尽管这些方法有效,但现有压缩 MHA 的策略往往忽视 KV 缓存的内在属性。本文研究 KV 缓存的低秩特性,并提出一种压缩 KV 头的新方法。具体而言,我们精心优化从 MHA 到 GQA 的转换,以最小化压缩误差;同时,为保持与 rotary position embeddings(RoPE,旋转位置嵌入)的兼容性,还针对带有 RoPE 的键缓存引入专门策略。实验表明,我们的方法可压缩一半甚至四分之三的 KV 头,同时保持与原始 LLM 相当的性能,这为在资源受限环境中更高效地部署 LLM 提供了有前景的方向。

英文摘要:

The advent of pre-trained large language models (LLMs) has revolutionized various natural language processing tasks. These models predominantly employ an auto-regressive decoding mechanism that utilizes Key-Value (KV) caches to eliminate redundant calculations for previous tokens. Nevertheless, as context lengths and batch sizes increase, the linear expansion in memory footprint of KV caches becomes a key bottleneck of LLM deployment, which decreases generation speeds significantly. To mitigate this issue, previous techniques like multi-query attention (MQA) and grouped-query attention (GQA) have been developed, in order to reduce KV heads to accelerate inference with comparable accuracy to multi-head attention (MHA). Despite their effectiveness, existing strategies for compressing MHA often overlook the intrinsic properties of the KV caches. In this work, we explore the low-rank characteristics of the KV caches and propose a novel approach for compressing KV heads. In particular, we carefully optimize the MHA-to-GQA transformation to minimize compression error, and to remain compatible with rotary position embeddings (RoPE), we also introduce specialized strategies for key caches with RoPE. We demonstrate that our method can compress half or even three-quarters of KV heads while maintaining performance comparable to the original LLMs, which presents a promising direction for more efficient LLM deployment in resource-constrained environments.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑