arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.09054cs.LGcs.AIcs.ARcs.CL
收藏

Keyformer:通过关键 Token 选择减少 KV Cache 以实现高效生成式推理

Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inference

The University of British Columbia(不列颠哥伦比亚大学);d-Matrix

Muhammad Adnan, Akhil Arunkumar, Gaurav Jain, Prashant J. Nair, Ilya Soloveychik, Purushotham Kamath

AI总结:

本文提出推理时方法 Keyformer,通过新评分函数选择承载约 90% 注意力权重的关键 token 并仅缓存其 KV,在 GPT-J 等模型上保持准确率,同时将延迟降至 1/2.1、吞吐提升 2.4 倍。

摘要:

Transformer 已成为大型语言模型(LLM)的基础架构。在生成式语言模型中,推理过程包括两个主要阶段:提示处理和 token 生成。token 生成占据计算工作量的大部分,主要涉及向量-矩阵乘法以及与键值(Key-Value,KV)Cache 的交互。由于将权重和 KV cache 数据从内存系统传输到计算单元会产生开销,该阶段受到内存带宽限制。在需要长上下文和大规模文本生成的应用中,这一内存瓶颈尤为突出,而这两类能力对 LLM 正变得越来越关键。本文提出一种名为“Keyformer”的创新推理时方法,以缓解与 KV cache 大小和内存带宽利用相关的挑战。Keyformer 利用了如下观察:在生成式推理中,约 90% 的注意力权重集中于一特定 token 子集,这些 token 被称为“关键”token。Keyformer 通过一种新的评分函数识别这些关键 token,并在 KV cache 中仅保留它们。该方法在不损害模型准确率的情况下,有效减小了 KV cache 大小并降低了内存带宽使用。我们在三种基础模型 GPT-J、Cerebras-GPT 和 MPT 上评估 Keyformer 的性能,这些模型采用了不同的位置嵌入算法。评估涵盖多种任务,尤其侧重涉及扩展上下文的摘要和对话任务。Keyformer 对 KV cache 的缩减使推理延迟降低 2.1 倍,并使 token 生成吞吐量提高 2.4 倍,同时保持模型准确率。

英文摘要:

Transformers have emerged as the underpinning architecture for Large Language Models (LLMs). In generative language models, the inference process involves two primary phases: prompt processing and token generation. Token generation, which constitutes the majority of the computational workload, primarily entails vector-matrix multiplications and interactions with the Key-Value (KV) Cache. This phase is constrained by memory bandwidth due to the overhead of transferring weights and KV cache values from the memory system to the computing units. This memory bottleneck becomes particularly pronounced in applications that require long-context and extensive text generation, both of which are increasingly crucial for LLMs. This paper introduces "Keyformer", an innovative inference-time approach, to mitigate the challenges associated with KV cache size and memory bandwidth utilization. Keyformer leverages the observation that approximately 90% of the attention weight in generative inference focuses on a specific subset of tokens, referred to as "key" tokens. Keyformer retains only the key tokens in the KV cache by identifying these crucial tokens using a novel score function. This approach effectively reduces both the KV cache size and memory bandwidth usage without compromising model accuracy. We evaluate Keyformer's performance across three foundational models: GPT-J, Cerebras-GPT, and MPT, which employ various positional embedding algorithms. Our assessment encompasses a variety of tasks, with a particular emphasis on summarization and conversation tasks involving extended contexts. Keyformer's reduction of KV cache reduces inference latency by 2.1x and improves token generation throughput by 2.4x, while preserving the model's accuracy.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑