arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.00167cs.AI
收藏

Finch:提示引导的键值缓存压缩

Finch: Prompt-guided Key-Value Cache Compression

SAP Labs, France(思爱普实验室(法国));EURECOM, France(欧洲通信学院(法国))

Giulio Corallo, Paolo Papotti

AI总结:

针对大语言模型长上下文处理受上下文窗口和GPU内存限制的问题,提出Finch提示引导KV缓存压缩方法,无需微调即可在93倍高压缩率下保留语义处理长输入。

摘要:

近期的大语言模型应用,如检索增强生成(Retrieval-Augmented Generation)和聊天机器人,使得处理更长输入上下文的需求不断增长。然而,这一需求受到固有局限的阻碍。在架构层面,模型受限于训练时定义的上下文窗口。此外,处理大量文本需要可观的GPU内存。我们提出了一种名为Finch的新方法,利用自注意力的预训练模型权重来压缩输入上下文。给定一个提示和一段长文本,Finch以提示为条件,在文本分块上迭代识别最相关的键(Key,K)和值(Value,V)对。仅这些键值对会被存储在KV缓存中,在上下文窗口限定的空间内,最终形成长文本的压缩版本。我们的方案使模型即使在高达93倍的高压缩率下也能处理大输入,同时保留语义完整性,且无需微调。

英文摘要:

Recent large language model applications, such as Retrieval-Augmented Generation and chatbots, have led to an increased need to process longer input contexts. However, this requirement is hampered by inherent limitations. Architecturally, models are constrained by a context window defined during training. Additionally, processing extensive texts requires substantial GPU memory. We propose a novel approach, Finch, to compress the input context by leveraging the pre-trained model weights of the self-attention. Given a prompt and a long text, Finch iteratively identifies the most relevant Key (K) and Value (V) pairs over chunks of the text conditioned on the prompt. Only such pairs are stored in the KV cache, which, within the space constrained by the context window, ultimately contains a compressed version of the long text. Our proposal enables models to consume large inputs even with high compression (up to 93x) while preserving semantic integrity without the need for fine-tuning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑