arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2606.09441cs.AIcs.AR
收藏

SIFT: 利用注意力不变性实现RAG预填充快速计算的索引选择

SIFT: Selective-Index For Fast Compute of RAG Prefill by Exploiting Attention Invariance

Georgia Institute of Technology(佐治亚理工学院);Microsoft(微软)

Rya Sanovar, Srikant Bharadwaj, Hritvik Taneja, Moinuddin Qureshi

AI总结:

针对RAG查询中文档重复导致预填充计算冗余和TTFT增加的问题,提出SIFT方法,通过离线提取文档高注意力分数位置并利用注意力不变性,在预填充时仅计算标记位置,将TTFT提升1.71倍且精度损失在1%以内。

摘要:

检索增强生成(RAG)向LLM查询注入相关文档以提高响应质量。这种注入增加了提示长度并减慢了首个令牌生成时间(TTFT)。与标准查询不同,RAG查询具有上下文复用的独特属性,即相同文档在用户查询中重复出现。因此,为每个RAG查询完全重新计算文档会导致冗余计算并增加TTFT。先前的工作离线预计算RAG文档的KV张量,并在在线预填充期间粗略地重新计算一些令牌。然而,由于高延迟的磁盘传输,这种KV复用在现代GPU上通常比完全重新计算更慢。此外,这种粗粒度的重新计算会降低准确性。为了解决这些限制,本文提出了SIFT:利用注意力不变性实现RAG预填充快速计算的索引选择。SIFT离线处理文档,并提取每个文档中高注意力分数的细粒度位置。接下来,我们识别出以下注意力不变性见解,使我们能够在运行时利用提取的位置:(1)局部注意力不变性:文档内高注意力分数的位置不受周围文档的影响。这有助于我们预测文档自注意力中高分数出现的位置。(2)交叉注意力一致性:具有高文档内注意力的键也会吸引后续文档的交叉注意力。这有助于我们预测文档对未来文档注意力中高分数出现的位置。关键的是,SIFT不存储任何KV数据,仅以两个紧凑的位向量的形式存储高分数位置。SIFT的存储比KV张量小24000倍,避免了昂贵的磁盘传输。在预填充期间,SIFT仅计算标记位置的注意力,将TTFT提升1.71倍,同时将精度保持在完全重新计算的1%以内。

英文摘要:

Retrieval-Augmented Generation (RAG) injects LLM queries with relevant documents to improve response quality. This injection increases prompt length and slows time to first token (TTFT). Unlike standard queries, RAG queries have a unique property of context reuse where the same documents recur across user queries. Thus, fully recomputing documents for every RAG query does redundant compute and increases TTFT. Prior works precompute KV tensors of RAG documents offline and coarsely recompute some tokens during online prefill. However, such KV reuse is often slower than full recomputation on modern GPUs due to high-latency disk transfers. Further, such a coarse-grained recomputation degrades accuracy. To address these limitations, this paper proposes SIFT: Selective-Index For Fast Compute of RAG Prefill by Exploiting Attention Invariance. SIFT processes documents offline and extracts fine-grained locations of high attention scores for each document. Next, we identify the following attention invariance insights that enable us to exploit the extracted locations during runtime: (1) Local-Attention Invariance: The location of high attention scores within a document remain invariant to surrounding documents. This helps us predict the location of high scores where the document attends to itself. (2) Cross-Attention Consistency: Keys with high intra-document attention also attract cross-attention from subsequent documents. This helps us predict the location of high scores where the document attends to future documents. Critically, SIFT stores no KV data and only stores locations of high scores in the form of two compact bit vectors. SIFT's storage is up to 24,000x smaller than KV tensors, obviating costly disk transfers. During prefill, SIFT computes the attention only for the marked locations and improves TTFT by 1.71x while holding accuracy within 1% of full recompute.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑