arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.08722cs.LGcs.AI
收藏

QUOKA:面向查询的键值选择以实现高效的LLM预填

QUOKA: Query-Oriented KV Selection For Efficient LLM Prefill

Qualcomm AI Research(高通人工智能研究)

Dalton Jones, Junyoung Park, Matthew Morse, Mingu Lee, Chris Lott, Harper Langston

AI总结:

QUOKA通过面向查询的键值选择,在保持准确性的同时显著加速了transformer推理,尤其在CPU上实现了接近7倍的速度提升。

摘要:

我们提出了QUOKA:面向查询的键值选择以实现高效的注意力,这是一种免训练且硬件无关的稀疏注意力算法,用于加速在分块预填下的transformer推理。虽然许多查询在注意力运算中集中在较小的键组上,但我们发现低余弦相似度相对于均值查询的查询与更多的键相互作用,并对最终注意力logits贡献最大。通过优先考虑这些低余弦相似度查询,预填阶段的完整注意力行为可以近似。QUOKA利用这一观察,通过(1)首先保留少量代表性查询和(2)然后选择与这些查询最匹配的键来加速注意力。通过在Needle-In-A-Haystack、LongBench、RULER和Math500上的实验,我们显示,尽管在时间到第一个token上实现了3倍的减少,在Nvidia GPU上实现了5倍的加速,并在Intel Xeon CPU上实现了接近7倍的加速,QUOKA在接近基线准确性的情况下,每注意力评估使用88%更少的键值对。

英文摘要:

We present QUOKA: Query-oriented KV selection for efficient attention, a training-free and hardware agnostic sparse attention algorithm for accelerating transformer inference under chunked prefill. While many queries focus on a smaller group of keys in the attention operator, we observe that queries with low cosine similarity with respect to the mean query interact more strongly with more keys and have the greatest contribution to final attention logits. By prioritizing these low cosine similarity queries, the behavior of full attention during the prefill stage can be closely approximated. QUOKA leverages this observation, accelerating attention by (1) first retaining a small set of representative queries and (2) then subselectin the keys most aligned with those queries. Through experiments on Needle-In-A-Haystack, LongBench, RULER, and Math500, we show that, while realizing a 3x reduction in time-to-first-token, 5x speedup in attention on Nvidia GPUs and up to nearly a 7x speedup on Intel Xeon CPUs, QUOKA achieves near-baseline accuracy, utilizing 88% fewer key-value pairs per attention evaluation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑