arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.07397cs.LGcs.AI
收藏

在参加之前侦察:Sketch-and-Walk稀疏注意力用于高效的LLM推理

Scout Before You Attend: Sketch-and-Walk Sparse Attention for Efficient LLM Inference

Department of Computer Science, Rice University(计算机科学系, Rice大学);Department of Computer Science, Stevens Institute of Technology(计算机科学系, Stevens理工学院)

Hoang Anh Duy Le, Sahil Joshi, Zeyu Yang, Zhaozhuo Xu, Anshumali Shrivastava

AI总结:

Sketch&Walk通过无需训练的稀疏注意力方法,在高效LLM推理中实现高精度和速度提升。

摘要:

自注意力机制在长上下文LLM推理的prefill和decode阶段中主导了计算和内存成本。为解决这一挑战,我们引入了Sketch&Walk注意力,一种无需训练的稀疏注意力方法,通过轻量级的草图和确定性行走来确定稀疏性。Sketch&Walk利用Hadamard草图法获得低成本的注意力分数近似值,然后通过行走机制在层间聚合这些估计值,捕捉token之间超越直接交互的注意力影响。累积的行走分数用于选择top-k注意力块,使动态稀疏性成为可能,通过单一的无需训练的算法应用于prefill和decode阶段,同时配合定制的稀疏注意力内核。在广泛的模型和任务中,Sketch&Walk在20%的注意力密度下保持接近无损的准确性,并在某些情况下略微优于密集注意力,同时实现高达6倍的推理加速。

英文摘要:

Self-attention dominates the computational and memory cost of long-context LLM inference across both prefill and decode phases. To address this challenge, we introduce Sketch&Walk Attention, a training-free sparse attention method that determines sparsity with lightweight sketches and deterministic walk. Sketch&Walk applies Hadamard sketching to get inexpensive approximations of attention scores, then aggregates these estimates across layers via a walk mechanism that captures attention influence beyond direct interactions between tokens. The accumulated walk scores are used to select top-k attention blocks, enabling dynamic sparsity with a single training-free algorithm that applies uniformly to both the prefill and decode phases, together with custom sparse attention kernels. Across a wide range of models and tasks, Sketch&Walk maintains near-lossless accuracy at 20% attention density and can slightly outperform dense attention in some settings, while achieving up to 6x inference speedup.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑