在参加之前侦察:Sketch-and-Walk稀疏注意力用于高效的LLM推理
Scout Before You Attend: Sketch-and-Walk Sparse Attention for Efficient LLM Inference
AI总结:
Sketch&Walk通过无需训练的稀疏注意力方法,在高效LLM推理中实现高精度和速度提升。
摘要:
自注意力机制在长上下文LLM推理的prefill和decode阶段中主导了计算和内存成本。为解决这一挑战,我们引入了Sketch&Walk注意力,一种无需训练的稀疏注意力方法,通过轻量级的草图和确定性行走来确定稀疏性。Sketch&Walk利用Hadamard草图法获得低成本的注意力分数近似值,然后通过行走机制在层间聚合这些估计值,捕捉token之间超越直接交互的注意力影响。累积的行走分数用于选择top-k注意力块,使动态稀疏性成为可能,通过单一的无需训练的算法应用于prefill和decode阶段,同时配合定制的稀疏注意力内核。在广泛的模型和任务中,Sketch&Walk在20%的注意力密度下保持接近无损的准确性,并在某些情况下略微优于密集注意力,同时实现高达6倍的推理加速。
英文摘要:
Self-attention dominates the computational and memory cost of long-context LLM inference across both prefill and decode phases. To address this challenge, we introduce Sketch&Walk Attention, a training-free sparse attention method that determines sparsity with lightweight sketches and deterministic walk. Sketch&Walk applies Hadamard sketching to get inexpensive approximations of attention scores, then aggregates these estimates across layers via a walk mechanism that captures attention influence beyond direct interactions between tokens. The accumulated walk scores are used to select top-k attention blocks, enabling dynamic sparsity with a single training-free algorithm that applies uniformly to both the prefill and decode phases, together with custom sparse attention kernels. Across a wide range of models and tasks, Sketch&Walk maintains near-lossless accuracy at 20% attention density and can slightly outperform dense attention in some settings, while achieving up to 6x inference speedup.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。