Focus-dLLM: 通过置信度引导的上下文聚焦加速长上下文扩散语言模型推理
Focus-dLLM: Accelerating Long-Context Diffusion LLM Inference via Confidence-Guided Context Focusing
AI总结：
Focus-dLLM通过置信度引导的上下文聚焦技术，实现了长上下文扩散语言模型推理的高效加速。
摘要：
扩散大语言模型（dLLMs）在非自回归解码范式下展现出强大的长上下文处理能力。然而，双向全注意力机制带来的显著计算成本限制了推理效率。尽管稀疏注意力具有潜力，但现有方法仍不够有效。这源于在解码尚未完成的token时需要估计注意力重要性，而扩散过程中未被掩码的token位置是未知的。本文提出Focus-dLLM，一种全新的无需训练的注意力稀疏化框架，专门用于准确且高效的长上下文dLLM推理。基于发现相邻步骤中token置信度存在强相关性，我们首先设计了一个过去置信度引导的指标来预测未被掩码的区域。在此基础上，我们提出了一种sink-aware的剪枝策略，以准确估计并移除冗余的注意力计算，同时保留具有高度影响的注意力sink。为进一步减少开销，该策略在不同层之间重用已识别的sink位置，利用观察到的跨层一致性。实验结果表明，我们的方法在32K上下文长度下实现了超过29倍的无损加速。代码已公开在：https://github.com/Longxmas/Focus-dLLM
英文摘要：
Diffusion Large Language Models (dLLMs) deliver strong long-context processing capability in a non-autoregressive decoding paradigm. However, the considerable computational cost of bidirectional full attention limits the inference efficiency. Although sparse attention is promising, existing methods remain ineffective. This stems from the need to estimate attention importance for tokens yet to be decoded, while the unmasked token positions are unknown during diffusion. In this paper, we present Focus-dLLM, a novel training-free attention sparsification framework tailored for accurate and efficient long-context dLLM inference. Based on the finding that token confidence strongly correlates across adjacent steps, we first design a past confidence-guided indicator to predict unmasked regions. Built upon this, we propose a sink-aware pruning strategy to accurately estimate and remove redundant attention computation, while preserving highly influential attention sinks. To further reduce overhead, this strategy reuses identified sink locations across layers, leveraging the observed cross-layer consistency. Experimental results show that our method offers more than $29\times$ lossless speedup under $32K$ context length. The code is publicly available at: https://github.com/Longxmas/Focus-dLLM
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
