注意力跨度混合：通过异质滑动窗口长度优化LLM推理效率
Mixture of Attention Spans: Optimizing LLM Inference Efficiency with Heterogeneous Sliding-Window Lengths
AI总结：
MoA通过异质滑动窗口长度优化LLM推理效率，提升上下文长度和检索准确率，减少内存消耗并提高吞吐量。
摘要：
滑动窗口注意力为长上下文场景中的大语言模型（LLM）提供了硬件高效的内存和吞吐量解决方案。现有方法通常在所有注意力头和输入大小上使用单一窗口长度。然而，这种统一方法无法捕捉LLM中固有的异构注意力模式，忽略了其不同的准确率-延迟权衡。为了解决这一挑战，我们提出了*注意力跨度混合*（MoA），它会自动为不同的头和层定制不同的滑动窗口长度配置。MoA构建并导航各种窗口长度及其相对于输入大小的缩放规则的搜索空间。它会分析模型，评估潜在配置，并确定每个头的最佳长度配置。MoA适应于不同的输入大小，揭示了一些注意力头会扩展其关注范围以适应更长的输入，而其他头则持续专注于固定长度的局部上下文。实验表明，MoA在相同平均滑动窗口长度下将有效上下文长度增加了3.9倍，提高了检索准确率1.5-7.1倍，超过Vicuna-{7B, 13B}和Llama3-{8B, 70B}模型的统一窗口基线。此外，MoA缩小了与全注意力的性能差距，将最大相对性能下降从9%-36%减少到三个长上下文理解基准测试中的5%以内。MoA实现了1.2-1.4倍的GPU内存减少，比FlashAttention2和vLLM提高了6.6-8.2倍和1.7-1.9倍的解码吞吐量，且对性能影响极小。我们的代码可在：https://github.com/thu-nics/MoA上获得。
英文摘要：
Sliding-window attention offers a hardware-efficient solution to the memory and throughput challenges of Large Language Models (LLMs) in long-context scenarios. Existing methods typically employ a single window length across all attention heads and input sizes. However, this uniform approach fails to capture the heterogeneous attention patterns inherent in LLMs, ignoring their distinct accuracy-latency trade-offs. To address this challenge, we propose *Mixture of Attention Spans* (MoA), which automatically tailors distinct sliding-window length configurations to different heads and layers. MoA constructs and navigates a search space of various window lengths and their scaling rules relative to input sizes. It profiles the model, evaluates potential configurations, and pinpoints the optimal length configurations for each head. MoA adapts to varying input sizes, revealing that some attention heads expand their focus to accommodate longer inputs, while other heads consistently concentrate on fixed-length local contexts. Experiments show that MoA increases the effective context length by 3.9x with the same average sliding-window length, boosting retrieval accuracy by 1.5-7.1x over the uniform-window baseline across Vicuna-{7B, 13B} and Llama3-{8B, 70B} models. Moreover, MoA narrows the performance gap with full attention, reducing the maximum relative performance drop from 9%-36% to within 5% across three long-context understanding benchmarks. MoA achieves a 1.2-1.4x GPU memory reduction, boosting decode throughput by 6.6-8.2x and 1.7-1.9x over FlashAttention2 and vLLM, with minimal performance impact. Our code is available at: https://github.com/thu-nics/MoA
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
