arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2511.11571cs.LGcs.CL
收藏

优化块注意力混合

Optimizing Mixture of Block Attention

MIT(麻省理工学院);NVIDIA(英伟达)

Guangxuan Xiao, Junxian Guo, Kasra Mazaheri, Song Han

AI总结:

本文提出改进MoBA的FlashMoBA,通过优化块大小和键卷积提升路由精度,并在GPU上实现高效执行,使理论改进落地。

摘要:

混合块注意力(MoBA)(Lu等人,2025)是一种有前景的构建块,用于通过使查询能够稀疏地关注少量的关键值块,从而高效地处理LLMs中的长上下文。然而,MoBA性能的指导原则尚不明确,且缺乏高效的GPU实现,阻碍了其实际应用。在本文中,我们首先开发了一个统计模型来分析MoBA的底层机制。我们的模型揭示,性能严重依赖于路由器根据查询-键亲和力准确区分相关和不相关块的能力。我们推导出一个信噪比,正式将架构参数与这种检索准确性联系起来。受我们的分析启发,我们识别出两个改进路径:使用更小的块大小和在键上应用短卷积以聚类相关信号,从而提高路由准确性。虽然理论上更好,但小块大小在GPU上效率低下。为了弥合这一差距,我们引入了FlashMoBA,一种硬件-aware的CUDA内核,它使即使在我们理论推荐的小块大小下也能高效执行MoBA。我们通过从头训练LLMs验证了我们的见解,显示我们的改进MoBA模型与密集注意力基线性能相当。FlashMoBA在小块上相对于FlashAttention-2实现了高达14.7倍的加速,使我们的理论指导的改进变得实用。代码可在:https://github.com/mit-han-lab/flash-moba 上获得。

英文摘要:

Mixture of Block Attention (MoBA) (Lu et al., 2025) is a promising building block for efficiently processing long contexts in LLMs by enabling queries to sparsely attend to a small subset of key-value blocks, drastically reducing computational cost. However, the design principles governing MoBA's performance are poorly understood, and it lacks an efficient GPU implementation, hindering its practical adoption. In this paper, we first develop a statistical model to analyze MoBA's underlying mechanics. Our model reveals that performance critically depends on the router's ability to accurately distinguish relevant from irrelevant blocks based on query-key affinities. We derive a signal-to-noise ratio that formally connects architectural parameters to this retrieval accuracy. Guided by our analysis, we identify two key pathways for improvement: using smaller block sizes and applying a short convolution on keys to cluster relevant signals, which enhances routing accuracy. While theoretically better, small block sizes are inefficient on GPUs. To bridge this gap, we introduce FlashMoBA, a hardware-aware CUDA kernel that enables efficient MoBA execution even with the small block sizes our theory recommends. We validate our insights by training LLMs from scratch, showing that our improved MoBA models match the performance of dense attention baselines. FlashMoBA achieves up to 14.7x speedup over FlashAttention-2 for small blocks, making our theoretically-grounded improvements practical. Code is available at: https://github.com/mit-han-lab/flash-moba.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑