arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.11016cs.ARcs.AI
收藏

从缓冲区到寄存器:通过混合键合3D NPU协同设计解锁细粒度FlashAttention

From Buffers to Registers: Unlocking Fine-Grained FlashAttention with Hybrid-Bonded 3D NPU Co-Design

SKLP, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China(科学技术信息研究所、中国科学院、北京);University of Chinese Academy of Sciences, Beijing, China(中国科学院大学、北京)

Jinxin Yu, Yudong Pan, Mengdi Wang, Huawei Li, Yinhe Han, Xiaowei Li, Ying Wang

AI总结:

本文提出3D-Flow架构和3D-FlashAttention方法,通过混合键合3D堆叠空间加速器实现细粒度FlashAttention,降低能耗并提升速度。

摘要:

基于Transformer的模型主导了现代AI工作负载,但因其二次注意力复杂度和不断增长的模型规模而加剧了内存瓶颈。现有加速器,如Groq和Cerebras,通过大容量片上缓存来缓解掉电交通,而算法创新如FlashAttention通过融合运算来避免生成大型注意力矩阵。然而,随着掉电交通的减少,我们的测量显示,在长序列工作负载中,片上SRAM访问占据了超过60%的能量,使缓存访问成为新的瓶颈。我们提出了3D-Flow,一种混合键合的3D堆叠空间加速器,能够在垂直划分的PE层级之间实现寄存器到寄存器的通信。不同于受限于基于NoC的路由到路由传输的2D多数组架构,3D-Flow利用亚10微米的垂直TSV来维持周期级的运算流水线,具有最小的开销。在此架构之上,我们设计了3D-FlashAttention,一种细粒度调度方法,平衡各层级的延迟,形成无气泡的垂直数据流,无需片上SRAM循环。在Transformer工作负载(OPT和QWEN模型)上的评估显示,我们的3D空间加速器将46-93%的能耗降低,并相对于最先进的2D和3D设计实现了1.4x-7.6x的加速。

英文摘要:

Transformer-based models dominate modern AI workloads but exacerbate memory bottlenecks due to their quadratic attention complexity and ever-growing model sizes. Existing accelerators, such as Groq and Cerebras, mitigate off-chip traffic with large on-chip caches, while algorithmic innovations such as FlashAttention fuse operators to avoid materializing large attention matrices. However, as off-chip traffic decreases, our measurements show that on-chip SRAM accesses account for over 60% of energy in long-sequence workloads, making cache access the new bottleneck. We propose 3D-Flow, a hybrid-bonded, 3D-stacked spatial accelerator that enables register-to-register communication across vertically partitioned PE tiers. Unlike 2D multi-array architectures limited by NoC-based router-to-router transfers, 3D-Flow leverages sub-10 um vertical TSVs to sustain cycle-level operator pipelining with minimal overhead. On top of this architecture, we design 3D-FlashAttention, a fine-grained scheduling method that balances latency across tiers, forming a bubble-free vertical dataflow without on-chip SRAM roundtrips. Evaluations on Transformer workloads (OPT and QWEN models) show that our 3D spatial accelerator reduces 46-93% energy consumption and achieves 1.4x-7.6x speedups compared to state-of-the-art 2D and 3D designs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑