arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2512.06989cs.LGcs.AIcs.CL
收藏

Flash多头前馈网络

Flash Multi-Head Feed-Forward Network

ShanghaiTech University(上海科技大学);Ant Group(蚂蚁集团)

Minshen Zhang, Xiang Hu, Jianguo Li, Wei Wu, Kewei Tu

AI总结:

FlashMHF通过I/O感知融合内核和动态加权并行子网络设计,提升了FFN的效率和可扩展性,减少内存消耗并加速推理。

摘要:

我们探索多头前馈网络(MH-FFN)作为Transformer架构中前馈网络(FFN)的替代方案,受单头注意力与FFN结构相似性的启发。尽管多头机制在注意力中增强了表达能力,但简单地将多头机制应用于FFN面临两个挑战:内存消耗随头数增加而增加,以及随着模型规模扩大,中间层大小的增长与固定头维度之间的比例失衡,这会降低可扩展性和表达能力。为了解决这些挑战,我们提出了Flash多头前馈网络(FlashMHF),其两个关键创新点是:一种I/O感知的融合内核计算,能够在SRAM中实时生成输出,类似于FlashAttention;以及一种使用动态加权并行子网络的设计,以保持中间层和头维度之间的比例平衡。在从128M到1.3B参数的模型上验证,FlashMHF在困惑度和下游任务准确性上均优于SwiGLU前馈网络,同时将峰值内存使用量减少了3-5倍,并将推理速度提高了高达1.08倍。我们的工作确立了多头设计作为FFN的优越架构原则,将FlashMHF作为Transformer中FFN的强大、高效且可扩展的替代方案。

英文摘要:

We explore Multi-Head FFN (MH-FFN) as a replacement of FFN in the Transformer architecture, motivated by the structural similarity between single-head attention and FFN. While multi-head mechanisms enhance expressivity in attention, naively applying them to FFNs faces two challenges: memory consumption scaling with the head count, and an imbalanced ratio between the growing intermediate size and the fixed head dimension as models scale, which degrades scalability and expressive power. To address these challenges, we propose Flash Multi-Head FFN (FlashMHF), with two key innovations: an I/O-aware fused kernel computing outputs online in SRAM akin to FlashAttention, and a design using dynamically weighted parallel sub-networks to maintain a balanced ratio between intermediate and head dimensions. Validated on models from 128M to 1.3B parameters, FlashMHF consistently improves perplexity and downstream task accuracy over SwiGLU FFNs, while reducing peak memory usage by 3-5x and accelerating inference by up to 1.08x. Our work establishes the multi-head design as a superior architectural principle for FFNs, presenting FlashMHF as a powerful, efficient, and scalable alternative to FFNs in Transformers.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑