H-FA：一种混合浮点数和对数方法用于硬件加速的FlashAttention
H-FA: A Hybrid Floating-Point and Logarithmic Approach to Hardware Accelerated FlashAttention
AI总结：
H-FA通过混合浮点数和对数域计算，优化FlashAttention的硬件实现，实现26.5%的面积和23.4%的功率减少，同时保持性能。
摘要：
Transformer通过其强大的注意力机制显著推动了人工智能和机器学习的发展。然而，计算长序列的注意力可能会成为计算瓶颈。FlashAttention通过将softmax和矩阵运算融合为一种分块计算模式，使性能与序列长度解耦。尽管最初设计用于GPU，其简单性也使其适合直接硬件加速。为了提高硬件实现，我们使用混合浮点数和固定点对数域表示来计算FlashAttention。浮点数用于计算查询和键矩阵的注意力分数，而对数计算简化了softmax归一化和与值矩阵相乘的融合计算。这种转换称为H-FA，用固定点算术在对数域中高效实现的加法和减法替代了向量-wide的浮点数乘法和除法运算。指数函数的评估被有效省略并与其余运算融合，最终结果直接返回到浮点运算，而无需额外的硬件开销。28nm硬件实现结果表明，与仅使用浮点数据路径构建的FlashAttention并行硬件架构相比，H-FA在平均情况下实现了26.5%的面积减少和23.4%的功率减少，而不会影响性能。
英文摘要：
Transformers have significantly advanced AI and machine learning through their powerful attention mechanism. However, computing attention on long sequences can become a computational bottleneck. FlashAttention mitigates this by fusing the softmax and matrix operations into a tiled computation pattern that decouples performance from sequence length. Though designed for GPUs, its simplicity also makes it well suited for direct hardware acceleration. To improve hardware implementation, we compute FlashAttention using a mixture of floating-point and fixed-point logarithm domain representations. Floating-point is used to compute attention scores from query and key matrices, while logarithmic computation simplifies the fused computation of softmax normalization and the multiplication with the value matrix. This transformation, called H-FA, replaces vector-wide floating-point multiplication and division operations by additions and subtractions implemented efficiently with fixed-point arithmetic in the logarithm domain. Exponential function evaluations are effectively omitted and fused with the rest operations, and the final result is directly returned to floating-point arithmetic without any additional hardware overhead. Hardware implementation results at 28nm demonstrate that H-FA achieves a 26.5% reduction in area and a 23.4% reduction in power, on average, compared to FlashAttention parallel hardware architectures built solely with floating-point datapaths, without hindering performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
