arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2605.16617cs.DC
收藏

超越IEEE-754 SGEMM在GPU上使用BF16张量核心进行科学计算的数值和性能特性

Exceeding the Numerical and Performance Characteristics of IEEE-754 SGEMM with BFloat16 Tensor Cores on GPUs for Scientific Computing

NVIDIA Corporation(英伟达公司)

Harun Bayraktar, Cole Brower, John Gunnels, Greg Henry, Cherin Joseph, Jack Kosaian, Dmitry Lyakh, Lukas Mosimann, Victor Podlozhnyuk, Addison Richards, Paul Springer, Haicheng Wu

AI总结:

本文研究了使用BF16张量核心进行FP32矩阵乘法的仿真实现,展示了其在数值和性能上超越原生FP32的能力,并讨论了完整的库级实现以正确处理下溢数。

摘要:

由于其增加的数值强度和能效,低精度浮点计算资源,主要应用于人工智能领域,增长速度比高精度相对而言更快。这导致了各种努力,旨在利用丰富的低精度硬件来模拟高精度数学计算。本文研究了一个特定用例,即使用现代GPU上的BF16张量核心来服务单精度(FP32)矩阵乘法运算。鉴于BF16和FP32具有相同的动态范围,将BF16操作累积到FP32累加器(全速)以及Blackwell GPU架构特有的BF16算术特性,如集成的缩放硬件,使得这种仿真实现具有高度动机。本文考察了通过BF16仿真实现的FP32矩阵乘法的性能、效率、能效和数值特性,并展示了其在科学应用中超越原生FP32的数值和性能特性。我们还讨论了完整的库级实现,以正确处理下溢数。

英文摘要:

Largely due to their increased native capacity for numerical intensity and power efficiency, reduced-precision floating-point computing resources, primarily used in artificial intelligence (AI) applications, have expanded at a greater rate than their higher-precision relatives. This has led to various efforts focused upon leveraging plentiful reduced-precision hardware to mimic higher-precision mathematical calculations. This paper studies a specific use case, namely the use of bfloat16 (BF16) Tensor Cores found on modern GPUs in service of single precision (FP32) matrix multiply operations. Given that BF16 and FP32 share the same dynamic range, the option to accumulate BF16 operations into FP32 accumulators (at full-speed), and additional BF16 arithmetic characteristics specific to the Blackwell GPU architecture, such as integrated scaling hardware, such emulation is highly motivated. This paper examines the performance, efficiency, power, and numerical characteristics of FP32 matrix multiplication via BF16-based emulation and demonstrates how it exceeds numerical and performance characteristics of native FP32 for scientific applications. We also discuss a full library-ready implementation that correctly deals with denormals.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑