对NVIDIA Blackwell架构的微基准测试:深入的架构分析
Microbenchmarking NVIDIA's Blackwell Architecture: An in-depth Architectural Analysis
AI总结:
本文通过微基准测试评估NVIDIA Blackwell架构在内存子系统、张量核心流水线和浮点精度方面的性能,发现其在混合精度训练中比H200提升1.85倍和1.55倍,能效提高32%。
摘要:
随着GPU架构迅速发展以满足exascale计算和机器学习日益增长的需求,架构创新的性能影响在各种工作负载中仍缺乏系统的理解。NVIDIA Blackwell(B200)引入了显著的架构进步,包括第五代张量核心、张量内存(TMEM)、解压缩引擎(DE)和双芯片设计;然而,量化这些改进的系统化方法仍落后于硬件开发周期。我们贡献了一个开源的微基准测试套件,为优化工作负载以充分利用现代GPU架构的丰富功能集提供实用见解。这项工作使应用程序开发者能够做出有根据的架构决策,并指导未来的GPU设计方向。我们研究了Blackwell GPU并将其与H200一代在内存子系统、张量核心流水线和浮点精度(FP32,FP16,FP8,FP6,FP4)方面进行比较。我们对密集和稀疏GEMM、变压器推断和训练工作负载的系统评估显示,B200张量核心改进在ResNet-50和GPT-1.3B混合精度训练吞吐量上分别达到1.85倍和1.55倍,比H200具有32%更好的能效。
英文摘要:
As GPU architectures rapidly evolve to meet the growing demands of exascale computing and machine learning, the performance implications of architectural innovations remain poorly understood across diverse workloads. NVIDIA Blackwell (B200) introduces significant architectural advances, including fifth-generation tensor cores, tensor memory (TMEM), a decompression engine (DE), and a dual-chip design; however, systematic methodologies for quantifying these improvements lag behind hardware development cycles. We contribute an open-source microbenchmark suite that provides practical insights into optimizing workloads to fully utilize the rich feature sets of modern GPU architectures. This work enables application developers to make informed architectural decisions and guides future GPU design directions. We study Blackwell GPUs and compare them to the H200 generation with respect to the memory subsystem, tensor core pipeline, and floating-point precisions (FP32, FP16, FP8, FP6, FP4). Our systematic evaluation of dense and sparse GEMM, transformer inference, and training workloads shows that B200 tensor core enhancements achieve 1.85x ResNet-50 and 1.55x GPT-1.3B mixed-precision training throughput, with 32 percent better energy efficiency than H200.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。