GPU上灵活的高性能GEMM内核
Flexible Performant GEMM Kernels on GPUs
摘要：
通用矩阵乘法或GEMM内核在高性能计算和机器学习中占据中心位置。最近的NVIDIA GPU包括GEMM加速器，例如NVIDIA的Tensor Cores。它们的开发受到两种语言问题的阻碍：它要么需要低级编程，这意味着程序员的生产力很低，要么使用只提供有限组件集的库。因为根据已建立的组件来改写算法通常会引入开销，库缺乏灵活性限制了探索新算法的自由。因此，使用GEMM的研究人员无法同时享受编程效率、高性能和研究灵活性。在本文中，我们解决这个问题。我们提出了三套抽象和接口的科学朱莉娅程序设计语言内的程序GEMM。接口和抽象是针对研究人员的需求和Julia的功能共同设计的，以实现足够的关注点分离和灵活性，从而以多种不同的方式轻松扩展基本GEMM，而无需付出性能代价。将我们的GEMM与最先进的库cuBLAS和CUTLASS进行比较，我们证明了我们的性能与库相同，在某些情况下甚至超过了它，而无需在CUDA C++或汇编中编写一行代码，并且不会面临灵活性限制。
英文摘要：
General Matrix Multiplication or GEMM kernels take centre place in high performance computing and machine learning. Recent NVIDIA GPUs include GEMM accelerators, such as NVIDIA's Tensor Cores. Their exploitation is hampered by the two-language problem: it requires either low-level programming which implies low programmer productivity or using libraries that only offer a limited set of components. Because rephrasing algorithms in terms of established components often introduces overhead, the libraries' lack of flexibility limits the freedom to explore new algorithms. Researchers using GEMMs can hence not enjoy programming productivity, high performance, and research flexibility at once. In this paper we solve this problem. We present three sets of abstractions and interfaces to program GEMMs within the scientific Julia programming language. The interfaces and abstractions are co-designed for researchers' needs and Julia's features to achieve sufficient separation of concerns and flexibility to easily extend basic GEMMs in many different ways without paying a performance price. Comparing our GEMMs to state-of-the-art libraries cuBLAS and CUTLASS, we demonstrate that our performance is in the same ballpark of the libraries, and in some cases even exceeds it, without having to write a single line of code in CUDA C++ or assembly, and without facing flexibility limitations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
