Volta张量核的自动生成
Automatic Kernel Generation for Volta Tensor Cores
摘要:
神经网络中一个常见的计算习惯用法是对矩阵乘法的结果进行逐点运算。这样的操作序列通常在深度学习编译器中表示为计算图。当编译到GPU目标时,这些计算可以单独映射到由cuBLAS和cuDNN等库提供的手动调优实现。这些库还为NVIDIA GPU中的目标张量核提供现成支持,通过专门支持混合精度矩阵运算,可以大幅提升性能。此外,还可以使用CUDA API或内联汇编指令直接对张量核进行编程,从而为此类计算自动生成高效CUDA内核提供可能性。当通过将多个操作融合到单个设备函数中而不是为每个操作调用单独的内核来为整个计算图生成有效代码时,自动内核生成尤其重要。多面体编译技术为仿射循环嵌套序列的分析和变换提供了一种系统的方法。在本文中,我们描述了一种多面体的方法来生成高效的CUDA内核矩阵乘法使用内联汇编指令编程张量核心的NVIDIA Volta GPU。此外,我们基于这种方法为涉及矩阵乘法和逐点操作(如偏置加法,ReLU激活等)的计算序列生成融合内核。对这些技术的实验评估表明,自动生成的内核可以提供比手动调整的库实现更好的性能,加速倍数高达2.55倍。
英文摘要:
A commonly occurring computation idiom in neural networks is to perform some pointwise operations on the result of a matrix multiplication. Such a sequence of operations is typically represented as a computation graph in deep learning compilers. When compiling to a GPU target, these computations can be individually mapped to manually tuned implementations provided by libraries such as cuBLAS and cuDNN. These libraries also provide off-the-shelf support for targeting tensor cores in NVIDIA GPUs, which can lead to huge performance boosts through their specialized support for mixed-precision matrix math. Alternatively, tensor cores can be programmed directly using CUDA APIs or inline assembly instructions, which opens up the possibility of generating efficient CUDA kernels automatically for such computations. Automatic kernel generation is particularly crucial when it is beneficial to generate efficient code for an entire computation graph by fusing several operations into a single device function instead of invoking a separate kernel for each of them. Polyhedral compilation techniques provide a systematic approach for the analysis and transformation of a sequence of affine loop-nests. In this paper, we describe a polyhedral approach to generate efficient CUDA kernels for matrix multiplication using inline assembly instructions for programming tensor cores on NVIDIA Volta GPUs. Furthermore, we build on this approach to generate fused kernels for computation sequences involving matrix multiplication and pointwise operations such as bias addition, ReLU activation etc. Experimental evaluation of these techniques show that automatically generated kernels can provide significantly better performance than manually tuned library implementations, with speedups ranging up to 2.55X.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。