在NVIDIA Volta GPU上用CUTLASS实现Strassen算法
Implementing Strassen's Algorithm with CUTLASS on NVIDIA Volta GPUs
摘要:
Strassen算法的传统GPU实现(Strassen)通常依赖于现有的高性能矩阵乘法(GEMM),以空间换取时间。因此,这种方法只能实现实际的加速相对较大的,“平方”矩阵由于额外的内存开销,其使用是有限的,由于相当大的工作空间。我们提出了新的Strassen图元的GPU,可以组成生成一个家庭的Strassen算法。我们的算法利用GPU上的内存和线程层次结构,重用从GEMM继承的共享内存和寄存器文件,融合额外的操作,并避免额外的工作空间。我们进一步利用内核内和内核间的并行性,流,并采用原子操作。我们还为NVIDIA Volta GPU开发了一个性能模型,以选择适当的阻塞参数并预测GEMM和Strassen的性能。总体而言,我们的1级Strassen可以实现高达1.11倍的加速比,交叉点小至1,536,而NVIDIA Tesla V100 GPU上的cublasSgemm。通过额外的工作空间,我们的2级Strassen可以实现1.19倍的加速比,交叉点为7,680。
英文摘要:
Conventional GPU implementations of Strassen's algorithm (Strassen) typically rely on the existing high-performance matrix multiplication (GEMM), trading space for time. As a result, such approaches can only achieve practical speedup for relatively large, "squarish" matrices due to the extra memory overhead, and their usages are limited due to the considerable workspace. We present novel Strassen primitives for GPUs that can be composed to generate a family of Strassen algorithms. Our algorithms utilize both the memory and thread hierarchies on GPUs, reusing shared memory and register files inherited from GEMM, fusing additional operations, and avoiding extra workspace. We further exploit intra- and inter-kernel parallelism by batching, streaming, and employing atomic operations. We also develop a performance model for NVIDIA Volta GPUs to select the appropriate blocking parameters and predict the performance for GEMM and Strassen. Overall, our 1-level Strassen can achieve up to 1.11x speedup with a crossover point as small as 1,536 compared to cublasSgemm on a NVIDIA Tesla V100 GPU. With additional workspace, our 2-level Strassen can achieve 1.19x speedup with a crossover point at 7,680.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。