非对称位宽操作数间的高吞吐量矩阵-矩阵乘法
High Throughput Matrix-Matrix Multiplication between Asymmetric Bit-Width Operands
摘要：
非对称位宽操作数之间的矩阵乘法，特别是8位和4位操作数之间的矩阵乘法可能成为许多重要工作负载的基本内核，包括神经网络和机器学习。虽然用于对称位宽操作数的现有SIMD矩阵乘法指令可以通过零或符号扩展窄操作数以匹配其他操作数的大小来支持混合精度的操作数，但是它们不能利用操作数之一的窄位宽的益处。我们提出了一种新的SIMD矩阵乘法指令，在其输入上使用混合精度（8位和4位操作数），并将乘积值累加为较窄的16位输出乘积，进而允许128位向量宽度处的SIMD操作每指令处理更多数量的数据元素，以提高处理吞吐量和内存带宽利用率，而不增加寄存器读和写，CPU中的端口带宽。与使用现有的非对称操作数大小指令获得的吞吐量相比，所提出的非对称操作数大小SIMD指令在矩阵乘法的吞吐量方面提供了2倍的改进，同时对于代表性的机器学习工作负载引起来自16位乘法的可忽略不计（0.05%）的溢出。非对称操作数大小指令不仅可以提高CPU中的矩阵乘法吞吐量，而且可以有效地支持现有技术的DNN硬件加速器（例如，Google TPU中脉动阵列微架构等）并且在不违反各种实现约束的情况下无缝地提供矩阵乘法性能的类似改进。我们演示了如何脉动阵列架构设计的非对称操作数大小的指令可以修改，以支持一个非对称操作数大小的指令。
英文摘要：
Matrix multiplications between asymmetric bit-width operands, especially between 8- and 4-bit operands are likely to become a fundamental kernel of many important workloads including neural networks and machine learning. While existing SIMD matrix multiplication instructions for symmetric bit-width operands can support operands of mixed precision by zero- or sign-extending the narrow operand to match the size of the other operands, they cannot exploit the benefit of narrow bit-width of one of the operands. We propose a new SIMD matrix multiplication instruction that uses mixed precision on its inputs (8- and 4-bit operands) and accumulates product values into narrower 16-bit output accumulators, in turn allowing the SIMD operation at 128-bit vector width to process a greater number of data elements per instruction to improve processing throughput and memory bandwidth utilization without increasing the register read- and write-port bandwidth in CPUs. The proposed asymmetric-operand-size SIMD instruction offers 2x improvement in throughput of matrix multiplication in comparison to throughput obtained using existing symmetric-operand-size instructions while causing negligible (0.05%) overflow from 16-bit accumulators for representative machine learning workloads. The asymmetric-operand-size instruction not only can improve matrix multiplication throughput in CPUs, but also can be effective to support multiply-and-accumulate (MAC) operation between 8- and 4-bit operands in state-of-the-art DNN hardware accelerators (e.g., systolic array microarchitecture in Google TPU, etc.) and offer similar improvement in matrix multiply performance seamlessly without violating the various implementation constraints. We demonstrate how a systolic array architecture designed for symmetric-operand-size instructions could be modified to support an asymmetric-operand-sized instruction.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
