一种张量处理单元的数据中心内性能分析
In-Datacenter Performance Analysis of a Tensor Processing Unit
摘要：
许多架构师认为，成本-能源-性能的主要改进现在必须来自特定领域的硬件。本文评估了自2015年以来部署在计算机中心的自定义ASIC-称为张量处理单元（TPU）-加速神经网络（NN）的推理阶段。TPU的核心是一个65，536 8位MAC矩阵乘法单元，可提供92 TeraOps/秒（TOPS）的峰值吞吐量和一个大型（28 MiB）软件管理片内存储器。TPU的确定性执行模型比CPU和GPU的时变优化（缓存、乱序执行、多线程、多处理、预取等）更好地匹配NN应用程序的99百分位响应时间要求。这有助于平均吞吐量超过保证的延迟。缺乏这些功能有助于解释为什么尽管有无数的MAC和大内存，TPU却相对较小且功耗较低。我们将TPU与服务器级Intel Haswell CPU和Nvidia K80 GPU进行了比较，它们是部署在相同处理器中的同时代产品。我们的工作负载是在高级TensorFlow框架中编写的，使用生产NN应用程序（MLP，CNN和LSTM），这些应用程序代表了我们分析中心95%的NN推理需求。尽管某些应用的利用率较低，但TPU的平均速度比其当代GPU或CPU快约15 - 30倍，TOPS/W高约30 - 80倍。此外，在TPU中使用GPU的GDDR5内存将使TOPS达到三倍，并将TOPS/Watt提高到GPU的近70倍和CPU的200倍。
英文摘要：
Many architects believe that major improvements in cost-energy-performance must now come from domain-specific hardware. This paper evaluates a custom ASIC---called a Tensor Processing Unit (TPU)---deployed in datacenters since 2015 that accelerates the inference phase of neural networks (NN). The heart of the TPU is a 65,536 8-bit MAC matrix multiply unit that offers a peak throughput of 92 TeraOps/second (TOPS) and a large (28 MiB) software-managed on-chip memory. The TPU's deterministic execution model is a better match to the 99th-percentile response-time requirement of our NN applications than are the time-varying optimizations of CPUs and GPUs (caches, out-of-order execution, multithreading, multiprocessing, prefetching, ...) that help average throughput more than guaranteed latency. The lack of such features helps explain why, despite having myriad MACs and a big memory, the TPU is relatively small and low power. We compare the TPU to a server-class Intel Haswell CPU and an Nvidia K80 GPU, which are contemporaries deployed in the same datacenters. Our workload, written in the high-level TensorFlow framework, uses production NN applications (MLPs, CNNs, and LSTMs) that represent 95% of our datacenters' NN inference demand. Despite low utilization for some applications, the TPU is on average about 15X - 30X faster than its contemporary GPU or CPU, with TOPS/Watt about 30X - 80X higher. Moreover, using the GPU's GDDR5 memory in the TPU would triple achieved TOPS and raise TOPS/Watt to nearly 70X the GPU and 200X the CPU.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
