面向边缘设备的Google神经网络模型：分析和缓解机器学习推理瓶颈
Google Neural Network Models for Edge Devices: Analyzing and Mitigating Machine Learning Inference Bottlenecks
摘要：
新兴的边缘计算平台通常包含机器学习（ML）加速器，可以加速各种神经网络（NN）模型的推理。这些模型被设计为适合边缘计算平台的有限区域和能量约束，每个模型针对各种应用（例如，面部检测、语音识别、翻译、图像描述、视频分析）。为了了解边缘机器学习加速器的性能，我们使用24个Google Edge NN模型（涵盖各种NN模型类型）并分析每个模型中的每个NN层来表征商业Google Edge TPU的性能。我们发现Edge TPU存在三个主要缺点：（1）它的运行速度明显低于峰值计算吞吐量，（2）它的运行速度明显低于其理论能效，（3）它的存储系统是一个很大的能量和性能瓶颈。我们的表征表明，Edge TPU的一刀切的整体设计忽略了不同NN模型之间以及同一NN模型中不同NN层之间的高度异质性，导致了我们观察到的缺点。我们提出了一个新的加速框架，称为门萨。Mensa集成了多个异构边缘ML加速器（包括片上和近数据加速器），每个加速器都满足NN模型和层的特定子集的特征。在NN推理过程中，对于每个NN层，Mensa决定在哪个加速器上调度该层，同时考虑每个加速器对该层的最优性和层间通信成本。在所有24个Google edge NN模型中，Mensa将能效和吞吐量提高了3.0倍和3.1倍，比Edge TPU提高了2.4倍和4.3倍，而Eyeriss~v2是最先进的加速器。
英文摘要：
Emerging edge computing platforms often contain machine learning (ML) accelerators that can accelerate inference for a wide range of neural network (NN) models. These models are designed to fit within the limited area and energy constraints of the edge computing platforms, each targeting various applications (e.g., face detection, speech recognition, translation, image captioning, video analytics). To understand how edge ML accelerators perform, we characterize the performance of a commercial Google Edge TPU, using 24 Google edge NN models (which span a wide range of NN model types) and analyzing each NN layer within each model. We find that the Edge TPU suffers from three major shortcomings: (1) it operates significantly below peak computational throughput, (2) it operates significantly below its theoretical energy efficiency, and (3) its memory system is a large energy and performance bottleneck. Our characterization reveals that the one-size-fits-all, monolithic design of the Edge TPU ignores the high degree of heterogeneity both across different NN models and across different NN layers within the same NN model, leading to the shortcomings we observe. We propose a new acceleration framework called Mensa. Mensa incorporates multiple heterogeneous edge ML accelerators (including both on-chip and near-data accelerators), each of which caters to the characteristics of a particular subset of NN models and layers. During NN inference, for each NN layer, Mensa decides which accelerator to schedule the layer on, taking into account both the optimality of each accelerator for the layer and layer-to-layer communication costs. Averaged across all 24 Google edge NN models, Mensa improves energy efficiency and throughput by 3.0x and 3.1x over the Edge TPU, and by 2.4x and 4.3x over Eyeriss~v2, a state-of-the-art accelerator.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
