基于CPU的CNN模型推理优化
Optimizing CNN Model Inference on CPUs
摘要：
卷积神经网络（CNN）模型的流行和CPU的普及意味着CPU上CNN模型推理的更好性能可以为大量用户提供显著的收益。为了提高CPU上CNN推理的性能，目前的方法（如MXNet和Intel OpenVINO）通常将模型视为图，并使用Intel MKL-DNN等高性能库来实现图的操作。虽然在来自现成库的各个操作上实现了合理的性能，但该解决方案使得在图级别进行优化变得不灵活，因为本地操作级别优化是预定义的。因此，它是限制性的，并且错过了整体优化端到端推理流水线的机会。本文介绍了\emph{NeoCPU}，一种在CPU上进行CNN模型推理的综合方法，它采用了全栈和系统的优化方案。\emph{NeoCPU}将操作作为模板进行优化，而不依赖于第三方库，这使得通过操作级和图级联合优化进一步提高性能。实验表明，\emph{NeoCPU}实现了高达3.45 $\times$更低的延迟CNN模型推理比目前最先进的各种流行的CPU上的实现。
英文摘要：
The popularity of Convolutional Neural Network (CNN) models and the ubiquity of CPUs imply that better performance of CNN model inference on CPUs can deliver significant gain to a large number of users. To improve the performance of CNN inference on CPUs, current approaches like MXNet and Intel OpenVINO usually treat the model as a graph and use the high-performance libraries such as Intel MKL-DNN to implement the operations of the graph. While achieving reasonable performance on individual operations from the off-the-shelf libraries, this solution makes it inflexible to conduct optimizations at the graph level, as the local operation-level optimizations are predefined. Therefore, it is restrictive and misses the opportunity to optimize the end-to-end inference pipeline as a whole. This paper presents \emph{NeoCPU}, a comprehensive approach of CNN model inference on CPUs that employs a full-stack and systematic scheme of optimizations. \emph{NeoCPU} optimizes the operations as templates without relying on third-parties libraries, which enables further improvement of the performance via operation- and graph-level joint optimization. Experiments show that \emph{NeoCPU} achieves up to 3.45$\times$ lower latency for CNN model inference than the current state-of-the-art implementations on various kinds of popular CPUs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
