Proximu$：通过近缓存计算在多核CPU中有效扩展DNN推理
Proximu$: Efficiently Scaling DNN Inference in Multi-core CPUs through Near-Cache Compute
摘要：
深度神经网络（DNN）推理正在成为众多公用事业和服务的基础。CPU继续扩展其DNN推理的原始计算能力以及成熟的高性能库，以提取最佳性能。虽然通用CPU为数据中心和边缘的DNN推理提供了独特的吸引力优势，但它们主要是为了优化单线程性能而发展的。对于高度并行、面向吞吐量的DNN推理，这会导致功耗和性能效率低下，从而影响原始性能扩展和整体性能功耗比。我们提出了Proximu $\$$，在这里我们系统地解决了CPU DNN推理的功耗和性能扩展的根本效率低下问题。通过将轻量级张量计算分布在多级缓存层次结构中的所有缓存附近，可以有效地扩展性能。这最大化了系统中现有带宽资源的累积利用率，并最小化了数据的移动。通过简单的ISA扩展（对结构化的、循环的工作负载行为进行编码），功耗大大降低。这使预解码工作的批量卸载成为可能，在轻量级近缓存单元中展开循环，有效地绕过了宽无序（OOO）CPU流水线的功耗级。在许多DNN模型中，Proximu $\$$的卷积性能/功耗提高了2.3倍，原始性能提高了2倍至3.94倍。类似地，Proximu $\$$实现了1.8倍的内部产品性能/瓦增长，性能扩展2.8倍。Proximu $\$$无需更改编程模型，无需增加缓存容量或带宽，只需最少的额外硬件，即可实现前所未有的CPU效率提升，同时在AI时代实现与用于DNN推理的最先进领域特定加速器（DSA）相似的性能。
英文摘要：
Deep Neural Network (DNN) inference is emerging as the fundamental bedrock for a multitude of utilities and services. CPUs continue to scale up their raw compute capabilities for DNN inference along with mature high performance libraries to extract optimal performance. While general purpose CPUs offer unique attractive advantages for DNN inference at both datacenter and edge, they have primarily evolved to optimize single thread performance. For highly parallel, throughput-oriented DNN inference, this results in inefficiencies in both power and performance, impacting both raw performance scaling and overall performance/watt. We present Proximu$\$$, where we systematically tackle the root inefficiencies in power and performance scaling for CPU DNN inference. Performance scales efficiently by distributing light-weight tensor compute near all caches in a multi-level cache hierarchy. This maximizes the cumulative utilization of the existing bandwidth resources in the system and minimizes movement of data. Power is drastically reduced through simple ISA extensions that encode the structured, loop-y workload behavior. This enables a bulk offload of pre-decoded work, with loop unrolling in the light-weight near-cache units, effectively bypassing the power-hungry stages of the wide Out-of-Order (OOO) CPU pipeline. Across a number of DNN models, Proximu$\$$ achieves a 2.3x increase in convolution performance/watt with a 2x to 3.94x scaling in raw performance. Similarly, Proximu$\$$ achieves a 1.8x increase in inner-product performance/watt with 2.8x scaling in performance. With no changes to the programming model, no increase in cache capacity or bandwidth and minimal additional hardware, Proximu$\$$ enables unprecedented CPU efficiency gains while achieving similar performance to state-of-the-art Domain Specific Accelerators (DSA) for DNN inference in this AI era.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
