PipeOrgan:具有灵活空间组织与互连的高效算子间流水线
PipeOrgan: Efficient Inter-operation Pipelining with Flexible Spatial Organization and Interconnects
AI总结:
本文提出 PipeOrgan,一种面向 DNN 加速器的灵活空间数据组织策略,通过支持可变流水线深度与粒度并改进网格拓扑,在 XR-bench 上实现 1.95 倍性能提升。
摘要:
由于近期深度神经网络(DNN)模型呈现内存受限趋势,面向 DNN 加速器的算子间流水线正在成为一种有前景的优化方法。算子间流水线通过在计算阵列内部将某一层的输出作为下一层的输入进行转发,减少了昂贵的片上全局存储和片外存储访问,先前工作已证明这是一种有效的优化手段。然而,算子间流水线的设计空间非常巨大,且该空间尚未被充分探索。特别是,确定合适的流水线深度和粒度(或完全不采用流水线)在很大程度上取决于层的形状以及权重和激活的数据量,即使在同一领域内这些因素也各不相同。此外,现有工作将底层硬件划分为较大的块,并将每一层映射到一个块上,这需要在执行中途或通过全局缓冲区进行通信。然而,对于细粒度算子间流水线而言,将下一层 tile 的对应消费者放置在当前层 tile 的生产者附近,是更好地利用细粒度空间复用的一种方式。为了在底层硬件上支持可变层数(即合适的深度)并支持层的多种空间组织方式(与流水线粒度相适应),我们提出了 PipeOrgan,这是一类新的空间数据组织策略,可在不同流水线深度和粒度下实现处理单元(PE)之间高能效且无拥塞的通信。PipeOrgan 利用灵活的空间组织,并可根据流水线粒度将层分配到 PE 上。我们还提出了对传统网格拓扑的改进,以提高粗粒度分配的性能。在 XR-bench 工作负载上,PipeOrgan 相比最先进的流水线数据流实现了 1.95 倍的性能提升。
英文摘要:
Because of the recent trends in Deep Neural Networks (DNN) models being memory-bound, inter-operator pipelining for DNN accelerators is emerging as a promising optimization. Inter-operator pipelining reduces costly on-chip global memory and off-chip memory accesses by forwarding the output of a layer as the input of the next layer within the compute array, which is proven to be an effective optimization by previous works. However, the design space of inter-operator pipelining is huge, and the space is not yet fully explored. In particular, identifying the right depth and granularity of pipelining (or no pipelining at all) is significantly dependent on the layer shapes and data volumes of weights and activations, and these are different even within a domain. Moreover, works divide the substrate into large chunks and map one layer onto each chunk, which requires communicating halfway through or through the global buffer. However, for fine-grained inter-operation pipelining, placing the corresponding consumer of the next layer tile close to the producer tile of the current layer is a better way to exploit fine-grained spatial reuse. In order to support variable number of layers (ie the right depth) and support multiple spatial organizations of layers (in accordance with the pipelining granularity) on the substrate, we propose PipeOrgan, a new class of spatial data organization strategy for energy efficient and congestion-free communication between the PEs for various pipeline depth and granularity. PipeOrgan takes advantage of flexible spatial organization and can allocate layers to PEs based on the granularity of pipelining. We also propose changes to the conventional mesh topology to improve the performance of coarse-grained allocation. PipeOrgan achieves 1.95x performance improvement over the state-of-the-art pipelined dataflow on XR-bench workloads.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。