MAFAT:面向加速边缘推理的神经网络内存感知融合与分块
MAFAT: Memory-Aware Fusing and Tiling of Neural Networks for Accelerated Edge Inference
AI总结:
针对边缘设备内存不足导致卷积神经网络推理延迟高的问题,提出MAFAT方法,通过多组卷积层独立融合分块、内存预测加搜索优化配置,在降内存的同时实现低延迟推理。
摘要:
在资源受限的边缘设备上本地运行高算力需求的机器学习(ML)网络是一项日益突出的研究挑战。带有大型卷积层的ML网络很容易超出可用内存,由于操作系统过度交换内存页面而导致延迟升高。此前的内存缩减技术(如剪枝和量化)会降低模型精度,且通常需要重新训练。另一种方案是分布式方法,将卷积拆分为等效的小型子计算,但这类实现会带来通信开销,且需要多设备组成的网络。不过,分布式拆分思路也可通过将网络细分为更小的运算,在单设备上以更低的内存占用运行。本文将分布式拆分的先前研究拓展为单设备上的内存感知执行方案。我们的方法拓展了已有的融合策略,支持多组卷积层分别独立进行融合与分块。这使得能够在开销与数据复用之间进行权衡,从而专门降低内存占用。我们提出了一种内存使用预测器,结合搜索算法,可为任意卷积层集合提供优化的融合与分块配置。将该方法应用于YOLOv2目标检测网络的实验结果表明,在严重内存受限条件下,我们的方法能以不到一半的内存运行,且加速比最高可达2.78。此外,我们的算法返回的配置,其延迟与手动搜索测得的最佳延迟差距在6%以内。
英文摘要:
A rising research challenge is running costly machine learning (ML) networks locally on resource-constrained edge devices. ML networks with large convolutional layers can easily exceed available memory, increasing latency due to excessive OS swapping. Previous memory reduction techniques such as pruning and quantization reduce model accuracy and often require retraining. Alternatively, distributed methods partition the convolutions into equivalent smaller sub-computations, but the implementations introduce communication costs and require a network of devices. Distributed partitioning approaches can, however, also be used to run in a reduced memory footprint on a single device by subdividing the network into smaller operations. In this paper, we extend prior work on distributed partitioning into a memory-aware execution on a single device. Our approach extends prior fusing strategies to allow for multiple groups of convolutional layers that are fused and tiled independently. This enables trading off overhead versus data reuse in order to specifically reduces memory footprint. We propose a memory usage predictor coupled with a search algorithm to provide optimized fusing and tiling configurations for an arbitrary set of convolutional layers. When applied to the YOLOv2 object detection network, results show that our approach can run in less than half the memory, and with a speedup of up to 2.78 under severe memory constraints. Additionally, our algorithm will return a configuration with a latency that is within 6% of the best latency measured in a manual search.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。