面向CNN的带交错算子划分的协同推理
Cooperative Inference with Interleaved Operator Partitioning for CNNs
AI总结：
针对物联网设备部署深度学习模型的资源限制问题，提出交错算子划分（IOP）策略及对应模型分割算法，无需激活拼接即可降低协同推理延迟，相比现有方法在分类模型上实现更快加速与更低峰值内存占用。
摘要：
在物联网（IoT）设备上部署深度学习模型常因内存资源和计算能力有限面临挑战，协同推理是解决该问题的重要方法，需要对智能模型进行划分并分布式部署。现有协同推理方法执行水平划分时，要么以算子的输出通道为划分维度，要么以特征图的高度和宽度为划分维度。这种方式下，由于算子的激活值被拆分，必须先将它们拼接起来才能输入下一个算子，这会带来协同推理的延迟。本文提出了面向CNN模型的交错算子划分（Interleaved Operator Partitioning, IOP）策略。通过按输出通道维度划分一个算子，并按输入通道维度划分其后续算子，无需进行激活值拼接，从而减少了通信连接数量，最终降低协同推理延迟。基于IOP，我们进一步提出了一种最小化协同推理时间的模型分割算法，该算法根据获得的推理延迟收益，贪婪地选择算子进行IOP配对。实验结果表明，与CoEdge中使用的最先进划分方法相比，IOP策略在三个经典图像分类模型上实现了6.39%~16.83%的更快加速比，并将峰值内存占用降低了21.22%~49.98%。
英文摘要：
Deploying deep learning models on Internet of Things (IoT) devices often faces challenges due to limited memory resources and computing capabilities. Cooperative inference is an important method for addressing this issue, requiring the partitioning and distributive deployment of an intelligent model. To perform horizontal partitions, existing cooperative inference methods take either the output channel of operators or the height and width of feature maps as the partition dimensions. In this manner, since the activation of operators is distributed, they have to be concatenated together before being fed to the next operator, which incurs the delay for cooperative inference. In this paper, we propose the Interleaved Operator Partitioning (IOP) strategy for CNN models. By partitioning an operator based on the output channel dimension and its successive operator based on the input channel dimension, activation concatenation becomes unnecessary, thereby reducing the number of communication connections, which consequently reduces cooperative inference de-lay. Based on IOP, we further present a model segmentation algorithm for minimizing cooperative inference time, which greedily selects operators for IOP pairing based on the inference delay benefit harvested. Experimental results demonstrate that compared with the state-of-the-art partition approaches used in CoEdge, the IOP strategy achieves 6.39% ~ 16.83% faster acceleration and reduces peak memory footprint by 21.22% ~ 49.98% for three classical image classification models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
