arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2201.06769cs.LG
收藏

DEFER:深度神经网络的分布式边缘推理

DEFER: Distributed Edge Inference for Deep Neural Networks

Crystal Springs Uplands School(水晶泉高地学校);University of Southern California(南加州大学)

Arjun Parthasarathy, Bhaskar Krishnamachari

摘要:

深度神经网络(DNN)等现代机器学习工具正在自然语言处理、计算机视觉和物联网等多个领域发挥革命性作用。一旦经过训练,深度学习模型就可以部署在边缘计算机上,以对这些应用程序的实时数据进行分类和预测。特别是对于大型模型,单个边缘设备上有限的计算和内存资源可能成为推理管道的吞吐量瓶颈。为了提高吞吐量并降低每个设备的计算负载,我们提出了DEFER(分布式边缘推理),这是一个分布式边缘推理的框架,它将深度神经网络划分为可以跨多个计算节点分布的层。该架构由单个“调度器”节点组成,用于将DNN分区和推理数据分发到相应的计算节点。计算节点以串联模式连接,其中每个节点的计算结果被中继到后续节点。然后将结果返回给Dispatcher。我们量化的吞吐量,能源消耗,网络负载,和开销为我们的框架在现实的网络条件下使用CORE网络仿真器。我们发现,对于ResNet 50模型,具有8个计算节点的DEFER的推理吞吐量比单设备推理高出53%,每个节点的能耗低63%。我们进一步减少网络通信的需求和能源消耗使用ZFP串行化和LZ4压缩算法。我们使用TensorFlow和Keras ML库在Python中实现了DEFER,并将DEFER作为开源框架发布,以造福研究社区。

英文摘要:

Modern machine learning tools such as deep neural networks (DNNs) are playing a revolutionary role in many fields such as natural language processing, computer vision, and the internet of things. Once they are trained, deep learning models can be deployed on edge computers to perform classification and prediction on real-time data for these applications. Particularly for large models, the limited computational and memory resources on a single edge device can become the throughput bottleneck for an inference pipeline. To increase throughput and decrease per-device compute load, we present DEFER (Distributed Edge inFERence), a framework for distributed edge inference, which partitions deep neural networks into layers that can be spread across multiple compute nodes. The architecture consists of a single "dispatcher" node to distribute DNN partitions and inference data to respective compute nodes. The compute nodes are connected in a series pattern where each node's computed result is relayed to the subsequent node. The result is then returned to the Dispatcher. We quantify the throughput, energy consumption, network payload, and overhead for our framework under realistic network conditions using the CORE network emulator. We find that for the ResNet50 model, the inference throughput of DEFER with 8 compute nodes is 53% higher and per node energy consumption is 63% lower than single device inference. We further reduce network communication demands and energy consumption using the ZFP serialization and LZ4 compression algorithms. We have implemented DEFER in Python using the TensorFlow and Keras ML libraries, and have released DEFER as an open-source framework to benefit the research community.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑