arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2408.05247cs.DCcs.AIcs.LGcs.NI
收藏

边缘网络中早退机制与模型分布式推理的结合

Early-Exit meets Model-Distributed Inference at Edge Networks

University of Illinois Chicago(伊利诺伊大学芝加哥分校)

Marco Colocrese, Erdem Koyuncu, Hulya Seferoglu

AI总结:

本文提出MDI-Exit框架,将早退机制引入模型分布式推理,通过自适应确定早退、卸载和数据准入策略,在边缘设备上实现固定精度下处理更多数据或固定数据速率下获得更高精度。

摘要:

分布式推理技术可大致分为数据分布式和模型分布式方案。在数据分布式推理(DDI)中,每个工作节点承载完整的深度神经网络(DNN)模型,但只处理数据的一个子集。然而,将数据馈送到工作节点会产生高昂的通信成本,尤其是当数据量较大时。一种新兴的范式是模型分布式推理(MDI),其中每个工作节点只承载DNN层的一个子集。在MDI中,拥有数据的源设备处理DNN的若干层,并将输出发送到邻近设备,即卸载其余层的计算。当所有层以分布式方式处理完毕时,该过程结束。本文研究了带有早退机制的MDI的设计与开发,其主张是:对于某些数据而言,无需处理模型的所有层即可达到期望精度,也就是说,如果达到目标精度,我们可以在不处理所有层的情况下退出模型。我们设计了一个名为MDI-Exit的框架,该框架自适应地确定早退和卸载策略以及源端的数据准入策略。在NVIDIA Nano边缘设备真实测试平台上的实验结果表明,在精度固定的情况下,MDI-Exit能够处理更多数据;在数据速率固定的情况下,MDI-Exit能够获得更高的精度。

英文摘要:

Distributed inference techniques can be broadly classified into data-distributed and model-distributed schemes. In data-distributed inference (DDI), each worker carries the entire deep neural network (DNN) model but processes only a subset of the data. However, feeding the data to workers results in high communication costs, especially when the data is large. An emerging paradigm is model-distributed inference (MDI), where each worker carries only a subset of DNN layers. In MDI, a source device that has data processes a few layers of DNN and sends the output to a neighboring device, i.e., offloads the rest of the layers. This process ends when all layers are processed in a distributed manner. In this paper, we investigate the design and development of MDI with early-exit, which advocates that there is no need to process all the layers of a model for some data to reach the desired accuracy, i.e., we can exit the model without processing all the layers if target accuracy is reached. We design a framework MDI-Exit that adaptively determines early-exit and offloading policies as well as data admission at the source. Experimental results on a real-life testbed of NVIDIA Nano edge devices show that MDI-Exit processes more data when accuracy is fixed and results in higher accuracy for the fixed data rate.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑