arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2303.02384cs.CVcs.DCcs.LGeess.IV
收藏

使用早期退出的深度神经网络分层训练

Hierarchical Training of Deep Neural Networks Using Early Exiting

École Polytechnique Fédérale de Lausanne (EPFL)(洛桑联邦理工学院)

Yamin Sepehri, Pedram Pad, Ahmet Caner Yüzügüler, Pascal Frossard, L. Andrea Dunbar

AI总结:

本研究提出一种基于早期退出的深度神经网络分层训练方法,在边云拆分架构中拆分反向传播过程,可并行训练且不泄露原始数据,在低比特率信道下显著缩短多种模型在图像分类任务的训练时长,准确率损失极小,适用于低资源边缘设备的在线学习。

摘要:

深度神经网络为视觉任务提供了最先进的准确率,但训练需要大量资源,因此通常在远离采集数据的边缘设备的云服务器上训练,这一问题会增加通信成本、运行时间并引发隐私担忧。本研究提出了一种新颖的深度神经网络分层训练方法,该方法在边缘与云端工作节点的拆分架构中使用早期退出机制,以降低通信成本、缩短训练运行时间并缓解隐私问题。该方法提出了早期退出的全新用例:在训练阶段将神经网络的反向传播过程拆分到边缘端和云端执行。我们解决了现有多数方法存在的问题——由于训练阶段的串行特性,这些方法无法同时训练分层的各层级,或是以牺牲隐私为代价实现并行训练。相比之下,我们的方法可同时利用边缘和云端工作节点,不会向云端共享原始输入数据,且反向传播过程中无需通信。针对不同神经网络架构开展的多项仿真和端侧实验证明了该方法的有效性。结果表明,当通过低比特率信道与云端通信时,在CIFAR-10分类任务中,该方法将VGG-16和ResNet-18架构的训练运行时间分别缩短了29%和61%;在Tiny ImageNet分类任务中,分别缩短了25%和81%。在获得运行时间提升的同时,准确率下降可忽略不计。该方法适用于在搭载传感器的低资源设备(如手机或机器人)上作为边云系统的一部分进行高精度深度神经网络的在线学习,让这类设备在面对新任务和新数据类别时更具灵活性。

英文摘要:

Deep neural networks provide state-of-the-art accuracy for vision tasks but they require significant resources for training. Thus, they are trained on cloud servers far from the edge devices that acquire the data. This issue increases communication cost, runtime and privacy concerns. In this study, a novel hierarchical training method for deep neural networks is proposed that uses early exits in a divided architecture between edge and cloud workers to reduce the communication cost, training runtime and privacy concerns. The method proposes a brand-new use case for early exits to separate the backward pass of neural networks between the edge and the cloud during the training phase. We address the issues of most available methods that due to the sequential nature of the training phase, cannot train the levels of hierarchy simultaneously or they do it with the cost of compromising privacy. In contrast, our method can use both edge and cloud workers simultaneously, does not share the raw input data with the cloud and does not require communication during the backward pass. Several simulations and on-device experiments for different neural network architectures demonstrate the effectiveness of this method. It is shown that the proposed method reduces the training runtime for VGG-16 and ResNet-18 architectures by 29% and 61% in CIFAR-10 classification and by 25% and 81% in Tiny ImageNet classification when the communication with the cloud is done over a low bit rate channel. This gain in the runtime is achieved whilst the accuracy drop is negligible. This method is advantageous for online learning of high-accuracy deep neural networks on sensor-holding low-resource devices such as mobile phones or robots as a part of an edge-cloud system, making them more flexible in facing new tasks and classes of data.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑