ResiliNet：分布式神经网络中的故障恢复推理
ResiliNet: Failure-Resilient Inference in Distributed Neural Networks
摘要：
联邦学习旨在训练分布式深度模型，而无需与集中式服务器共享原始数据。类似地，在神经网络的分布式推理中，通过划分网络并将其分布在多个物理节点上，激活和梯度在物理节点之间交换，而不是原始数据。然而，当神经网络被划分并分布在物理节点之间时，物理节点的故障导致放置在这些节点上的神经单元的故障，这导致显著的性能下降。目前的方法集中在分布式神经网络中训练的弹性。然而，在分布式神经网络的推理弹性较少探索。我们介绍ResiliNet，一个计划，使推理的分布式神经网络弹性物理节点故障。ResiliNet结合了两个概念来提供弹性：跳过超连接，一个类似于resnet中的跳过连接的分布式神经网络中跳过节点的概念，以及一种称为失败的新技术，本文介绍了这种技术。Failout使用dropout在训练期间模拟物理节点故障条件，并且专门设计用于提高分布式神经网络的弹性。使用三个数据集的实验和消融研究的结果证实了ResiliNet为分布式神经网络提供推理弹性的能力。
英文摘要：
Federated Learning aims to train distributed deep models without sharing the raw data with the centralized server. Similarly, in distributed inference of neural networks, by partitioning the network and distributing it across several physical nodes, activations and gradients are exchanged between physical nodes, rather than raw data. Nevertheless, when a neural network is partitioned and distributed among physical nodes, failure of physical nodes causes the failure of the neural units that are placed on those nodes, which results in a significant performance drop. Current approaches focus on resiliency of training in distributed neural networks. However, resiliency of inference in distributed neural networks is less explored. We introduce ResiliNet, a scheme for making inference in distributed neural networks resilient to physical node failures. ResiliNet combines two concepts to provide resiliency: skip hyperconnection, a concept for skipping nodes in distributed neural networks similar to skip connection in resnets, and a novel technique called failout, which is introduced in this paper. Failout simulates physical node failure conditions during training using dropout, and is specifically designed to improve the resiliency of distributed neural networks. The results of the experiments and ablation studies using three datasets confirm the ability of ResiliNet to provide inference resiliency for distributed neural networks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
