一种并行分布式神经网络训练框架
A Framework for Parallel and Distributed Training of Neural Networks
摘要:
本文的目的是开发一个通用的框架,用于训练神经网络(NN)在分布式环境中,训练数据被划分在一组代理,通过一个稀疏的,可能随时间变化的,连接模式相互通信。在这种分布式场景中,训练问题可以被公式化为非凸社会成本函数的(正则化)优化,由局部(非凸)成本之和给出,其中每个代理贡献相对于其局部数据集定义的单个误差项。为了设计一个灵活而有效的解决方案,我们定制了最近提出的框架,在网络上的非凸优化,它取决于(原始)凸分解技术来处理非凸性,和一个动态的共识过程中传播的代理之间的信息。训练标准的几个典型选择(例如,平方损失、交叉熵等)以及正则化(例如,$\ell_2$范数、稀疏诱导惩罚等)都包含在框架中,并在本文中进行了探讨。收敛到一个固定的解决方案的社会非凸问题是保证在温和的假设。此外,我们展示了一种原则性的方式,允许每个代理利用可能的多核架构(例如,本地云),以便并行化其本地优化步骤,从而产生本质上既分布式(跨代理)又并行(在每个代理内部)的策略。一组全面的实验结果验证了所提出的方法。
英文摘要:
The aim of this paper is to develop a general framework for training neural networks (NNs) in a distributed environment, where training data is partitioned over a set of agents that communicate with each other through a sparse, possibly time-varying, connectivity pattern. In such distributed scenario, the training problem can be formulated as the (regularized) optimization of a non-convex social cost function, given by the sum of local (non-convex) costs, where each agent contributes with a single error term defined with respect to its local dataset. To devise a flexible and efficient solution, we customize a recently proposed framework for non-convex optimization over networks, which hinges on a (primal) convexification-decomposition technique to handle non-convexity, and a dynamic consensus procedure to diffuse information among the agents. Several typical choices for the training criterion (e.g., squared loss, cross entropy, etc.) and regularization (e.g., $\ell_2$ norm, sparsity inducing penalties, etc.) are included in the framework and explored along the paper. Convergence to a stationary solution of the social non-convex problem is guaranteed under mild assumptions. Additionally, we show a principled way allowing each agent to exploit a possible multi-core architecture (e.g., a local cloud) in order to parallelize its local optimization step, resulting in strategies that are both distributed (across the agents) and parallel (inside each agent) in nature. A comprehensive set of experimental results validate the proposed approach.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。