This paper studies the distributed linearly separable computation problem, which is a generalization of many existing distributed computing problems such as distributed gradient descent and distributed linear transform. In this problem, a master asks $N$ distributed workers to compute a linearly separable function of $K$ datasets, which is a set of $K_c$ linear combinations of $K$ messages (each message is a function of one dataset). We assign some datasets to each worker, which then computes the corresponding messages and returns some function of these messages, such that from the answers of any $N_r$ out of $N$ workers the master can recover the task function. In the literature, the specific case where $K_c = 1$ or where the computation cost is minimum has been considered. In this paper, we focus on the general case (i.e., general $K_c$ and general computation cost) and aim to find the minimum communication cost. We first propose a novel converse bound on the communication cost under the constraint of the popular cyclic assignment (widely considered in the literature), which assigns the datasets to the workers in a cyclic way. Motivated by the observation that existing strategies for distributed computing fall short of achieving the converse bound, we propose a novel distributed computing scheme for some system parameters. The proposed computing scheme is optimal for any assignment when $K_c$ is large and is optimal under cyclic assignment when the numbers of workers and datasets are equal or $K_c$ is small. In addition, it is order optimal within a factor of 2 under cyclic assignment for the remaining cases.
Distributed Linearly Separable Computation with Arbitrary Heterogeneous Data Assignment
Huazhong University of Science and Technology(华中科技大学);Guangxi Normal University(广西师范大学);University of Shanghai for Science and Technology(上海理工大学);Technische Universität Berlin(柏林工业大学)
Learning ReLU Networks on Linearly Separable Data: Algorithm, Optimality, and Generalization
University of Minnesota(明尼苏达大学);Beijing Institute of Technology(北京理工大学);Tongji University(同济大学)
具有Rectified Linear Unit(ReLU)激活函数的神经网络(也称为ReLU网络)在各个领域取得了巨大的经验成功。尽管如此,现有的学习ReLU网络的结果要么假设底层数据分布是高斯分布,要么要求网络大小和/或训练大小足够大。在这种情况下,学习双层ReLU网络的问题是在二进制分类设置中进行的,其中数据是线性可分离的,并且采用铰链损失标准。利用随机噪声扰动的力量,本文提出了一种新的随机梯度下降(SGD)算法,该算法可以训练任何单隐藏层ReLU网络以达到全局最优,尽管存在无限多个坏的局部极小值,极大值和鞍点。这是同类结果中的第一个,不需要对数据分布,训练/网络大小或初始化进行假设。收敛到一个全局最小的迭代算法进行了分析,通过建立一个上限和一个下限的非零更新的数量进行。此外,为使用利用经典压缩边界的新型SGD训练的ReLU网络开发了泛化保证。这些保证突出了可靠学习ReLU网络和泄漏ReLU网络在样本复杂性方面的关键区别(至少在最坏的情况下)。仿真数据和实际图像的数值试验验证了算法的有效性和理论的实用价值。