The introduction of the new multi-user linearly-separable distributed computing framework, has recently revealed how a parallel treatment of users can yield large parallelization gains with relatively low computation and communication costs. These gains stem from a new approach that converts the computing problem into a sparse matrix factorization problem; a matrix \(\mathbf{F}\) that describes the users' requests, is decomposed as \(\mathbf{F} = \mathbf{DE}\), where a \(γ\)-sparse \(\mathbf{E}\) defines the task allocation across \(N\) servers, and a \(δ\)-sparse \(\mathbf{D}\) defines the connectivity between \(N\) servers and \(K\) users as well as the decoding process. While this approach provides near-optimal performance, its linear nature has raised data secrecy concerns.
We adopt an information-theoretic secrecy framework requiring that each user learns nothing more than its own requested function. Our main results provide (i) a necessary condition stating that for each user $k$ observing \(α_k\) server responses, the common randomness visible to that user must span a subspace of dimension greater than \(α_k-1\), and (ii) a necessary and sufficient condition requiring that removing from \(\mathbf{D}\) the columns corresponding to the servers observed by a user leaves a matrix of rank at least \(K-1\). Based on these conditions, we design a general, cost-preserving secrecy-enforcing transformation valid over both finite and real fields, obtained by appending to \(\mathbf{E}\) a basis of \(\mathrm{Null}(\mathbf{D})\) and carefully injecting shared randomness. This scheme preserves communication and computation costs, guarantees perfect information-theoretic secrecy over finite fields, and in the real case yields an explicit mutual-information bound that can be made arbitrarily small by increasing the variance of Gaussian common randomness.
Distributed Linearly Separable Computation with Arbitrary Heterogeneous Data Assignment
Huazhong University of Science and Technology（华中科技大学）；Guangxi Normal University（广西师范大学）；University of Shanghai for Science and Technology（上海理工大学）；Technische Universität Berlin（柏林工业大学）
Learning ReLU Networks on Linearly Separable Data: Algorithm, Optimality, and Generalization
University of Minnesota（明尼苏达大学）；Beijing Institute of Technology（北京理工大学）；Tongji University（同济大学）
具有Rectified Linear Unit（ReLU）激活函数的神经网络（也称为ReLU网络）在各个领域取得了巨大的经验成功。尽管如此，现有的学习ReLU网络的结果要么假设底层数据分布是高斯分布，要么要求网络大小和/或训练大小足够大。在这种情况下，学习双层ReLU网络的问题是在二进制分类设置中进行的，其中数据是线性可分离的，并且采用铰链损失标准。利用随机噪声扰动的力量，本文提出了一种新的随机梯度下降（SGD）算法，该算法可以训练任何单隐藏层ReLU网络以达到全局最优，尽管存在无限多个坏的局部极小值，极大值和鞍点。这是同类结果中的第一个，不需要对数据分布，训练/网络大小或初始化进行假设。收敛到一个全局最小的迭代算法进行了分析，通过建立一个上限和一个下限的非零更新的数量进行。此外，为使用利用经典压缩边界的新型SGD训练的ReLU网络开发了泛化保证。这些保证突出了可靠学习ReLU网络和泄漏ReLU网络在样本复杂性方面的关键区别（至少在最坏的情况下）。仿真数据和实际图像的数值试验验证了算法的有效性和理论的实用价值。
