arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1405.1665cs.LGcs.ITmath.IT
收藏

分布式统计估计的通信代价与精确性

On Communication Cost of Distributed Statistical Estimation and Dimensionality

Princeton University(普林斯顿大学);Simons Institute, UC Berkeley(加州大学伯克利分校西蒙斯研究所)

Ankit Garg, Tengyu Ma, Huy L. Nguyen

摘要:

我们探讨了分布式学习问题的维度和通信成本之间的联系。具体来说,我们研究的问题估计的平均$\vecθ$的一个未知的$d$维高斯分布的分布设置。在此问题中,来自未知分布的样本分布在$m$不同的机器之间。目标是在最佳极大极小速率下估计平均$\vecθ$,同时尽可能少地传输比特。我们表明,在这种情况下,通信成本的规模线性的维度的数量,即一个需要单独处理不同的维度。应用这一结果以前的下界为一个维度的交互式设置\cite{ZDJW13}和我们的改进的同时设置的界限,我们证明了新的下界的$Ω(md/\log(m))$和$Ω(md)$的通信位需要实现最小最大平方损失,在交互式和同时设置。为了补充,我们还展示了一个交互式协议实现最小最大平方损失与$O(md)$位的通信,这提高了简单的同时协议的对数因子。在一般情况下,给出了强下界,我们开始研究结构参数分布参数估计问题。具体来说,当参数被承诺是$s$稀疏,我们展示了一个简单的基于阈值的协议,实现了相同的平方损失,同时节省了$d/s$因素的通信。我们推测,通信和平方损失之间的权衡表明,该协议基本上是最佳的对数因子。

英文摘要:

We explore the connection between dimensionality and communication cost in distributed learning problems. Specifically we study the problem of estimating the mean $\vecθ$ of an unknown $d$ dimensional gaussian distribution in the distributed setting. In this problem, the samples from the unknown distribution are distributed among $m$ different machines. The goal is to estimate the mean $\vecθ$ at the optimal minimax rate while communicating as few bits as possible. We show that in this setting, the communication cost scales linearly in the number of dimensions i.e. one needs to deal with different dimensions individually. Applying this result to previous lower bounds for one dimension in the interactive setting \cite{ZDJW13} and to our improved bounds for the simultaneous setting, we prove new lower bounds of $Ω(md/\log(m))$ and $Ω(md)$ for the bits of communication needed to achieve the minimax squared loss, in the interactive and simultaneous settings respectively. To complement, we also demonstrate an interactive protocol achieving the minimax squared loss with $O(md)$ bits of communication, which improves upon the simple simultaneous protocol by a logarithmic factor. Given the strong lower bounds in the general setting, we initiate the study of the distributed parameter estimation problems with structured parameters. Specifically, when the parameter is promised to be $s$-sparse, we show a simple thresholding based protocol that achieves the same squared loss while saving a $d/s$ factor of communication. We conjecture that the tradeoff between communication and squared loss demonstrated by this protocol is essentially optimal up to logarithmic factor.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑