arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1609.05181cs.ITcs.DCcs.LGmath.IT
收藏

分布式学习中数据置乱的信息论极限

Information Theoretic Limits of Data Shuffling for Distributed Learning

University of Arizona(亚利桑那大学)

Mohamed Attia, Ravi Tandon

摘要:

数据混洗是分布式学习算法的基本构建块之一,它增加了学习过程中每一步的统计增益。在每次迭代中,不同的混洗数据点由中心节点分配给一组分布式工作者来执行本地计算,这导致了通信瓶颈。本文的重点是形式化和理解的基本信息理论之间的权衡存储(每个工人)和最坏情况下的数据洗牌问题的通信开销。我们完全描述了信息理论的权衡$K=2$,和$K=3$工人,为任何值的存储容量,并表明,增加跨工人的存储可以通过利用编码来减少通信开销。我们提出了一种新的和系统的数据传输和存储更新策略,为每个数据洗牌迭代,它保留了整个工人的存储的结构属性,并有助于在随后的数据洗牌迭代的通信开销最小化。

英文摘要:

Data shuffling is one of the fundamental building blocks for distributed learning algorithms, that increases the statistical gain for each step of the learning process. In each iteration, different shuffled data points are assigned by a central node to a distributed set of workers to perform local computations, which leads to communication bottlenecks. The focus of this paper is on formalizing and understanding the fundamental information-theoretic trade-off between storage (per worker) and the worst-case communication overhead for the data shuffling problem. We completely characterize the information theoretic trade-off for $K=2$, and $K=3$ workers, for any value of storage capacity, and show that increasing the storage across workers can reduce the communication overhead by leveraging coding. We propose a novel and systematic data delivery and storage update strategy for each data shuffle iteration, which preserves the structural properties of the storage across the workers, and aids in minimizing the communication overhead in subsequent data shuffling iterations.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑