arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2001.05438cs.ITmath.IT
收藏

基于扩展离散矩阵的低复杂度分布式计算

Low Complexity Distributed Computing via Binary Matrices with Extension to Stragglers

Shailja Agrawal, Prasad Krishnan

摘要:

我们考虑MapReduce的分布式计算框架,它包括三个阶段,Map阶段,Shuffle阶段和Reduce阶段。对于这个框架,我们建议使用称为\textit{computing matrices}的二进制矩阵(具有$0,1$条目)来描述映射阶段和洗牌阶段。类似的二进制矩阵最近被提出用于编码缓存框架。二进制计算矩阵中的1和0的结构捕获了MapReduce框架的map阶段。我们提出了一个新的简单的编码数据洗牌计划,这个二进制矩阵模型的基础上,计算矩阵的\textit{identity submatrix cover}。这种新的编码混洗方案通常具有比现有方案更大的通信负载,但是在文件分割和所需的相关联的索引和协调方面具有比文献中的公知的早期方案更少的复杂度开销的优点。我们还表明,存在一个二进制矩阵为基础的分布式计算计划与我们的新的数据洗牌计划,严格小于两倍的通信负载比已知的最佳方案在文献中。这种新方案的结构使其能够以简单的方式应用于具有离散者的MapReduce框架,从无离散者的情况中借用其优点和缺点。最后,使用来自组合设计的二进制矩阵,我们展示了特定类别的计算方案,具有非常低的\textit{file complexity}(文件中的子文件的数量),与同等参数的最佳方案相比,通信负载略高。

英文摘要:

We consider the distributed computing framework of MapReduce, which consists of three phases, the Map phase, the Shuffle phase and the Reduce phase. For this framework, we propose the use of binary matrices (with $0,1$ entries) called \textit{computing matrices} to describe the map phase and the shuffle phase. Similar binary matrices were recently proposed for the coded caching framework. The structure of ones and zeroes in the binary computing matrix captures the map phase of the MapReduce framework. We present a new simple coded data shuffling scheme for this binary matrix model, based on a \textit{identity submatrix cover} of the computing matrix. This new coded shuffling scheme has in general a larger communication load than existing schemes, but has the advantage of less complexity overhead than the well-known earlier schemes in literature in terms of the file-splitting and associated indexing and coordination required. We also show that there exists a binary matrix based distributed computing scheme with our new data-shuffling scheme which has strictly less than twice than the communication load of the known optimal scheme in literature. The structure of this new scheme enables it to be applied to the framework of MapReduce with stragglers also, in a straightforward manner, borrowing its advantages and disadvantages from the no-straggler situation. Finally, using binary matrices derived from combinatorial designs, we show specific classes of computing schemes with very low \textit{file complexity} (number of subfiles in the file), with marginally higher communication load compared to the optimal scheme for equivalent parameters.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑