arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1206.4377cs.DCcs.DS
收藏

Map-Reduce计算代价的上下限

Upper and Lower Bounds on the Cost of a Map-Reduce Computation

Foto N. Afrati, Anish Das Sarma, Semih Salihoglu, Jeffrey D. Ullman

摘要:

在本文中,我们研究了映射缩减计算中并行性和通信开销之间的权衡。对于任何不是“并行的”问题,我们对归约器的工作划分得越细,从而可以提取更多的并行性,映射器和归约器之间的总通信就越多。我们介绍了一个模型的问题,可以解决在一个单一的一轮地图减少计算。该模型使一个通用的配方,发现通信成本的最大数量的输入,可以分配给一个减少的函数的下限。我们使用该模型来分析三个问题的权衡:在汉明距离$d$上找到字符串对,在更大的图中找到三角形和其他模式,以及矩阵乘法。为了找到汉明距离为1的字符串,我们有精确匹配的上界和下界。对于三角形和许多其他图,我们有上界和下界,它们在一个常数因子内是相同的。对于矩阵乘法问题,我们为一轮映射约简算法提供了匹配的上界和下界。我们还能够探索用于矩阵乘法的两轮映射缩减算法,并表明对于给定的缩减器大小,这些算法从来没有比最好的一轮算法更多的通信,并且通常显着更少。

英文摘要:

In this paper we study the tradeoff between parallelism and communication cost in a map-reduce computation. For any problem that is not "embarrassingly parallel," the finer we partition the work of the reducers so that more parallelism can be extracted, the greater will be the total communication between mappers and reducers. We introduce a model of problems that can be solved in a single round of map-reduce computation. This model enables a generic recipe for discovering lower bounds on communication cost as a function of the maximum number of inputs that can be assigned to one reducer. We use the model to analyze the tradeoff for three problems: finding pairs of strings at Hamming distance $d$, finding triangles and other patterns in a larger graph, and matrix multiplication. For finding strings of Hamming distance 1, we have upper and lower bounds that match exactly. For triangles and many other graphs, we have upper and lower bounds that are the same to within a constant factor. For the problem of matrix multiplication, we have matching upper and lower bounds for one-round map-reduce algorithms. We are also able to explore two-round map-reduce algorithms for matrix multiplication and show that these never have more communication, for a given reducer size, than the best one-round algorithm, and often have significantly less.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑