arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1112.3765cs.DCcs.CC
收藏

并行外部存储器中MapReduce的效率

The Efficiency of MapReduce in Parallel External Memory

Gero Greiner, Riko Jacob

摘要:

自2004年引入以来,MapReduce框架已经成为大规模分布式和并行计算的标准方法之一。与其在实践中的广泛应用相反,理论基础仍然有限,并且只有很少的工作尚未将MapReduce与主要的计算模型相提并论。在将MapReduce框架与PRAM和BSP在其宏观结构中联系起来的先驱工作之后,我们专注于框架本身提供的功能,考虑并行外部内存模型(PEM)。在这方面,我们提出了上界和下界的并行I/O的复杂性,匹配到常数因子的洗牌步骤。shuffle步骤是一个通信阶段,其中一个MapReduce调用的所有信息都从map worker传输到reduce worker。因此,与之前的工作相比,我们将重点转向内部沟通步骤。我们得到的结果进一步结转到BSP* 模型。一方面,这表明与PEM相比,MapReduce中表达的算法可以“隐藏”多少复杂性。另一方面,我们的结果限制了MapReduce方法在I/O效率方面的最坏情况下的性能损失。

英文摘要:

Since its introduction in 2004, the MapReduce framework has become one of the standard approaches in massive distributed and parallel computation. In contrast to its intensive use in practise, theoretical footing is still limited and only little work has been done yet to put MapReduce on a par with the major computational models. Following pioneer work that relates the MapReduce framework with PRAM and BSP in their macroscopic structure, we focus on the functionality provided by the framework itself, considered in the parallel external memory model (PEM). In this, we present upper and lower bounds on the parallel I/O-complexity that are matching up to constant factors for the shuffle step. The shuffle step is the single communication phase where all information of one MapReduce invocation gets transferred from map workers to reduce workers. Hence, we move the focus towards the internal communication step in contrast to previous work. The results we obtain further carry over to the BSP* model. On the one hand, this shows how much complexity can be "hidden" for an algorithm expressed in MapReduce compared to PEM. On the other hand, our results bound the worst-case performance loss of the MapReduce approach in terms of I/O-efficiency.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑