arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1905.01748cs.DS
收藏

MapReduce满足细粒度复杂性:用于APSP、矩阵乘法、3- SUM及其他的MapReduce算法

MapReduce Meets Fine-Grained Complexity: MapReduce Algorithms for APSP, Matrix Multiplication, 3-SUM, and Beyond

MohammadTaghi Hajiaghayi, Silvio Lattanzi, Saeed Seddighin, Cliff Stein

摘要:

分布式处理框架,如MapReduce、Hadoop和Spark是处理大量数据的流行系统。在这些框架中设计有效的算法是一个具有挑战性的问题,因为系统都需要并行性-因为数据集如此之大,需要多台机器-并且限制并行度-因为机器的数量在数据大小中呈次线性增长。虽然MapReduce已经有十几年的历史了~ \cite{dean2008mapreduce},但许多基本问题,如矩阵乘法,3-SUM和所有对最短路径,都缺乏有效的MapReduce算法。我们在MapReduce设置中研究这些问题。我们的主要贡献是展示每个机器上可用的内存和每个问题所需的机器总数之间的平滑权衡。总的来说,我们把每台机器可用的内存作为一个参数,目标是最小化轮数和机器数。在本文中,我们建立在著名的MapReduce的理论框架,由Karloff,Suri,和Vassilvitskii ~ \cite{karloff2010model},并给出了算法的许多这些问题。在这种情况下,有效算法的关键在于定义一个次线性数量的大型(多项式大小)子问题,然后可以并行求解。我们给出了MapReduce友好分区的策略,从而为上述所有问题提供了新的算法。具体来说,我们给恒定轮算法的正交向量(OV)和3-SUM问题,和$O(\log n)$轮算法矩阵乘法,所有对最短路径(APSP),快速傅立叶变换(FFT),等等。在所有这些中,我们展示了机器数量和每台机器内存之间的权衡。

英文摘要:

Distributed processing frameworks, such as MapReduce, Hadoop, and Spark are popular systems for processing large amounts of data. The design of efficient algorithms in these frameworks is a challenging problem, as the systems both require parallelism---since datasets are so large that multiple machines are necessary---and limit the degree of parallelism---since the number of machines grows sublinearly in the size of the data. Although MapReduce is over a dozen years old~\cite{dean2008mapreduce}, many fundamental problems, such as Matrix Multiplication, 3-SUM, and All Pairs Shortest Paths, lack efficient MapReduce algorithms. We study these problems in the MapReduce setting. Our main contribution is to exhibit smooth trade-offs between the memory available on each machine, and the total number of machines necessary for each problem. Overall, we take the memory available to each machine as a parameter, and aim to minimize the number of rounds and number of machines. In this paper, we build on the well-known MapReduce theoretical framework initiated by Karloff, Suri, and Vassilvitskii ~\cite{karloff2010model} and give algorithms for many of these problems. The key to efficient algorithms in this setting lies in defining a sublinear number of large (polynomially sized) subproblems, that can then be solved in parallel. We give strategies for MapReduce-friendly partitioning, that result in new algorithms for all of the above problems. Specifically, we give constant round algorithms for the Orthogonal Vector (OV) and 3-SUM problems, and $O(\log n)$-round algorithms for Matrix Multiplication, All Pairs Shortest Paths (APSP), and Fast Fourier Transform (FFT), among others. In all of these we exhibit trade-offs between the number of machines and memory per machine.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑