arXiv 1702.048502017-02-16cs.DCcs.ITmath.IT

编码TeraSort

Coded TeraSort

Songze Li, Sucha Supittayapornpong, Mohammad Ali Maddah-Ali, A. Salman Avestimehr

备注：: to appear in proceedings of 2017 International Workshop on Parallel and Distributed Computing for Large Scale Machine Learning and Big Data Analytics

摘要：

我们专注于排序，这是许多机器学习算法的构建块，并提出了一种新的分布式排序算法，命名为编码的TeraSort，它大大提高了Hadoop MapReduce中的TeraSort基准的执行时间。编码TeraSort的关键思想是在数据中施加结构化冗余，以实现克服TeraSort的数据重排瓶颈的网络内编码机会。我们在Amazon EC2集群上实证评估了CodedTeraSort算法的性能，并证明了与TeraSort相比，对于典型的感兴趣设置，它实现了1.97倍-3.39倍的加速比。

英文摘要：

We focus on sorting, which is the building block of many machine learning algorithms, and propose a novel distributed sorting algorithm, named Coded TeraSort, which substantially improves the execution time of the TeraSort benchmark in Hadoop MapReduce. The key idea of Coded TeraSort is to impose structured redundancy in data, in order to enable in-network coding opportunities that overcome the data shuffling bottleneck of TeraSort. We empirically evaluate the performance of CodedTeraSort algorithm on Amazon EC2 clusters, and demonstrate that it achieves 1.97x - 3.39x speedup, compared with TeraSort, for typical settings of interest.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。