关于数据偏度、落后者和MapReduce进度指标
On data skewness, stragglers, and MapReduce progress indicators
摘要：
我们解决了预测MapReduce应用程序性能的问题，设计了准确的进度指示器，让程序员了解作业执行过程中完成计算时间的百分比。通过大量的实验，我们发现，最先进的进度指标（包括Hadoop提供的进度指标）可能会受到数据偏斜、负载不平衡和分散任务的严重损害。这主要是由于他们隐含的假设，即运行时间线性依赖于输入大小。因此，我们设计了一种新的配置文件引导的进度指标，称为NearestFit，没有线性假设假设的情况下运作，并利用最近邻回归和统计曲线拟合技术的精心组合。我们的理论进度模型需要细粒度的配置文件数据，这在实践中可能非常难以管理。为了克服这个问题，我们通过空间和时间高效的数据流算法来计算模型中使用的一些量的精确近似值。我们在Hadoop 2.6.0之上实施了NearestFit。在Amazon EC2平台上对各种真实基准进行的广泛实证评估表明，NearestFit在实际应用中是实用的。它的准确性通常非常好，即使在竞争对手产生不可忽略的错误和广泛的预测波动的情况下。总的来说，NearestFit显著改善了MapReduce进度分析的当前状态。
英文摘要：
We tackle the problem of predicting the performance of MapReduce applications, designing accurate progress indicators that keep programmers informed on the percentage of completed computation time during the execution of a job. Through extensive experiments, we show that state-of-the-art progress indicators (including the one provided by Hadoop) can be seriously harmed by data skewness, load unbalancing, and straggling tasks. This is mainly due to their implicit assumption that the running time depends linearly on the input size. We thus design a novel profile-guided progress indicator, called NearestFit, that operates without the linear hypothesis assumption and exploits a careful combination of nearest neighbor regression and statistical curve fitting techniques. Our theoretical progress model requires fine-grained profile data, that can be very difficult to manage in practice. To overcome this issue, we resort to computing accurate approximations for some of the quantities used in our model through space- and time-efficient data streaming algorithms. We implemented NearestFit on top of Hadoop 2.6.0. An extensive empirical assessment over the Amazon EC2 platform on a variety of real-world benchmarks shows that NearestFit is practical w.r.t. space and time overheads and that its accuracy is generally very good, even in scenarios where competitors incur non-negligible errors and wide prediction fluctuations. Overall, NearestFit significantly improves the current state-of-art on progress analysis for MapReduce.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
