网络负载分析和MapReduce应用的配置
Network Load Analysis and Provisioning of MapReduce Applications
摘要：
本文研究了固定大小MapReduce应用程序在shuffle阶段的配置参数与网络负载之间的依赖关系，并提出了一种分析方法来建模这种依赖关系。我们的方法包括三个关键阶段：分析，建模和预测。在第一阶段，应用程序使用不同的MapReduce配置参数（这里是映射器数量和缩减器数量）运行几次，以分析应用程序在给定集群上的shuffle阶段的网络负载。然后，这些参数和网络负载之间的关系，建立了多元线性回归模型。为了进行评估，三个应用程序（WordCount，WARTOM Mainlog解析和TeraSort）被用来评估我们的技术在4节点MapReduce私有集群。
英文摘要：
In this paper, we study the dependency between configuration parameters and network load of fixed-size MapReduce applications in shuffle phase and then propose an analytical method to model this dependency. Our approach consists of three key phases: profiling, modeling, and prediction. In the first stage, an application is run several times with different sets of MapReduce configuration parameters (here number of mappers and number of reducers) to profile the network load of the application in the shuffle phase on a given cluster. Then, the relation between these parameters and the network load is modeled by multivariate linear regression. For evaluation, three applications (WordCount, Exim Mainlog parsing, and TeraSort) are utilized to evaluate our technique on a 4-node MapReduce private cluster.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
