arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1603.07936cs.DC
收藏

OptEx:一个具有截止期意识的Spark成本优化模型

OptEx: A Deadline-Aware Cost Optimization Model for Spark

Subhajit Sidhanta, Wojciech Golab, Supratik Mukhopadhyay

摘要:

我们提出了OptEx,Apache Spark,一个流行的并行处理引擎上的作业执行的封闭形式的模型。据我们所知,OptEx是第一个在Spark上对作业完成时间进行分析建模的工作。该模型可用于估计云上给定Spark作业的完成时间,包括输入数据集的大小,迭代次数,组成底层集群的节点数量。实验结果表明,OptEx产生的平均相对误差为6%的估计作业完成时间。此外,该模型可以应用于估计在SLO中指定的完成期限下在云上运行给定Spark作业的成本最优集群组成(即,服务水平目标)。我们的实验表明,OptEx是能够正确地估计成本最优的集群组成运行一个给定的Spark作业下的SLO的最后期限的准确性为98%。

英文摘要:

We present OptEx, a closed-form model of job execution on Apache Spark, a popular parallel processing engine. To the best of our knowledge, OptEx is the first work that analytically models job completion time on Spark. The model can be used to estimate the completion time of a given Spark job on a cloud, with respect to the size of the input dataset, the number of iterations, the number of nodes comprising the underlying cluster. Experimental results demonstrate that OptEx yields a mean relative error of 6% in estimating the job completion time. Furthermore, the model can be applied for estimating the cost optimal cluster composition for running a given Spark job on a cloud under a completion deadline specified in the SLO (i.e., Service Level Objective). We show experimentally that OptEx is able to correctly estimate the cost optimal cluster composition for running a given Spark job under an SLO deadline with an accuracy of 98%.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑