ConEx：高效探索大数据系统的可扩展性以获得更好的性能
ConEx: Efficient Exploration of Big-Data System Configurations for Better Performance
摘要：
配置空间的复杂性使得大数据软件系统很难进行良好的配置。考虑Hadoop，有超过900个参数，开发人员通常只使用Hadoop发行版提供的默认配置。业绩损失的机会成本是巨大的。流行的基于学习的自动调优软件方法不能很好地扩展到大数据系统，因为收集训练数据的成本很高。我们提出了一种新的方法的基础上的组合的进化马尔可夫链蒙特卡罗（EMCMC）采样和成本降低技术，以成本有效地找到性能更好的配置大数据系统。为了降低成本，我们开发并实验测试和验证了两种方法：使用规模较大的大数据作业作为较大作业的目标函数的代理，并使用动态作业相似性度量来推断针对一种大数据问题获得的结果将适用于类似问题。我们的实验结果表明，我们的方法有望显着提高大数据系统的性能，它优于基于随机抽样，基本遗传算法（GA）和预测模型学习的竞争方法。我们的实验结果支持这样的结论，即我们的方法已经有力地证明了显着和具有成本效益地提高大数据系统的性能的潜力。
英文摘要：
Configuration space complexity makes the big-data software systems hard to configure well. Consider Hadoop, with over nine hundred parameters, developers often just use the default configurations provided with Hadoop distributions. The opportunity costs in lost performance are significant. Popular learning-based approaches to auto-tune software does not scale well for big-data systems because of the high cost of collecting training data. We present a new method based on a combination of Evolutionary Markov Chain Monte Carlo (EMCMC) sampling and cost reduction techniques to cost-effectively find better-performing configurations for big data systems. For cost reduction, we developed and experimentally tested and validated two approaches: using scaled-up big data jobs as proxies for the objective function for larger jobs and using a dynamic job similarity measure to infer that results obtained for one kind of big data problem will work well for similar problems. Our experimental results suggest that our approach promises to significantly improve the performance of big data systems and that it outperforms competing approaches based on random sampling, basic genetic algorithms (GA), and predictive model learning. Our experimental results support the conclusion that our approach has strongly demonstrated potential to significantly and cost-effectively improve the performance of big data systems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
