基于学习的大数据分析框架参数自动调整
Learning-based Automatic Parameter Tuning for Big Data Analytics Frameworks
摘要：
大数据分析框架（BDAF）已被广泛用于数据处理应用。这些框架向用户提供了大量的配置参数，这导致了一个困扰用户的调优问题。为了解决这个问题，已经提出了许多自动调谐方法。然而，它仍然是一个关键的挑战，以产生足够的样本在高维参数空间内的时间约束。在本文中，我们提出了AutoTune-一个自动参数调整系统，旨在优化BDAF上的应用程序执行时间。AutoTune首先从生产系统中构建一个较小规模的测试台，以便它可以生成更多样本，从而在给定的时间约束下训练更好的预测模型。此外，AutoTune算法产生一组样本，可以在高维参数空间上提供广泛的覆盖范围，并使用训练的预测模型搜索更有希望的配置。AutoTune是使用Spark框架和HiBench基准部署在公共云上实现和评估的。大量的实验结果表明，AutoTune平均提高了63.70%的默认配置，并在五个国家的最先进的调优算法的6%-23%。
英文摘要：
Big data analytics frameworks (BDAFs) have been widely used for data processing applications. These frameworks provide a large number of configuration parameters to users, which leads to a tuning issue that overwhelms users. To address this issue, many automatic tuning approaches have been proposed. However, it remains a critical challenge to generate enough samples in a high-dimensional parameter space within a time constraint. In this paper, we present AutoTune--an automatic parameter tuning system that aims to optimize application execution time on BDAFs. AutoTune first constructs a smaller-scale testbed from the production system so that it can generate more samples, and thus train a better prediction model, under a given time constraint. Furthermore, the AutoTune algorithm produces a set of samples that can provide a wide coverage over the high-dimensional parameter space, and searches for more promising configurations using the trained prediction model. AutoTune is implemented and evaluated using the Spark framework and HiBench benchmark deployed on a public cloud. Extensive experimental results illustrate that AutoTune improves on default configurations by 63.70% on average, and on the five state-of-the-art tuning algorithms by 6%-23%.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
