arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2203.14889cs.DC
收藏

LOCAT:Spark SQL应用程序的低开销在线配置自动调整

LOCAT: Low-Overhead Online Configuration Auto-Tuning of Spark SQL Applications

Jinhan Xin, Kai Hwang, Zhibin Yu

摘要:

Spark SQL已经在行业中广泛部署,但调整其性能具有挑战性。最近的研究试图采用机器学习(ML)来解决这个问题,但存在两个缺点。首先,收集训练样本需要很长时间(高开销)。第二,对于同一应用程序的一个输入数据大小的最佳配置对于其他应用程序可能不是最佳的。为了解决这些问题,我们提出了一种新的贝叶斯优化(BO)的方法命名为LOCAT自动调整Spark SQL应用程序的配置在线。LOCAT创新了三种技术。第一种技术,称为QCSA,通过查询配置敏感性分析(QCSA)收集训练样本时,消除配置不敏感的查询。第二种技术,称为DAGP,是数据大小感知高斯过程(DAGP),其将应用程序的性能建模为配置参数以及输入数据大小的函数的分布。第三种技术称为IICP,它会根据性能调整重要配置参数(IICP),并且只调整重要的配置参数。因此,LOCAT可以以较低的开销调整Spark SQL应用程序的配置,并适应不同的输入数据大小。我们采用了来自基准测试套件TPC-DS、TPC-H和HiBench的Spark SQL应用程序,这些应用程序运行在两个显著不同的集群上,一个是四节点ARM集群,一个是八节点x86集群,来评估LOCAT。在ARM集群上的实验结果表明,LOCAT将最先进方法的优化过程加速了至少4.1倍和高达9.7倍;此外,LOCAT将应用程序性能提高了至少1.9倍和高达2.4倍。在x86集群上,LOCAT显示了与ARM集群上类似的结果。

英文摘要:

Spark SQL has been widely deployed in industry but it is challenging to tune its performance. Recent studies try to employ machine learning (ML) to solve this problem, but suffer from two drawbacks. First, it takes a long time (high overhead) to collect training samples. Second, the optimal configuration for one input data size of the same application might not be optimal for others. To address these issues, we propose a novel Bayesian Optimization (BO) based approach named LOCAT to automatically tune the configurations of Spark SQL applications online. LOCAT innovates three techniques. The first technique, named QCSA, eliminates the configuration-insensitive queries by Query Configuration Sensitivity Analysis (QCSA) when collecting training samples. The second technique, dubbed DAGP, is a Datasize-Aware Gaussian Process (DAGP) which models the performance of an application as a distribution of functions of configuration parameters as well as input data size. The third technique, called IICP, Identifies Important Configuration Parameters (IICP) with respect to performance and only tunes the important ones. As such, LOCAT can tune the configurations of a Spark SQL application with low overhead and adapt to different input data sizes. We employ Spark SQL applications from benchmark suites TPC-DS, TPC-H, and HiBench running on two significantly different clusters, a four-node ARM cluster and an eight-node x86 cluster, to evaluate LOCAT. The experimental results on the ARM cluster show that LOCAT accelerates the optimization procedures of the state-of-the-art approaches by at least 4.1x and up to 9.7x; moreover, LOCAT improves the application performance by at least 1.9x and up to 2.4x. On the x86 cluster, LOCAT shows similar results to those on the ARM cluster.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑