OneStopTuner：用于Spark应用程序调优的JVM端到端架构
OneStopTuner: An End to End Architecture for JVM Tuning of Spark Applications
摘要：
Java是广泛使用的大数据框架的支柱，例如Apache Spark，由于其生产力，基于JVM执行的可移植性以及对丰富库集的支持。但是，这些应用程序的性能可能会有很大的差异，这取决于从所有现有JVM标志中选择的运行时标志。手动调优这些标志既麻烦又容易出错。自动调优方法可以简化任务，但当前的解决方案要么需要相当长的处理时间，要么只针对标志的子集，以避免时间和空间需求。在本文中，我们提出了OneStopTuner，一个基于机器学习的新框架自动调整JVM标志。OneStopTuner通过利用批处理模式主动学习来表征用户应用程序，从而控制数据生成量。根据用户选择的优化指标，OneStopTuner然后通过对生成的数据应用特征选择算法来丢弃不相关的JVM标志。最后，它采用样本有效的方法，如贝叶斯优化和回归引导贝叶斯优化的入围JVM标志，以找到所选的标志集的最佳值。我们在广泛使用的Spark基准测试中评估了OneStopTuner，并将其性能与传统的基于模拟退火的自动调整方法进行了比较。我们证明，为了优化执行时间，OneStopTuner选择的标志提供了高达1.35倍的速度比默认的Spark执行，相比之下，1.15倍的速度通过使用模拟退火提出的标志配置。OneStopTuner能够将数据生成的执行次数减少70%，并且能够比基于标准模拟退火的方法快2.4倍地建议最佳标志配置，不包括数据生成时间。
英文摘要：
Java is the backbone of widely used big data frameworks, such as Apache Spark, due to its productivity, portability from JVM-based execution, and support for a rich set of libraries. However, the performance of these applications can widely vary depending on the runtime flags chosen out of all existing JVM flags. Manually tuning these flags is both cumbersome and error-prone. Automated tuning approaches can ease the task, but current solutions either require considerable processing time or target a subset of flags to avoid time and space requirements. In this paper, we present OneStopTuner, a Machine Learning based novel framework for autotuning JVM flags. OneStopTuner controls the amount of data generation by leveraging batch mode active learning to characterize the user application. Based on the user-selected optimization metric, OneStopTuner then discards the irrelevant JVM flags by applying feature selection algorithms on the generated data. Finally, it employs sample efficient methods such as Bayesian optimization and regression guided Bayesian optimization on the shortlisted JVM flags to find the optimal values for the chosen set of flags. We evaluated OneStopTuner on widely used Spark benchmarks and compare its performance with the traditional simulated annealing based autotuning approach. We demonstrate that for optimizing execution time, the flags chosen by OneStopTuner provides a speedup of up to 1.35x over default Spark execution, as compared to 1.15x speedup by using the flag configurations proposed by simulated annealing. OneStopTuner was able to reduce the number of executions for data-generation by 70% and was able to suggest the optimal flag configuration 2.4x faster than the standard simulated annealing based approach, excluding the time for data-generation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
