面向自适应、细粒度参数调优的Spark优化器
A Spark Optimizer for Adaptive, Fine-Grained Parameter Tuning
AI总结:
本文提出一种Spark优化器,采用混合编译时/运行时方法在AQE架构下进行多粒度参数调优,通过多目标优化在1-2秒内适应用户性价比偏好,在TPC-H/TPC-DS上显著降低延迟并优于现有方法。
摘要:
随着Spark成为常见的大数据分析平台,其日益增长的复杂性使得众多参数的自动调优对性能至关重要。我们对Spark参数调优的工作特别受到两个近期趋势的推动:基于运行时统计的Spark自适应查询执行(AQE),以及日益普及的Spark云部署(这使得性价比推理对终端用户至关重要)。本文提出了我们的Spark优化器设计,该优化器控制新AQE架构中每个查询的所有可调参数以探索其性能优势,同时将调优问题置于理论完备的多目标优化(MOO)设置中,以更好地适应用户的性价比偏好。为此,我们提出了一种新颖的混合编译时/运行时方法,用于对多样且相关的Spark参数进行多粒度调优,以及一套建模和优化技术,以在满足云使用1-2秒的严格时间约束下,解决MOO设置中的调优问题。使用TPC-H和TPC-DS基准的评估结果证明了我们方法的卓越性能:(i)当优先考虑延迟时,在0.7-0.8秒的平均求解时间下,它分别实现了TPC-H和TPC-DS 63%和65%的降低,优于仅减少18-25%延迟且求解时间为2.6-15秒的最具竞争力的MOO方法。(ii)当在延迟和成本之间转移偏好时,我们的方法主导了替代方法的解决方案,展现出对变化偏好的卓越适应性。
英文摘要:
As Spark becomes a common big data analytics platform, its growing complexity makes automatic tuning of numerous parameters critical for performance. Our work on Spark parameter tuning is particularly motivated by two recent trends: Spark's Adaptive Query Execution (AQE) based on runtime statistics, and the increasingly popular Spark cloud deployments that make cost-performance reasoning crucial for the end user. This paper presents our design of a Spark optimizer that controls all tunable parameters of each query in the new AQE architecture to explore its performance benefits and, at the same time, casts the tuning problem in the theoretically sound multi-objective optimization (MOO) setting to better adapt to user cost-performance preferences. To this end, we propose a novel hybrid compile-time/runtime approach to multi-granularity tuning of diverse, correlated Spark parameters, as well as a suite of modeling and optimization techniques to solve the tuning problem in the MOO setting while meeting the stringent time constraint of 1-2 seconds for cloud use. Evaluation results using TPC-H and TPC-DS benchmarks demonstrate the superior performance of our approach: (i) When prioritizing latency, it achieves 63% and 65% reduction for TPC-H and TPC-DS, respectively, under an average solving time of 0.7-0.8 sec, outperforming the most competitive MOO method that reduces only 18-25% latency with 2.6-15 sec solving time. (ii) When shifting preferences between latency and cost, our approach dominates the solutions of alternative methods, exhibiting superior adaptability to varying preferences.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。