Rover：一种基于广义迁移学习的在线 Spark SQL 调优服务
Rover: An online Spark SQL tuning service via generalized transfer learning
AI总结：
Rover 提出广义迁移学习框架，结合专家辅助贝叶斯优化与受控历史迁移，实现在线 Spark SQL 高效安全调优，在 12k 真实任务中平均节省 50.1% 内存开销。
摘要：
分布式数据分析引擎（如 Spark）是工业界处理海量数据的常见选择。然而，Spark SQL 的性能高度依赖于配置的选择，而最优配置会随所执行的工作负载而变化。在各种 Spark SQL 调优方案中，贝叶斯优化（BO）是一种流行的框架，在预算充足的情况下能够找到接近最优的配置，但它存在重复优化问题，在实际生产环境中并不实用。在应用迁移学习来加速调优过程时，我们注意到两个领域特定的挑战：1）大多数先前工作侧重于迁移调优历史，而 Spark 工程师的专家知识在提升调优性能方面具有巨大潜力，但迄今为止尚未得到充分研究；2）历史任务应当被谨慎利用，使用不相似的历史任务会导致生产环境中性能恶化。在本文中，我们提出了 Rover，一个已部署的在线 Spark SQL 调优服务，用于在工业工作负载上实现高效且安全的搜索。为应对上述挑战，我们提出了广义迁移学习，基于外部知识来提升调优性能，包括专家辅助贝叶斯优化和受控历史迁移。在公开基准和真实世界任务上的实验表明，Rover 优于竞争性基线方法。值得注意的是，Rover 在 20 次迭代内为 12k 个真实世界 Spark SQL 任务平均节省了 50.1% 的内存开销，其中 76.2% 的任务实现了超过 60% 的显著内存降低。
英文摘要：
Distributed data analytic engines like Spark are common choices to process massive data in industry. However, the performance of Spark SQL highly depends on the choice of configurations, where the optimal ones vary with the executed workloads. Among various alternatives for Spark SQL tuning, Bayesian optimization (BO) is a popular framework that finds near-optimal configurations given sufficient budget, but it suffers from the re-optimization issue and is not practical in real production. When applying transfer learning to accelerate the tuning process, we notice two domain-specific challenges: 1) most previous work focus on transferring tuning history, while expert knowledge from Spark engineers is of great potential to improve the tuning performance but is not well studied so far; 2) history tasks should be carefully utilized, where using dissimilar ones lead to a deteriorated performance in production. In this paper, we present Rover, a deployed online Spark SQL tuning service for efficient and safe search on industrial workloads. To address the challenges, we propose generalized transfer learning to boost the tuning performance based on external knowledge, including expert-assisted Bayesian optimization and controlled history transfer. Experiments on public benchmarks and real-world tasks show the superiority of Rover over competitive baselines. Notably, Rover saves an average of 50.1% of the memory cost on 12k real-world Spark SQL tasks in 20 iterations, among which 76.2% of the tasks achieve a significant memory reduction of over 60%.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
