预测云中Lakehouse性能：查询运行时方差的实证探索
Predicting Lakehouse Performance in Clouds: An Empirical Exploration of Query Runtime Variance
AI总结：
本文通过跨云实验量化分布式Lakehouse查询的运行时方差，分析其来源（如数据局部性、共租户负载和缓存效应），并评估方差对查询性能预测模型及低碳调度等负载管理技术的影响。
摘要：
数据分析越来越多地运行在分布式lakehouse系统上，平台运营商必须优化货币、资源和环境成本。查询性能预测（QPP）有助于平衡这些成本，并支持自适应资源缩放和低碳调度等工作负载管理技术。然而，lakehouse中的运行时可能存在显著差异，且运行时方差对QPP准确性和工作负载编排的影响此前尚未针对lakehouse系统进行系统研究。本文通过研究分布式lakehouse分析查询观察到的运行时方差及其对QPP的影响来填补这一空白。首先，我们使用Kubernetes部署在三个公共云和一个私有云上，跨多个数据库规模和三个分析基准，量化了运行间方差。结果表明，同一查询的重复执行在运行时上可能相差近两倍。其次，我们进行了因子分析研究，评估了运行时方差的关键来源，如数据局部性、共租户负载和缓存效应。第三，我们研究了方差如何影响最先进的QPP模型，揭示解决方差的关键来源可以将预测误差降低高达80%。最后，我们展示了作为依赖性能预测的工作负载管理技术示例的低碳调度的下游影响，表明考虑运行时方差可以显著降低碳成本。
英文摘要：
Data analytics increasingly runs on distributed lakehouse systems, where platform operators must optimise monetary, resource, and environmental costs. Query Performance Prediction (QPP) helps to balance these costs and supports workload management techniques, such as adaptive resource scaling and low-carbon scheduling. However, runtimes in lakehouses can vary substantially, and the impact of runtime variance on QPP accuracy and workload orchestration has not previously been systematically studied for lakehouse systems. This paper addresses this gap by investigating the runtime variance observed for distributed lakehouse analytical queries and its impact on QPP. First, we quantify the run-to-run variance using Kubernetes deployments across three public clouds and one private cloud, spanning multiple database scales and three analytical benchmarks. Our results demonstrate that repeated executions of the same query can vary in runtime by nearly twofold. Second, we conduct a factor analysis study assessing key sources of this runtime variance such as data locality, co-tenant load, and caching effects. Third, we examine how variance influences state-of-the-art QPP models, revealing that addressing key sources of variance can reduce prediction error up to 80%. Finally, we demonstrate the downstream implications for low-carbon scheduling as an example of a workload management technique that relies on performance prediction, showing that accounting for runtime variance can lead to a significant reduction in carbon costs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
