在阿里云MaxCompute中学习查询优化器:挑战、分析与解决方案
Learned Query Optimizer in Alibaba MaxCompute: Challenges, Analysis, and Solutions
AI总结:
LOAM是为MaxCompute设计的学习查询优化框架,通过无需统计数据的计划编码和领域适应技术,实现高达30%的CPU成本节省。
摘要:
现有的学习查询优化器仍然无法适应现代分布式、多租户数据仓库,原因在于理想化的建模假设和设计选择。以阿里云的MaxCompute为例,我们揭示了任何可部署的学习查询优化器面临的四个根本性、系统无关的挑战:1)高度动态的执行环境导致计划成本存在较大波动;2)潜在缺乏用于成本估算所需的数据统计信息;3)传统模型细化不可行;4)不同工作负载下的收益不确定。这些挑战暴露了理论进展与生产现实之间的深层不匹配,并要求对学习优化器进行系统优先的重新设计。为弥合这一差距,我们提出了LOAM,一个专为MaxCompute设计的学习查询优化框架。其设计原则和技术可推广并适用于类似系统。在架构上,LOAM引入了无需统计数据的计划编码,利用操作符语义和历史执行来推断数据分布的细节,并显式编码训练查询的执行环境以学习其对计划成本的影响。对于在线查询,在预测时间环境未知时,LOAM提供了一个可实现性能的理论界和一种实用策略来平滑环境对成本估算的影响。在系统运行方面,LOAM将领域适应技术整合到训练中,以有效泛化到在线查询计划而无需传统细化。此外,LOAM还包括一个轻量级的项目选择器以优先部署高收益的项目。LOAM在生产工作负载上实现了高达30%的CPU成本节省,这可能转化为实际的资源节省。
英文摘要:
Existing learned query optimizers remain ill-suited to modern distributed, multi-tenant data warehouses due to idealized modeling assumptions and design choices. Using Alibaba's MaxCompute as a representative, we surface four fundamental, system-agnostic challenges for any deployable learned query optimizer: 1) highly dynamic execution environments that induce large variance in plan costs; 2) potential absence of input statistics needed for cost estimation; 3) infeasibility of conventional model refinement; and 4) uncertain benefits across different workloads. These challenges expose a deep mismatch between theoretical advances and production realities and demand a principled, deployment-first redesign of learned optimizers. To bridge this gap, we present LOAM, a one-stop learned query optimization framework for MaxCompute. Its design principles and techniques generalize and are readily adaptable to similar systems. Architecturally, LOAM introduces a statistics-free plan encoding that leverages operator semantics and historical executions to infer details about data distributions and explicitly encodes the execution environments of training queries to learn their impacts on plan costs. For online queries with unknown environments at prediction time, LOAM provides a theoretical bound on the achievable performance and a practical strategy to smooth the environmental impacts on cost estimations. For system operating, LOAM integrates domain adaptation techniques into training to generalize effectively to online query plans without requiring conventional refinement. Additionally, LOAM includes a lightweight project selector to prioritize high-benefit deployment projects. LOAM has seen up to 30% CPU cost savings over MaxCompute's native query optimizer on production workloads, which could translate to substantial real-world resource savings.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。