多资源并行查询调度与优化
Multi-Resource Parallel Query Scheduling and Optimization
摘要:
调度查询执行计划在无共享并行系统中是特别复杂的问题,其中每个站点由本地时间共享(例如,CPU或磁盘)和空间共享(例如,存储器)资源,并通过消息传递与远程站点通信。并行查询调度的早期工作采用(a)并行任务调度的一维模型,有效地忽略了资源共享的潜在好处,或(b)全局可访问的资源单元的模型,这是只适用于共享内存架构,因为他们不能捕捉系统资源的亲和力网站。在本文中,我们开发了一种通用的方法,捕捉调度分布式,多维资源单位的所有形式的并行内部和跨查询和操作符的全部复杂性。我们提出了一个基于层次的列表调度启发式算法的独立查询任务(即,物理操作符流水线),其对于给定程度的分区并行性(具有取决于每个站点的时间共享和空间共享资源的数量以及克隆的粒度的最坏情况性能比)而言可证明是接近最优的。我们还提出了扩展来处理逻辑运算符中的阻塞约束(例如,散列连接)流水线和丛式查询计划以及在线任务到达(例如,在动态或多查询执行环境中)。我们的调度算法上实现的一个详细的仿真模型的实验验证了其有效性相比,现有的方法在现实的设置。基于我们的分析和实验结果,我们重新审视了开放的问题,设计高效的并行查询优化的成本模型,并提出了一个解决方案,捕获所有重要参数的并行执行。
英文摘要:
Scheduling query execution plans is a particularly complex problem in shared-nothing parallel systems, where each site consists of a collection of local time-shared (e.g., CPU(s) or disk(s)) and space-shared (e.g., memory) resources and communicates with remote sites by message-passing. Earlier work on parallel query scheduling employs either (a) one-dimensional models of parallel task scheduling, effectively ignoring the potential benefits of resource sharing, or (b) models of globally accessible resource units, which are appropriate only for shared-memory architectures, since they cannot capture the affinity of system resources to sites. In this paper, we develop a general approach capturing the full complexity of scheduling distributed, multi-dimensional resource units for all forms of parallelism within and across queries and operators. We present a level-based list scheduling heuristic algorithm for independent query tasks (i.e., physical operator pipelines) that is provably near-optimal for given degrees of partitioned parallelism (with a worst-case performance ratio that depends on the number of time-shared and space-shared resources per site and the granularity of the clones). We also propose extensions to handle blocking constraints in logical operator (e.g., hash-join) pipelines and bushy query plans as well as on-line task arrivals (e.g., in a dynamic or multi-query execution environment). Experiments with our scheduling algorithms implemented on top of a detailed simulation model verify their effectiveness compared to existing approaches in a realistic setting. Based on our analytical and experimental results, we revisit the open problem of designing efficient cost models for parallel query optimization and propose a solution that captures all the important parameters of parallel execution.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。