一种尺寸不能容纳所有:一种自适应的无共享关系数据库中的斜散列连接调度器
One Size Cannot Fit All: a Self-Adaptive Dispatcher for Skewed Hash Join in Shared-nothing RDBMSs
摘要:
无共享体系结构在各种商用分布式关系数据库管理系统中被广泛采用。由于该架构,查询可以并行处理,并通过按需水平扩展集群来加速。尽管如此,负载均衡在所有分布式RDBMS中一直是一个具有挑战性的问题,包括无共享的RDBMS,它受到数据分布偏斜的影响。在这项工作中,我们专注于一个代表性的操作,即哈希连接,并研究如何偏度之间的集群节点将影响负载平衡和最终的效率,在共享的RDBMS中的任意查询。我们发现,现有的分布式散列连接(Dist-HJ)解决方案可能无法提供令人满意的性能时,值是在探测和构建表中倾斜。为了解决这个问题,我们提出了一种新的Dist-HJ解决方案,即分区和复制(PnR)。虽然PnR在某些偏斜情况下提供了最好的效率,但我们对一组无共享RDBMS的详尽实验表明,没有一个Dist-HJ解决方案在所有(数据偏斜)情况下都能获胜。为此,我们进一步提出了一个自适应Dist-HJ解决方案,内置子运营商成本模型,动态选择最佳Dist-HJ实现策略在运行时根据目标查询的数据倾斜。我们实现的解决方案在我们的商业无共享的关系数据库,即开午数据库(原名ZNBase)和实证研究证明,自适应模型实现了最佳的性能相比,在许多现有的关系数据库中采用的一系列解决方案。
英文摘要:
Shared-nothing architecture has been widely adopted in various commercial distributed RDBMSs. Thanks to the architecture, query can be processed in parallel and accelerated by scaling up the cluster horizontally on demand. In spite of that, load balancing has been a challenging issue in all distributed RDBMSs, including shared-nothing ones, which suffers much from skewed data distribution. In this work, we focus on one of the representative operator, namely Hash Join, and investigate how skewness among the nodes of a cluster will affect the load balance and eventual efficiency of an arbitrary query in shared-nothing RDBMSs. We found that existing Distributed Hash Join (Dist-HJ) solutions may not provide satisfactory performance when a value is skewed in both the probe and build tables. To address that, we propose a novel Dist-HJ solution, namely Partition and Replication (PnR). Although PnR provide the best efficiency in some skewness scenario, our exhaustive experiments over a group of shared-nothing RDBMSs show that there is not a single Dist-HJ solution that wins in all (data skew) scenarios. To this end, we further propose a self-adaptive Dist-HJ solution with a builtin sub-operator cost model that dynamically select the best Dist-HJ implementation strategy at runtime according to the data skew of the target query. We implement the solution in our commercial shared-nothing RDBMSs, namely KaiwuDB (former name ZNBase) and empirical study justifies that the self-adaptive model achieves the best performance comparing to a series of solution adopted in many existing RDBMSs.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。