arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1303.6609cs.DBcs.DCcs.DS
收藏

在大规模数据分析中利用抽象物理设计

Exploiting Opportunistic Physical Design in Large-scale Data Analytics

Jeff LeFevre, Jagan Sankaranarayanan, Hakan Hacigumus, Junichi Tatemura, Neoklis Polyzotis, Michael J. Carey

摘要:

大规模系统,如MapReduce和Hadoop,执行中间作业结果的积极具体化,以支持容错。当作业对应于数据分析师提交的探索性查询时,这些物化会产生大量的物化视图,这些物化视图通常会捕获来自同一分析师的连续查询之间的公共计算,甚至是测试类似假设的不同分析师的查询之间的公共计算。我们建议将这些视图视为机会主义物理设计,并将其用于查询优化的目的。我们开发了一种新的查询重写算法,解决了在这种情况下的两个主要挑战:如何搜索重写的大空间,以及如何推理包含UDF(大规模数据分析中的一个常见功能)的视图。该算法的灵感来自于非度量空间中的最近邻搜索,它可证明地找到了最小成本重写。我们提出了一个广泛的实验研究现实世界的数据集与原型数据分析系统的基础上蜂巢。结果表明,我们的方法可以在复杂的数据分析查询上产生显着的性能改进,平均减少61%的总执行时间,最多减少两个数量级。

英文摘要:

Large-scale systems, such as MapReduce and Hadoop, perform aggressive materialization of intermediate job results in order to support fault tolerance. When jobs correspond to exploratory queries submitted by data analysts, these materializations yield a large set of materialized views that typically capture common computation among successive queries from the same analyst, or even across queries of different analysts who test similar hypotheses. We propose to treat these views as an opportunistic physical design and use them for the purpose of query optimization. We develop a novel query-rewrite algorithm that addresses the two main challenges in this context: how to search the large space of rewrites, and how to reason about views that contain UDFs (a common feature in large-scale data analytics). The algorithm, which provably finds the minimum-cost rewrite, is inspired by nearest-neighbor searches in non-metric spaces. We present an extensive experimental study on real-world datasets with a prototype data-analytics system based on Hive. The results demonstrate that our approach can result in dramatic performance improvements on complex data-analysis queries, reducing total execution time by an average of 61% and up to two orders of magnitude.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑