arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1208.0087cs.DB
收藏

打开数据流优化中的黑箱

Opening the Black Boxes in Data Flow Optimization

Fabian Hueske, Mathias Peters, Matthias Sax, Astrid Rheinländer, Rico Bergmann, Aljoscha Krettek, Kostas Tzoumas

摘要:

用于大数据分析的许多系统采用数据流抽象来定义并行数据处理任务。在此设置中,表示为用户定义函数的自定义操作非常常见。我们解决的问题,执行数据流优化,在这个抽象层次上,操作符的语义是未知的。传统上,查询优化应用于具有已知代数语义的查询。在这项工作中,我们发现,少数的属性,而不是一个完整的代数规范,足以建立数据处理运营商的重新排序条件。我们表明,这些属性可以准确地估计黑盒运营商通过静态分析其用户定义的功能的通用代码。我们设计并实现了一个并行数据流的优化器,不承担知识的语义或代数性质的运营商。我们的评估证实,优化器可以应用常见的重写,如选择重新排序,浓密的连接顺序枚举,和有限形式的聚合下推,从而产生类似的重写能力,现代关系DBMS优化器。此外,它可以优化非关系数据流的操作符顺序,这是当今系统中的一个独特功能。

英文摘要:

Many systems for big data analytics employ a data flow abstraction to define parallel data processing tasks. In this setting, custom operations expressed as user-defined functions are very common. We address the problem of performing data flow optimization at this level of abstraction, where the semantics of operators are not known. Traditionally, query optimization is applied to queries with known algebraic semantics. In this work, we find that a handful of properties, rather than a full algebraic specification, suffice to establish reordering conditions for data processing operators. We show that these properties can be accurately estimated for black box operators by statically analyzing the general-purpose code of their user-defined functions. We design and implement an optimizer for parallel data flows that does not assume knowledge of semantics or algebraic properties of operators. Our evaluation confirms that the optimizer can apply common rewritings such as selection reordering, bushy join-order enumeration, and limited forms of aggregation push-down, hence yielding similar rewriting power as modern relational DBMS optimizers. Moreover, it can optimize the operator order of non-relational data flows, a unique feature among today's systems.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑