arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1708.08319cs.PLcs.DBcs.IR
收藏

通过代码转换快速访问分层嵌套的列数据

Fast Access to Columnar, Hierarchically Nested Data via Code Transformation

Jim Pivarski, Peter Elmer, Brian Bockelman, Zhe Zhang

摘要:

大数据查询系统以列式格式表示数据,以实现快速、有选择的访问,在某些情况下(例如Apache Drill),直接对列式数据执行计算,而无需行物化,从而避免运行时成本。然而,许多分析过程不能容易地或有效地表示为SQL。在高能物理中,大多数数据处理需要具有复杂依赖关系的嵌套循环。当面对这样的任务时,传统的方法是将列式数据转换回对象形式,通常需要付出一定的性能代价。本文描述了一种新的技术来转换过程代码,使其操作层次嵌套,列数据本机,没有行物化。它可以被看作是编译器在类型化抽象语法树上的传递,将对对象的引用重写为列数组查找。我们还将介绍转换后的代码和传统的面向对象的代码在高能物理上下文中的性能比较。

英文摘要:

Big Data query systems represent data in a columnar format for fast, selective access, and in some cases (e.g. Apache Drill), perform calculations directly on the columnar data without row materialization, avoiding runtime costs. However, many analysis procedures cannot be easily or efficiently expressed as SQL. In High Energy Physics, the majority of data processing requires nested loops with complex dependencies. When faced with tasks like these, the conventional approach is to convert the columnar data back into an object form, usually with a performance price. This paper describes a new technique to transform procedural code so that it operates on hierarchically nested, columnar data natively, without row materialization. It can be viewed as a compiler pass on the typed abstract syntax tree, rewriting references to objects as columnar array lookups. We will also present performance comparisons between transformed code and conventional object-oriented code in a High Energy Physics context.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑