arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2109.08053cs.DBcs.DC
收藏

Northlight:SparkSQL大气数据集的声明和优化分析

Northlight: Declarative and Optimized Analysis of Atmospheric Datasets in SparkSQL

Justus Henneberg, Felix Schuhknecht, Philipp Reutter, Nils Brast, Peter Spichtinger

摘要:

执行数据密集型分析是现代地球科学的重要组成部分。因此,大气物理学和气象学的研究经常需要处理非常大的观测和/或建模数据集。通常,这些数据集(a)具有高维度,即包含每个时空点的各种测量,(b)非常大,包含长时间段内的观测。此外,(c)对这些数据集进行的分析任务在结构上很复杂。多年来,二进制格式NetCDF已被确立为地球科学界分发和交换此类多维数据集的事实标准-以及可视化、处理和生成这些数据集的工具和API。不幸的是,这些访问方法通常缺乏(1)易于使用但丰富的查询接口或(2)针对这些数据集的特殊性定制的自动优化管道。因此,地球科学领域的研究人员(通常不是计算机科学家)每天都在不必要地与这些数据集进行有效的工作。因此,在这项工作中,我们的目标是解决上述问题。我们没有提出另一个专门的工具和接口来处理大气数据集,而是将复杂的NetCDF处理功能集成到已建立的SparkSQL引擎中-从而形成了我们的系统Northlight。与同类系统相比,Northlight引入了一组专门针对NetCDF处理量身定制的全自动优化。我们的实验表明,Northlight可以优雅地扩展分析任务的选择性,并且比最先进的管道性能高出6倍。

英文摘要:

Performing data-intensive analytics is an essential part of modern Earth science. As such, research in atmospheric physics and meteorology frequently requires the processing of very large observational and/or modeled datasets. Typically, these datasets (a) have high dimensionality, i.e. contain various measurements per spatiotemporal point, (b) are extremely large, containing observations over a long time period. Additionally, (c) the analytical tasks being performed on these datasets are structurally complex. Over the years, the binary format NetCDF has been established as a de-facto standard in distributing and exchanging such multi-dimensional datasets in the Earth science community -- along with tools and APIs to visualize, process, and generate them. Unfortunately, these access methods typically lack either (1) an easy-to-use but rich query interface or (2) an automatic optimization pipeline tailored towards the specialities of these datasets. As such, researchers from the field of Earth sciences (which are typically not computer scientists) unnecessarily struggle in efficiently working with these datasets on a daily basis. Consequently, in this work, we aim at resolving the aforementioned issues. Instead of proposing yet another specialized tool and interface to work with atmospheric datasets, we integrate sophisticated NetCDF processing capabilities into the established SparkSQL dataflow engine -- resulting in our system Northlight. In contrast to comparable systems, Northlight introduces a set of fully automatic optimizations specifically tailored towards NetCDF processing. We experimentally show that Northlight scales gracefully with the selectivity of the analysis tasks and outperforms the comparable state-of-the-art pipeline by up to a factor of 6x.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑