arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1702.00358cs.DB
收藏

OLA-RAW:基于Raw数据的可扩展探索

OLA-RAW: Scalable Exploration over Raw Data

Yu Cheng, Weijie Zhao, Florin Rusu

摘要:

在许多领域中,原位处理已经被提出作为一种新颖的数据探索解决方案,其生成大量原始数据,例如,天文学,因为它提供了对原始文件的即时SQL查询。然而,跨查询工作负载的原位处理的性能受到整个数据的完全扫描、标记化和解析的速度的限制。在线聚合(OLA)已被引入作为一种有效的数据探索方法,通过在实际处理过程中不断估计计算结果来更快地识别不感兴趣的模式-计算可以在估计足够准确而被认为不感兴趣时尽早停止。然而,现有的OLA解决方案具有随机混洗和/或采样数据的高前期成本。在本文中,我们提出了OLA-RAW,一个双水平抽样计划的并行在线聚集的原始数据。OLA-RAW中的采样是查询驱动的,并且在运行时查询执行期间仅在原位执行,而不进行数据重组。这是实现了一种新的资源感知的双层采样算法,同时处理随机块中的数据,并自适应地确定块内的采样元组的数量。为了避免从原始数据重复转换的成本,OLA-RAW增量地构建和维护内存驻留的双层示例概要。我们在现代原位数据处理系统中实现了OLA-RAW,并在几种真实和合成数据集和文件格式上评估了其性能。我们的研究结果表明,OLA-RAW选择的抽样计划,最大限度地减少执行时间,并保证在给定的工作负载中的每个查询所需的准确性。最终的结果是一个集中的数据探索过程,避免了不必要的工作,并丢弃了不感兴趣的数据。

英文摘要:

In-situ processing has been proposed as a novel data exploration solution in many domains generating massive amounts of raw data, e.g., astronomy, since it provides immediate SQL querying over raw files. The performance of in-situ processing across a query workload is, however, limited by the speed of full scan, tokenizing, and parsing of the entire data. Online aggregation (OLA) has been introduced as an efficient method for data exploration that identifies uninteresting patterns faster by continuously estimating the result of a computation during the actual processing---the computation can be stopped as early as the estimate is accurate enough to be deemed uninteresting. However, existing OLA solutions have a high upfront cost of randomly shuffling and/or sampling the data. In this paper, we present OLA-RAW, a bi-level sampling scheme for parallel online aggregation over raw data. Sampling in OLA-RAW is query-driven and performed exclusively in-situ during the runtime query execution, without data reorganization. This is realized by a novel resource-aware bi-level sampling algorithm that processes data in random chunks concurrently and determines adaptively the number of sampled tuples inside a chunk. In order to avoid the cost of repetitive conversion from raw data, OLA-RAW builds and maintains a memory-resident bi-level sample synopsis incrementally. We implement OLA-RAW inside a modern in-situ data processing system and evaluate its performance across several real and synthetic datasets and file formats. Our results show that OLA-RAW chooses the sampling plan that minimizes the execution time and guarantees the required accuracy for each query in a given workload. The end result is a focused data exploration process that avoids unnecessary work and discards uninteresting data.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑