可扩展的数据跳过
Extensible Data Skipping
摘要:
数据跳过通过基于元数据跳过不相关的数据对象(文件)来减少SQL查询的I/O。我们扩展了这个概念,允许开发人员使用灵活的API定义自己的数据,跳过元数据类型和索引。我们的框架是第一个原生支持任意数据类型(例如地理空间,日志)和用户定义函数(UDF)查询的数据跳过。我们将我们的框架与Apache Spark集成在一起,它现在部署在IBM的多个产品/服务中。我们展示了可扩展的数据跳过API,讨论了索引设计,并实现了各种元数据索引,每个索引只需要大约30行额外的代码。特别是,我们实现了数据跳过第三方库与地理空间的UDF和演示两个数量级的加速。我们的集中式元数据方法提供了一个x3.6的速度相比,即使是重写利用Parquet最小/最大元数据的查询。我们证明了可扩展的数据跳过是适用于广泛的应用程序,用户定义的索引实现显着的加速和成本节约非常低的开发成本。
英文摘要:
Data skipping reduces I/O for SQL queries by skipping over irrelevant data objects (files) based on their metadata. We extend this notion by allowing developers to define their own data skipping metadata types and indexes using a flexible API. Our framework is the first to natively support data skipping for arbitrary data types (e.g. geospatial, logs) and queries with User Defined Functions (UDFs). We integrated our framework with Apache Spark and it is now deployed across multiple products/services at IBM. We present our extensible data skipping APIs, discuss index design, and implement various metadata indexes, requiring only around 30 lines of additional code per index. In particular we implement data skipping for a third party library with geospatial UDFs and demonstrate speedups of two orders of magnitude. Our centralized metadata approach provides a x3.6 speed up even when compared to queries which are rewritten to exploit Parquet min/max metadata. We demonstrate that extensible data skipping is applicable to broad class of applications, where user defined indexes achieve significant speedups and cost savings with very low development cost.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。