Spatial Parquet：一种用于地理空间数据湖的列文件格式[扩展版]
Spatial Parquet: A Column File Format for Geospatial Data Lakes [Extended Version]
摘要：
现代数据分析应用程序更喜欢使用列存储格式，因为它们通过编码和压缩提高了存储效率。Parquet是最流行的列数据存储文件格式，它提供了几个开箱即用的好处。然而，Parquet并不支持地理空间数据。本文介绍了Spatial Parquet，一个有效支持地理空间数据的Parquet扩展。Spatial Parquet继承了Parquet对非空间数据的所有优点，例如丰富的数据类型、压缩和列/行过滤。此外，它还增加了三个新功能，以适应地理空间数据。首先，它引入了一种地理空间数据类型，可以将所有标准空间数据类型编码为与Parquet兼容的列格式。其次，它增加了一种新的无损和高效的编码方法，称为FP-delta，该方法可定制为有效地存储以浮点格式存储的地理空间坐标。第三，它添加了一个轻量级的空间索引，允许读者跳过文件的非相关部分，以提高阅读效率。在大规模真实数据上的实验表明，SpatialParquet可以将数据大小减少三分之一，即使没有压缩。压缩可以进一步减小存储大小。此外，当应用轻量级索引时，Spatial Parquet可以将阅读时间减少两个数量级。这个初始原型可以打开新的研究方向，以进一步改善列格式的地理空间数据存储。
英文摘要：
Modern data analytics applications prefer to use column-storage formats due to their improved storage efficiency through encoding and compression. Parquet is the most popular file format for column data storage that provides several of these benefits out of the box. However, geospatial data is not readily supported by Parquet. This paper introduces Spatial Parquet, a Parquet extension that efficiently supports geospatial data. Spatial Parquet inherits all the advantages of Parquet for non-spatial data, such as rich data types, compression, and column/row filtering. Additionally, it adds three new features to accommodate geospatial data. First, it introduces a geospatial data type that can encode all standard spatial data types in a column format compatible with Parquet. Second, it adds a new lossless and efficient encoding method, termed FP-delta, that is customized to efficiently store geospatial coordinates stored in floating-point format. Third, it adds a light-weight spatial index that allows the reader to skip non-relevant parts of the file for increased read efficiency. Experiments on large-scale real data showed that SpatialParquet can reduce the data size by a factor of three, even without compression. Compression can further reduce the storage size. Additionally, Spatial Parquet can reduce the reading time by two orders of magnitude when the light-weight index is applied. This initial prototype can open new research directions to further improve geospatial data storage in column format.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
