ArrayBridge：将声明性数组处理与高性能计算交织在一起
ArrayBridge: Interweaving declarative array processing with high-performance computing
摘要：
科学家们越来越多地转向小型计算机来生产和分析大规模阵列。尽管几十年的数据库研究都在颂扬声明式查询处理的优点，但科学家们仍然编写、调试和并行化命令式HPC内核，即使是最普通的查询。这种阻抗不匹配的部分原因是繁琐的数据加载过程;作为回应，数据库界提出了以科学文件格式访问数据的就地机制。然而，科学家们想要的不仅仅是从文件中读取数组的被动访问方法。本文介绍了ArrayBridge，一个双向数组视图机制的科学文件格式，其目的是使声明性数组操作与命令式文件为中心的分析互操作。我们的ArrayBridge原型实现使用HDF5作为底层阵列存储库，并无缝集成到SciDB开源阵列数据库系统中。除了快速查询外部数组对象外，ArrayBridge还可以像读取数组一样轻松地生成HDF5文件格式的数组。ArrayBridge还支持通过未修改的HDF5 API从命令式内核进行时间旅行查询，并自动消除数组版本之间的重复数据，以提高空间效率。我们在NERSC（一个大规模科学计算设施）中进行的广泛的性能评估表明，ArrayBridge表现出与原生SciDB存储引擎在统计上不可区分的性能和I/O可扩展性。
英文摘要：
Scientists are increasingly turning to datacenter-scale computers to produce and analyze massive arrays. Despite decades of database research that extols the virtues of declarative query processing, scientists still write, debug and parallelize imperative HPC kernels even for the most mundane queries. This impedance mismatch has been partly attributed to the cumbersome data loading process; in response, the database community has proposed in situ mechanisms to access data in scientific file formats. Scientists, however, desire more than a passive access method that reads arrays from files. This paper describes ArrayBridge, a bi-directional array view mechanism for scientific file formats, that aims to make declarative array manipulations interoperable with imperative file-centric analyses. Our prototype implementation of ArrayBridge uses HDF5 as the underlying array storage library and seamlessly integrates into the SciDB open-source array database system. In addition to fast querying over external array objects, ArrayBridge produces arrays in the HDF5 file format just as easily as it can read from it. ArrayBridge also supports time travel queries from imperative kernels through the unmodified HDF5 API, and automatically deduplicates between array versions for space efficiency. Our extensive performance evaluation in NERSC, a large-scale scientific computing facility, shows that ArrayBridge exhibits statistically indistinguishable performance and I/O scalability to the native SciDB storage engine.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
