HDF5的用户定义函数
User-Defined Functions for HDF5
摘要:
科学数据集以其具有挑战性的存储需求和转换其信息的相关处理管道而闻名。其中一些处理任务包括过滤、清理、聚合、规范化和数据格式转换--所有这些都会生成更多的数据。在本文中,我们提出了一种HDF 5文件格式的基础设施,使数据集值能够动态填充:任务相关的脚本可以附加到HDF 5文件中,并且仅在应用程序读取数据集时执行。我们提供了支持用户定义函数(UDF)的软件架构的详细信息,以及它如何与硬件加速器和计算存储集成。此外,我们描述了内置的安全模型,限制系统资源的UDF可以访问。最后,我们提出了几个用例,展示了如何使用UDF以超出本工作原始范围的方式扩展科学数据集。
英文摘要:
Scientific datasets are known for their challenging storage demands and the associated processing pipelines that transform their information. Some of those processing tasks include filtering, cleansing, aggregation, normalization, and data format translation -- all of which generate even more data. In this paper, we present an infrastructure for the HDF5 file format that enables dataset values to be populated on the fly: task-related scripts can be attached into HDF5 files and only execute when the dataset is read by an application. We provide details on the software architecture that supports user-defined functions (UDFs) and how it integrates with hardware accelerators and computational storage. Moreover, we describe the built-in security model that limits the system resources a UDF can access. Last, we present several use cases that show how UDFs can be used to extend scientific datasets in ways that go beyond the original scope of this work.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。