arXivDaily arXiv每日学术速递 周一至周五更新
arXiv cs/0306048cs.DC
收藏

并行netCDF:一种科学的高性能I/O接口

Parallel netCDF: A Scientific High-Performance I/O Interface

Jianwei Li, Wei-keng Liao, Alok Choudhary, Robert Ross, Rajeev Thakur, William Gropp, Rob Latham

摘要:

数据集存储、交换和访问在科学应用中起着至关重要的作用。为此目的,netCDF作为一个可移植的和有效的文件格式和编程接口,这是在许多科学应用领域流行。然而,原始的接口并没有提供一个有效的机制,并行数据存储和访问。在这项工作中,我们提出了一个新的并行接口写入和读取netCDF数据集。此接口是从串行netCDF接口派生而来的,更改最少,但定义了并行访问的语义,并为高性能而定制。底层的并行I/O是通过MPI-IO实现的,通过使用集体I/O优化可以显著提高性能。我们比较了HDF 5的实现策略,并对两者进行了分析。我们的测试表明,编程的便利性和显着的I/O性能的改善与此并行netCDF接口。

英文摘要:

Dataset storage, exchange, and access play a critical role in scientific applications. For such purposes netCDF serves as a portable and efficient file format and programming interface, which is popular in numerous scientific application domains. However, the original interface does not provide an efficient mechanism for parallel data storage and access. In this work, we present a new parallel interface for writing and reading netCDF datasets. This interface is derived with minimum changes from the serial netCDF interface but defines semantics for parallel access and is tailored for high performance. The underlying parallel I/O is achieved through MPI-IO, allowing for dramatic performance gains through the use of collective I/O optimizations. We compare the implementation strategies with HDF5 and analyze both. Our tests indicate programming convenience and significant I/O performance improvement with this parallel netCDF interface.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑