优化MPI-IO中的非连续访问
Optimizing Noncontiguous Accesses in MPI-IO
摘要:
许多并行应用程序的I/O访问模式由对大量不连续的小数据块的访问组成。但是,如果应用程序的I/O需求是通过发出许多小的、不同的I/O请求来满足的,则I/O性能会急剧下降。为了避免这个问题,MPI-IO允许用户通过单个I/O函数调用访问非连续数据,这与Unix I/O不同。在本文中,我们将解释MPI-IO的这一特性对于高性能的重要性,以及它如何使实现能够执行优化。我们首先对在MPI-IO中表达应用程序的I/O需求的不同方式进行了分类--我们将它们分为四级,称为level~0到level~3。我们证明,对于非连续访问模式的应用程序,如果用户编写他们的应用程序,使3级请求(非连续,集体),而不是0级请求(Unix风格)的I/O性能显着提高。然后,我们描述了我们的MPI-IO实现,ROMIO,提供高性能的非连续请求。我们详细解释了ROMIO执行的两个关键优化:一个进程的非连续请求的数据筛选和多个进程的非连续请求的集体I/O。我们描述了如何在多台机器和文件系统上可移植地实现这些优化,控制它们的内存需求,并实现高性能。我们展示了三个应用程序的性能和可移植性的性能结果-天体物理应用程序模板(DIST 3D),NAS BTIO基准测试,和非结构化代码(UNESCO)-在五个不同的并行机:惠普Exemplar,IBM SP,英特尔Paragon,NEC SX-4,SGI Origin 2000。
英文摘要:
The I/O access patterns of many parallel applications consist of accesses to a large number of small, noncontiguous pieces of data. If an application's I/O needs are met by making many small, distinct I/O requests, however, the I/O performance degrades drastically. To avoid this problem, MPI-IO allows users to access noncontiguous data with a single I/O function call, unlike in Unix I/O. In this paper, we explain how critical this feature of MPI-IO is for high performance and how it enables implementations to perform optimizations. We first provide a classification of the different ways of expressing an application's I/O needs in MPI-IO--we classify them into four levels, called level~0 through level~3. We demonstrate that, for applications with noncontiguous access patterns, the I/O performance improves dramatically if users write their applications to make level-3 requests (noncontiguous, collective) rather than level-0 requests (Unix style). We then describe how our MPI-IO implementation, ROMIO, delivers high performance for noncontiguous requests. We explain in detail the two key optimizations ROMIO performs: data sieving for noncontiguous requests from one process and collective I/O for noncontiguous requests from multiple processes. We describe how we have implemented these optimizations portably on multiple machines and file systems, controlled their memory requirements, and also achieved high performance. We demonstrate the performance and portability with performance results for three applications--an astrophysics-application template (DIST3D), the NAS BTIO benchmark, and an unstructured code (UNSTRUC)--on five different parallel machines: HP Exemplar, IBM SP, Intel Paragon, NEC SX-4, and SGI Origin2000.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。