利用节点内请求聚合提高MPI集体I/O性能
Improving MPI Collective I/O Performance With Intra-node Request Aggregation
摘要:
两阶段I/O是实现集体MPI-IO功能的众所周知的策略。它将调用进程之间的I/O请求重新分配为一种最小化文件访问成本的形式。随着现代并行计算机继续发展到亿亿次时代,这种请求重新分配的通信成本可能会迅速压倒集体I/O性能。从在多个计算节点上运行的并行作业中可以观察到这种效果,每个节点上有大量MPI进程。为了降低通信成本,我们提出了一种新的设计,集体I/O通过增加一个额外的通信层,在同一计算节点内的进程之间执行请求聚合。这种方法可以显着减少重新分配I/O请求时节点间的通信拥塞。我们评估的性能,并与原来的两阶段I/O上的Cray XC 40并行计算机与英特尔KNL处理器进行比较。使用I/O模式从两个大规模的生产应用程序和I/O基准,我们显示了性能提高了29倍,当运行16384 MPI进程在256个计算节点。
英文摘要:
Two-phase I/O is a well-known strategy for implementing collective MPI-IO functions. It redistributes I/O requests among the calling processes into a form that minimizes the file access costs. As modern parallel computers continue to grow into the exascale era, the communication cost of such request redistribution can quickly overwhelm collective I/O performance. This effect has been observed from parallel jobs that run on multiple compute nodes with a high count of MPI processes on each node. To reduce the communication cost, we present a new design for collective I/O by adding an extra communication layer that performs request aggregation among processes within the same compute nodes. This approach can significantly reduce inter-node communication congestion when redistributing the I/O requests. We evaluate the performance and compare with the original two-phase I/O on a Cray XC40 parallel computer with Intel KNL processors. Using I/O patterns from two large-scale production applications and an I/O benchmark, we show the performance improvement of up to 29 times when running 16384 MPI processes on 256 compute nodes.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。