arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1910.09650cs.DC
收藏

对Allreduce的节点感知改进

Node-Aware Improvements to Allreduce

Amanda Bienz, Luke N. Olson, William D. Gropp

摘要:

\texttt{MPI\_Allreduce}集合操作是许多并行代码库的核心内核,特别是对于每个进程的单个值的约简。常用的allreduce递归加倍算法获得了消息计数的下限,基于节点不可知的性能模型,对小的缩减大小产生了最优性。然而,该算法在节点集之间产生重复的消息。MPICH中的节点感知优化通过每个节点使用单个主进程来移除重复消息,从而在每个节点间步骤产生大量非活动进程。在本文中,我们提出了一种算法,使用每个节点的多个进程,以减少由一个单一的进程,提高性能的allreduce操作,特别是对于小的消息大小的节点间的消息通信的最大数量。

英文摘要:

The \texttt{MPI\_Allreduce} collective operation is a core kernel of many parallel codebases, particularly for reductions over a single value per process. The commonly used allreduce recursive-doubling algorithm obtains the lower bound message count, yielding optimality for small reduction sizes based on node-agnostic performance models. However, this algorithm yields duplicate messages between sets of nodes. Node-aware optimizations in MPICH remove duplicate messages through use of a single master process per node, yielding a large number of inactive processes at each inter-node step. In this paper, we present an algorithm that uses the multiple processes available per node to reduce the maximum number of inter-node messages communicated by a single process, improving the performance of allreduce operations, particularly for small message sizes.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑