基于文件通信的大规模并行化
Large Scale Parallelization Using File-Based Communications
摘要：
在本文中，我们提出了一种新的和新的基于文件的通信架构，使用本地文件系统的大规模并行化。这种新方法消除了在使用中央文件系统处理大型并行作业时文件系统过载和资源争用的问题。当发送和接收进程不在同一个节点上时，由于节点间的消息文件传输，新方法会产生额外的开销。然而，即使有这种额外的开销成本，除了在大规模并行作业的消息通信中的性能增强之外，它对整个集群操作的好处要大得多。例如，当运行一个2048进程的并行作业时，使用本地文件系统时，使用MPI_Bcast（）可以获得大约34倍的性能。此外，由于传输消息文件的安全性完全通过使用安全复制协议（scp）和文件系统许可来处理，因此除了HPC系统上通常需要的那些安全措施或端口之外，不需要额外的安全措施或端口。
英文摘要：
In this paper, we present a novel and new file-based communication architecture using the local filesystem for large scale parallelization. This new approach eliminates the issues with filesystem overload and resource contention when using the central filesystem for large parallel jobs. The new approach incurs additional overhead due to inter-node message file transfers when both the sending and receiving processes are not on the same node. However, even with this additional overhead cost, its benefits are far greater for the overall cluster operation in addition to the performance enhancement in message communications for large scale parallel jobs. For example, when running a 2048-process parallel job, it achieved about 34 times better performance with MPI_Bcast() when using the local filesystem. Furthermore, since the security for transferring message files is handled entirely by using the secure copy protocol (scp) and the file system permissions, no additional security measures or ports are required other than those that are typically required on an HPC system.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
