HPC集群网络和I/O通信的跨层可视化和分析
Cross-layer Visualization and Profiling of Network and I/O Communication for HPC Clusters
摘要:
理解和可视化HPC应用程序、MPI库、通信结构和文件系统之间的全栈性能权衡和相互作用是一项具有挑战性的工作。为HPC通信网络设计整体分析和可视化方法具有挑战性,因为不同级别的通信在通信结构上共存并相互交互。流量的分解对于理解不同层之间的相互作用以及应用程序的通信行为而不丢失网络流量的总体视图是必不可少的。不幸的是,现有的分析工具是不相交的,要么只关注分析和可视化HPC堆栈的几个级别,这限制了它们可以提供的见解,要么它们提供非常详细的信息,需要陡峭的学习曲线才能理解。我们的分析工具可视化旨在为HPC通信堆栈提供全面和实时的见解。在本文中,我们提出并实现了我们的可视化方法,使整体的洞察力,代表跨堆栈的指标。此外,我们提出并实现了一个低开销的I/O分析内的通信库,收集和存储的分析信息,然后研究的相关性和评估的I/O流量与MPI通信使用跨堆栈的方法由INAM。通过实验评估和用例,我们展示了实时跨栈通信分析的新优势,以检测瓶颈并了解通信性能。
英文摘要:
Understanding and visualizing the full-stack performance trade-offs and interplay between HPC applications, MPI libraries, the communication fabric, and the file system is a challenging endeavor. Designing a holistic profiling and visualization method for HPC communication networks is challenging since different levels of communication coexist and interact with each other on the communication fabric. A breakdown of traffic is essential to understand the interplay of different layers along with the application's communication behavior without losing a general view of network traffic. Unfortunately, existing profiling tools are disjoint and either focus on only profiling and visualizing a few levels of the HPC stack, which limits the insights they can provide, or they provide extremely detailed information which necessitates a steep learning curve to understand. We target our profiling tool visualization to provide holistic and real-time insights into HPC communication stacks. In this paper, we propose and implement our visualization methods to enable holistic insight for representing the cross-stack metrics. Moreover, we propose and implement a low-overhead I/O profiling inside the communication library, collect and store the profiling information, and then study the correlation and evaluation of I/O traffic with MPI communication using a cross-stack approach by INAM. Through experimental evaluations and use cases, we demonstrate novel benefits of our cross-stack communication analysis in real-time to detect bottlenecks and understand communication performance.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。