松耦合Petascale规划集体IO模型的设计与评价
Design and Evaluation of a Collective IO Model for Loosely Coupled Petascale Programming
摘要：
松耦合编程是一种强大的范例，用于在千万亿次系统上从科学程序快速创建更高级别的应用程序，通常使用脚本语言。这种范例是多任务计算（MTC）的一种形式，它专注于程序之间的数据传递，作为普通文件而不是消息。虽然它具有解耦生产者和消费者并允许现有应用程序并行执行而无需重新编码的显著优点，但其使用共享文件系统的典型实现方式对整个系统以及将分析和消费下游数据的用户造成了高性能负担。以前的工作在松散耦合的程序中实现了很大的加速，但是在所有共享文件系统访问中都进行了仔细的手动调优。在这项工作中，我们评估了一个原型集体IO模型为基础的文件MTC。该模型使输入数据文件能够高效且容易地分发到计算节点，并从计算节点收集输出结果。它消除了这种手动调优的需要，并使使用松散耦合模型的大规模集群编程变得更容易。我们的方法，灵感来自内存中的方法，集体操作的并行编程，建立在快速本地文件系统提供高速本地文件缓存的并行脚本，使用广播的方法来处理常见的输入数据的分布，并使用有效的分散/收集和缓存技术的输入和输出。我们描述的原型模型的设计，其实施的蓝色基因/P超级计算机，并提出了初步测量其性能的合成基准和大规模的分子动力学应用。
英文摘要：
Loosely coupled programming is a powerful paradigm for rapidly creating higher-level applications from scientific programs on petascale systems, typically using scripting languages. This paradigm is a form of many-task computing (MTC) which focuses on the passing of data between programs as ordinary files rather than messages. While it has the significant benefits of decoupling producer and consumer and allowing existing application programs to be executed in parallel with no recoding, its typical implementation using shared file systems places a high performance burden on the overall system and on the user who will analyze and consume the downstream data. Previous efforts have achieved great speedups with loosely coupled programs, but have done so with careful manual tuning of all shared file system access. In this work, we evaluate a prototype collective IO model for file-based MTC. The model enables efficient and easy distribution of input data files to computing nodes and gathering of output results from them. It eliminates the need for such manual tuning and makes the programming of large-scale clusters using a loosely coupled model easier. Our approach, inspired by in-memory approaches to collective operations for parallel programming, builds on fast local file systems to provide high-speed local file caches for parallel scripts, uses a broadcast approach to handle distribution of common input data, and uses efficient scatter/gather and caching techniques for input and output. We describe the design of the prototype model, its implementation on the Blue Gene/P supercomputer, and present preliminary measurements of its performance on synthetic benchmarks and on a large-scale molecular dynamics application.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
