Python Workflows on HPC Systems

OMB-Py: Python Micro-Benchmarks for Evaluating Performance of MPI Libraries on HPC Systems

Python has become a dominant programming language for emerging areas like Machine Learning (ML), Deep Learning (DL), and Data Science (DS). An attractive feature of Python is that it provides easy-to-use programming interface while allowing library developers to enhance performance of their applications by harnessing the computing power offered by High Performance Computing (HPC) platforms. Efficient communication is key to scaling applications on parallel systems, which is typically enabled by the Message Passing Interface (MPI) standard and compliant libraries on HPC hardware. mpi4py is a Py…

AI-coupled HPC Workflows

Increasingly, scientific discovery requires sophisticated and scalable workflows. Workflows have become the ``new applications,'' wherein multi-scale computing campaigns comprise multiple and heterogeneous executable tasks. In particular, the introduction of AI/ML models into the traditional HPC workflows has been an enabler of highly accurate modeling, typically reducing computational needs compared to traditional methods. This chapter discusses various modes of integrating AI/ML models to HPC computations, resulting in diverse types of AI-coupled HPC workflows. The increasing need of couplin…

ML驱动的HPC工作流中异构任务的异步执行

Asynchronous Execution of Heterogeneous Tasks in ML-driven HPC Workflows

本文研究了ML驱动HPC工作流的异步任务执行需求，建模了异步程度并提出关键指标，在Summit上大规模实验验证了异步执行带来的性能提升与模型一致。

设计AI驱动的高性能计算工作流的十二条快速技巧

Twelve quick tips for designing AI-driven HPC workflows

本文针对AI与HPC融合带来的新挑战，提出十二条实用技巧，涵盖容器化、作业数组、反馈循环和I/O优化，帮助设计高效、可扩展、可复现的AI驱动HPC工作流。

Enabling Machine Learning-Ready HPC Ensembles with Merlin

With the growing complexity of computational and experimental facilities, many scientific researchers are turning to machine learning (ML) techniques to analyze large scale ensemble data. With complexities such as multi-component workflows, heterogeneous machine architectures, parallel file systems, and batch scheduling, care must be taken to facilitate this analysis in a high performance computing (HPC) environment. In this paper, we present Merlin, a workflow framework to enable large ML-friendly ensembles of scientific HPC simulations. By augmenting traditional HPC with distributed compute …

pyWATTS: Python Workflow Automation Tool for Time Series

Time series data are fundamental for a variety of applications, ranging from financial markets to energy systems. Due to their importance, the number and complexity of tools and methods used for time series analysis is constantly increasing. However, due to unclear APIs and a lack of documentation, researchers struggle to integrate them into their research projects and replicate results. Additionally, in time series analysis there exist many repetitive tasks, which are often re-implemented for each project, unnecessarily costing time. To solve these problems we present \texttt{pyWATTS}, an ope…