High-performance computing platforms such as supercomputers have traditionally been designed to meet the compute demands of scientific applications. Consequently, they have been architected as producers and not consumers of data. The Apache Hadoop ecosystem has evolved to meet the requirements of data processing applications and has addressed many of the limitations of HPC platforms. There exist a class of scientific applications however, that need the collective capabilities of traditional high-performance computing environments and the Apache Hadoop ecosystem. For example, the scientific domains of bio-molecular dynamics, genomics and network science need to couple traditional computing with Hadoop/Spark based analysis. We investigate the critical question of how to present the capabilities of both computing environments to such scientific applications. Whereas this questions needs answers at multiple levels, we focus on the design of resource management middleware that might support the needs of both. We propose extensions to the Pilot-Abstraction to provide a unifying resource management layer. This is an important step that allows applications to integrate HPC stages (e.g. simulations) to data analytics. Many supercomputing centers have started to officially support Hadoop environments, either in a dedicated environment or in hybrid deployments using tools such as myHadoop. This typically involves many intrinsic, environment-specific details that need to be mastered, and often swamp conceptual issues like: How best to couple HPC and Hadoop application stages? How to explore runtime trade-offs (data localities vs. data movement)? This paper provides both conceptual understanding and practical solutions to the integrated use of HPC and Hadoop environments.
本文介绍了一种在HPC系统上处理大数据工作负载的自动化方法。我们描述了一种解决方案，该解决方案在HPC环境中动态创建基于YARN的统一集群，而无需配置和分配专用的Hadoop集群。最终用户可以选择在支持的框架的任意组合中编写解决方案，该解决方案可以从几个核心无缝扩展到数千个核心。这种环境的耦合为应用程序创建了一个平台，以利用本地HPC解决方案以及大数据框架。用户将获得多种语言的HPC Wales API，这将使他们能够将此流集成到他们的环境中，从而确保HPC访问的传统方式不会成为瓶颈。我们描述的行为的集群创建和性能结果Terasort。
CHIPP（Computing HTC in INAF Pilot Project）是一个由意大利天体物理研究所（INAF）资助并由INAF ICT办公室推动的意大利项目。CHIPP项目的主要目的是为INAF社区协调使用和访问现有的高吞吐量计算和高性能计算和数据处理资源（用于中小型项目）。今天，Tier 2/Tier 3系统（1，200 CPU/核心）在的里雅斯特和卡塔尼亚的INAF研究所提供，但在未来，该项目将发展包括其他计算基础设施。在过去的两年中，已经批准了30多个项目，总需求量为3000万CPU-h。大多数程序都是HPC，数据简化和分析，机器学习。在本文中，我们详细介绍了CHIPP的基础设施和头两年的活动结果。
Scalable Runtime Architecture for Data-driven, Hybrid HPC and ML Workflow Applications
Rutgers, the State University of New Jersey（罗格斯大学（新泽西州立大学））；Brookhaven National Laboratory（布鲁克海文国家实验室）；Princeton Plasma Physics Laboratory（普林斯顿等离子体物理实验室）；Princeton University（普林斯顿大学）
本文扩展 RADICAL-Pilot，提出支持基于服务执行的可扩展运行时架构，以实现跨本地与远程 HPC/云资源的 HPC/ML 无缝耦合和低开销并发 ML 执行。
