Docker等容器技术已经成为许多软件行业实践的重要组成部分,特别是那些与可复制性和可移植性有关的实践。容器化哲学影响了科学计算界,科学计算界已经开始采用甚至开发容器技术(例如Singularity)。将容器用于科学软件通常会带来与工业中遇到的挑战不同的挑战,并且需要不同的方法。对于HPC来说尤其如此。随着云中HPC的选择越来越多(包括NSF资助的云项目),人们有强烈的动机寻求解决方案,这些解决方案能够以可移植和可复制的方式在各种计算基础设施上灵活地开发和部署科学软件。云服务提供的灵活性使虚拟HPC集群能够按需扩展,XSEDE项目中的网络基础设施资源集成团队开发了一套工具,可在云中提供可扩展的基础设施。我们现在提出一个解决方案,它在一个支持MPI的Docker容器中使用Nix包管理器,该容器被转换为Singularity。它在每个映像中提供一致的安装、依赖项和环境,这些映像可跨科学计算基础架构进行复制和移植。我们使用部署在Jetstream和Aristotle Red Cloud OpenStack云中的Slurm调度器在自扩展虚拟集群中运行集群基准测试来演示这些容器的实用性。我们的结论是,这种技术是有用的科学软件应用程序容器的模板中使用的XSEDE计算环境,其他奇点HPC环境,和云计算环境。
英文摘要:
Container technologies such as Docker have become a crucial component of many software industry practices especially those pertaining to reproducibility and portability. The containerization philosophy has influenced the scientific computing community, which has begun to adopt - and even develop - container technologies (such as Singularity). Leveraging containers for scientific software often poses challenges distinct from those encountered in industry, and requires different methodologies. This is especially true for HPC. With an increasing number of options for HPC in the cloud (including NSF-funded cloud projects), there is strong motivation to seek solutions that provide flexibility to develop and deploy scientific software on a variety of computational infrastructures in a portable and reproducible way. The flexibility offered by cloud services enables virtual HPC clusters that scale on-demand, and the Cyberinfrastructure Resource Integration team in the XSEDE project has developed a set of tools which provides scalable infrastructure in the cloud. We now present a solution which uses the Nix package manager in an MPI-capable Docker container that is converted to Singularity. It provides consistent installations, dependencies, and environments in each image that are reproducible and portable across scientific computing infrastructures. We demonstrate the utility of these containers with cluster benchmark runs in a self-scaling virtual cluster using the Slurm scheduler deployed in the Jetstream and Aristotle Red Cloud OpenStack clouds. We conclude this technique is useful as a template for scientific software application containers to be used in the XSEDE compute environment, other Singularity HPC environments, and cloud computing environments.
HPC Containers for EBRAINS: Towards Portable Cross-Domain Software Environment
Simulation & Data Lab Neuroscience, Jülich Supercomputing Centre, Forschungszentrum Jülich(于利希研究中心);Kirchhoff Institute for Physics, Ruprecht-Karls-Universität(海德堡大学);ATHENA Research & Innovation Center(阿瑞塔研究与创新中心);Computer Engineering, FernUniversität in Hagen(哈根远程大学)
Towards Scalable Federated Container Orchestration: The CODECO Approach
fortiss GmbH(fortiss有限公司);Red Hat(红帽);IBM Research Europe - Zurich(IBM欧洲苏黎世研究中心);Siemens AG(西门子股份公司);i2CAT Foundation(i2CAT基金会);Telefónica(西班牙电信);Intracom Telecom(Intracom电信);Democritus University of Thrace / Athena Research Center(色雷斯德谟克利特大学/雅典研究中心);Universidad Politécnica de Madrid(马德里理工大学);University of Piraeus Research Centre(比雷埃夫斯大学研究中心);Netcompany;Universidad Carlos III de Madrid(马德里卡洛斯三世大学)
Accelerating discovery across scientific disciplines through reproducible workflows with AiiDAlab
Centre for Computational Chemistry, School of Chemistry, University of Bristol(布里斯托大学化学学院计算化学中心);PSI Center for Scientific Computing, Theory and Data, Paul Scherrer Institute(保罗谢尔研究所科学计算、理论与数据中心);Department of Information Technology and Electrical Engineering, ETH Zurich(苏黎世联邦理工学院信息技术与电气工程系);PSI Center for Neutron and Muon Sciences, Paul Scherrer Institute(保罗谢尔研究所中子与缪子科学中心)