HPC centers face increasing demand for software flexibility, and there is growing consensus that Linux containers are a promising solution. However, existing container build solutions require root privileges and cannot be used directly on HPC resources. This limitation is compounded as supercomputer diversity expands and HPC architectures become more dissimilar from commodity computing resources. Our analysis suggests this problem can best be solved with low-privilege containers. We detail relevant Linux kernel features, propose a new taxonomy of container privilege, and compare two open-source implementations: mostly-unprivileged rootless Podman and fully-unprivileged Charliecloud. We demonstrate that low-privilege container build on HPC resources works now and will continue to improve, giving normal users a better workflow to securely and correctly build containers. Minimizing privilege in this way can improve HPC user and developer productivity as well as reduce support workload for exascale applications.
XaaS Containers: Performance-Portable Representation With Source and IR Containers
ETH Zürich(苏黎世联邦理工学院);University of Chicago(芝加哥大学);Swiss National Supercomputing Centre (CSCS)(瑞士国家超级计算中心);Lawrence Livermore National Laboratory (LLNL)(劳伦斯利弗莫尔国家实验室);Argonne National Laboratory (ANL)(阿贡国家实验室)
本文提出 XaaS 性能可移植容器方案,通过源码和 IR 容器将性能关键决策推迟到部署时,并结合 LLM 辅助自动发现特化,从而兼顾容器化便利性与系统特化构建的性能优势。
HPC Containers for EBRAINS: Towards Portable Cross-Domain Software Environment
Simulation & Data Lab Neuroscience, Jülich Supercomputing Centre, Forschungszentrum Jülich(于利希研究中心);Kirchhoff Institute for Physics, Ruprecht-Karls-Universität(海德堡大学);ATHENA Research & Innovation Center(阿瑞塔研究与创新中心);Computer Engineering, FernUniversität in Hagen(哈根远程大学)
Self-Scaling Clusters and Reproducible Containers to Enable Scientific Computing
Docker等容器技术已经成为许多软件行业实践的重要组成部分,特别是那些与可复制性和可移植性有关的实践。容器化哲学影响了科学计算界,科学计算界已经开始采用甚至开发容器技术(例如Singularity)。将容器用于科学软件通常会带来与工业中遇到的挑战不同的挑战,并且需要不同的方法。对于HPC来说尤其如此。随着云中HPC的选择越来越多(包括NSF资助的云项目),人们有强烈的动机寻求解决方案,这些解决方案能够以可移植和可复制的方式在各种计算基础设施上灵活地开发和部署科学软件。云服务提供的灵活性使虚拟HPC集群能够按需扩展,XSEDE项目中的网络基础设施资源集成团队开发了一套工具,可在云中提供可扩展的基础设施。我们现在提出一个解决方案,它在一个支持MPI的Docker容器中使用Nix包管理器,该容器被转换为Singularity。它在每个映像中提供一致的安装、依赖项和环境,这些映像可跨科学计算基础架构进行复制和移植。我们使用部署在Jetstream和Aristotle Red Cloud OpenStack云中的Slurm调度器在自扩展虚拟集群中运行集群基准测试来演示这些容器的实用性。我们的结论是,这种技术是有用的科学软件应用程序容器的模板中使用的XSEDE计算环境,其他奇点HPC环境,和云计算环境。
The Hartree Centre, STFC(哈特里中心,STFC);Center for Computational Innovation, Rensselaer Polytechnic Institute(伦斯勒理工学院计算创新中心);Jülich Supercomputing Centre, Forschungszentrum Jülich(于利希超级计算中心,于利希研究中心);RWTH Aachen University(亚琛工业大学);IBM Quantum, IBM Research – Tokyo(IBM量子,IBM东京研究所);IBM Quantum, IBM Research—Zurich(IBM量子,IBM苏黎世研究所);Oak Ridge National Laboratory(橡树岭国家实验室);IBM Quantum, IBM T.J. Watson Research Center(IBM量子,IBM托马斯·J·沃森研究中心)