使用SquashFS和Singularity部署大型固定文件数据集
Deploying large fixed file datasets with SquashFS and Singularity
摘要：
共享高性能计算（HPC）平台，如XSEDE和Compute Canada提供的平台，使研究人员能够以云计算成本的一小部分进行大规模计算实验。大多数系统需要使用分布式文件系统（例如Lustre）来提供高度多用户、大容量的存储环境。由于网络争用和元数据性能的原因，文件数量增加时，这些方法会降低性能。我们展示了Singularity和SquashFS这两种技术的组合如何帮助开发人员、集成商、架构师和科学家在这些共享系统上部署大型数据集（O（10M）文件），同时将性能限制降到最低。与基于文件的常规数据集安装相比，建议的集成能够实现更高效的访问和索引，同时为用户和进程提供透明的文件访问。此外，该方法不需要目标系统上的管理权限。虽然这里研究的例子来自神经成像领域，但所采用的技术并不特定于该领域。目前，此解决方案仅限于只读数据集。我们建议采用这种技术，在共享的计算资源的消费和社区数据集的传播。
英文摘要：
Shared high-performance computing (HPC) platforms, such as those provided by XSEDE and Compute Canada, enable researchers to carry out large-scale computational experiments at a fraction of the cost of the cloud. Most systems require the use of distributed filesystems (e.g. Lustre) for providing a highly multi-user, large capacity storage environment. These suffer performance penalties as the number of files increases due to network contention and metadata performance. We demonstrate how a combination of two technologies, Singularity and SquashFS, can help developers, integrators, architects, and scientists deploy large datasets (O(10M) files) on these shared systems with minimal performance limitations. The proposed integration enables more efficient access and indexing than normal file-based dataset installations, while providing transparent file access to users and processes. Furthermore, the approach does not require administrative privileges on the target system. While the examples studied here have been taken from the field of neuroimaging, the technologies adopted are not specific to that field. Currently, this solution is limited to read-only datasets. We propose the adoption of this technology for the consumption and dissemination of community datasets across shared computing resources.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
