arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2212.03054cs.DCcs.OS
收藏

DisTRaC:加速临时数据存储的高性能计算处理

DisTRaC: Accelerating High Performance Compute Processing for Temporary Data Storage

Gabryel Mason-Williams, Dave Bond, Mark Basham

摘要:

高性能计算(HPC)集群通常会在代码执行过程中生成中间文件,并且消息传递并不总是能够为这些集群作业提供数据。在这些情况下,I/O返回到中央分布式存储以允许跨节点数据共享。这些系统通常具有高性能,其特点是每TB成本高,对工作负载类型(如调整为小型或大型文件I/O)敏感。但是,计算节点通常具有大量RAM,因此在处理系统寿命或可靠性不那么重要的中间文件时,可以使用本地RAM磁盘来获得性能优势。在本文中,我们将展示如何通过创建一个可以与对象存储系统Ceph交互的RAM块,以及创建一个部署工具来有效地在HPC基础设施上部署Ceph来解决这个问题。这项工作导致了一个系统,比中央高性能分布式存储系统在钻石减少I/O开销和处理时间Savu,层析成像数据处理应用程序,分别由81.04%和8.32%。

英文摘要:

High Performance Compute (HPC) clusters often produce intermediate files as part of code execution and message passing is not always possible to supply data to these cluster jobs. In these cases, I/O goes back to central distributed storage to allow cross node data sharing. These systems are often high performance and characterised by their high cost per TB and sensitivity to workload type such as being tuned to small or large file I/O. However, compute nodes often have large amounts of RAM, so when dealing with intermediate files where longevity or reliability of the system is not as important, local RAM disks can be used to obtain performance benefits. In this paper we show how this problem was tackled by creating a RAM block that could interact with the object storage system Ceph, as well as creating a deployment tool to deploy Ceph on HPC infrastructure effectively. This work resulted in a system that was more performant than the central high performance distributed storage system used at Diamond reducing I/O overhead and processing time for Savu, a tomography data processing application, by 81.04% and 8.32% respectively.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑