arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2108.10496cs.DC
收藏

预取对于基于云的大规模神经成像分析工作流的好处

The benefits of prefetching for large-scale cloud-based neuroimaging analysis workflows

Valerie Hayot-Sasson, Tristan Glatard, Ariel Rokem

摘要:

为了支持神经科学应用不断增长的需求,研究人员正在过渡到云计算,以获得可扩展、强大和弹性的基础设施。然而,驻留在对象存储中的大型数据集可能会在工作流执行期间导致显著的数据传输开销。预取是一种降低混合工作负载中读取成本的方法,它在先前任务的处理时间内掩盖了数据传输成本。我们提出了一个“滚动预取”的实现,这是一个Python库,它从AWS S3对象存储中实现了一种特定形式的预取,我们量化了它的好处。Rolling Prefetch扩展了S3Fs(一个通过文件对象公开AWS S3功能的Python库),以添加预取功能。在S3上存储的500 GB大脑连接数据集的测量分析性能中,我们发现预取提供了高达1.86倍的显着速度提升,即使在完全由数据加载组成的应用程序中也是如此。观察到的加速值与我们的理论分析是一致的。我们的研究结果表明,预取的科学数据处理云基础设施的实用性,并提供了适用于各种应用领域的实现。

英文摘要:

To support the growing demands of neuroscience applications, researchers are transitioning to cloud computing for its scalable, robust and elastic infrastructure. Nevertheless, large datasets residing in object stores may result in significant data transfer overheads during workflow execution. Prefetching, a method to mitigate the cost of reading in mixed workloads, masks data transfer costs within processing time of prior tasks. We present an implementation of "Rolling Prefetch", a Python library that implements a particular form of prefetching from AWS S3 object store, and we quantify its benefits. Rolling Prefetch extends S3Fs, a Python library exposing AWS S3 functionality via a file object, to add prefetch capabilities. In measured analysis performance of a 500 GB brain connectivity dataset stored on S3, we found that prefetching provides significant speed-ups of up to 1.86x, even in applications consisting entirely of data loading. The observed speed-up values are consistent with our theoretical analysis. Our results demonstrate the usefulness of prefetching for scientific data processing on cloud infrastructures and provide an implementation applicable to various application domains.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑