arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2101.02591cs.DB
收藏

ORNL中子散射数据简化工作流程中的有效数据管理

Efficient Data Management in Neutron Scattering Data Reduction Workflows at ORNL

William F Godoy, Peter F Peterson, Steven E Hahn, Jay J Billings

摘要:

橡树岭国家实验室(ORNL)的实验中子科学设施每天产生1.2TB的基于事件的原始数据,这些数据使用基于HDF 5文件格式构建的标准元数据丰富的NeXus模式存储。Mantid是一个开放的源数据分析框架,在世界各地的几个中子科学设施中使用,几个数据简化工作流程的性能在很大程度上取决于加载和处理算法所花费的时间。目前的工作介绍了新的数据管理算法,以解决已确定的输入输出(I/O)的瓶颈Mantid。首先,我们引入了一个内存中的二叉树元数据索引,它类似于NeXus数据访问模式,提供了一个可伸缩的搜索和提取机制。其次,Mantid算法中的数据封装被优化地重新设计,以减少与元数据I/O重构任务相关联的总计算和内存运行时占用。这项工作的结果显示,ORNL数据缩减工作流程的挂钟时间加快了11%到30%,具体取决于目标仪器特定数据的复杂性。尽管如此,我们强调需要更多的研究来解决减少实验数据量增加的挑战。

英文摘要:

Oak Ridge National Laboratory (ORNL) experimental neutron science facilities produce 1.2\,TB a day of raw event-based data that is stored using the standard metadata-rich NeXus schema built on top of the HDF5 file format. Performance of several data reduction workflows is largely determined by the amount of time spent on the loading and processing algorithms in Mantid, an open-source data analysis framework used across several neutron sciences facilities around the world. The present work introduces new data management algorithms to address identified input output (I/O) bottlenecks on Mantid. First, we introduce an in-memory binary-tree metadata index that resemble NeXus data access patterns to provide a scalable search and extraction mechanism. Second, data encapsulation in Mantid algorithms is optimally redesigned to reduce the total compute and memory runtime footprint associated with metadata I/O reconstruction tasks. Results from this work show speed ups in wall-clock time on ORNL data reduction workflows, ranging from 11\% to 30\% depending on the complexity of the targeted instrument-specific data. Nevertheless, we highlight the need for more research to address reduction challenges as experimental data volumes increase.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑