立足当下,面向未来:于利希 LOFAR 长期档案
Learning from the present for the future: the Juelich LOFAR Long-term Archive
AI总结:
本文总结于利希 LOFAR 长期档案的数据管理与访问瓶颈,分析 10 年可用性、访问模式和能耗,并提出未来更大规模射电天文数据档案的改进方案与需求。
摘要:
于利希研究中心自 2013 年起一直托管 LOFAR 档案的德国部分。这是德国最庞大的射电天文档案,目前存储近 22 拍字节(PB)的数据。未来的射电望远镜预计将需要长期数据存储量的大幅增加。在此,我们评估于利希 LOFAR 数据档案当前的数据管理状况,描述数据摄取、存储系统、向长期档案的导出以及请求链。我们分析了过去 10 年的数据可用性,并探究了背后的数据访问模式和该过程的能耗。我们确定硬件相关的限制因素,如网络带宽、缓存池可用性与性能,以及软件方面的问题,例如工作流调整和参数调优,是主要的数据存储瓶颈。相比之下,从档案中向用户提供数据的挑战在于从磁带档案中检索数据并对其进行暂存。基于这一分析,我们建议未来如何避免或缓解这些问题,并为未来更大规模的长期数据档案定义需求。
英文摘要:
The Forschungszentrum Juelich has been hosting the German part of the LOFAR archive since 2013. It is Germany's most extensive radio astronomy archive, currently storing nearly 22 petabytes (PB) of data. Future radio telescopes are expected to require a dramatic increase in long-term data storage. Here, we take stock of the current data management of the Juelich LOFAR Data Archive, describe the ingestion, the storage system, the export to the long-term archive, and the request chain. We analysed the data availability over the last 10 years and searched for the underlying data access pattern and the energy consumption of the process. We determine hardware-related limiting factors, such as network bandwidth and cache pool availability and performance, and software aspects, e.g. workflow adjustment and parameter tuning, as the main data storage bottlenecks. By contrast, the challenge in providing the data from the archive for the users lies in retrieving the data from the tape archive and staging them. Building on this analysis, we suggest how to avoid/mitigate these problems in the future and define the requirements for future even more extensive long-term data archives.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。