大尺度气候数据分析的数据传输性能评估和对CMIP6数据基础设施的建议
An Assessment of Data Transfer Performance for Large-Scale Climate Data Analysis and Recommendations for the Data Infrastructure for CMIP6
摘要：
我们记录的数据传输工作流程，数据传输性能，和其他方面的分期约56 TB的气候模型输出数据从分布式耦合模型相互比较项目（CMIP 5）存档的国家能源研究超级计算中心（NERSC）在劳伦斯伯克利国家实验室所需的跟踪和表征热带风暴，在中纬度地区的重要现象。我们提出这种分析，以说明目前的挑战，在组装多模型数据集在主要的计算设施的大规模研究CMIP 5数据。由于即将到来的CMIP 6模型比对阶段的存档规模更大，我们预计这种数据传输将变得越来越重要，也许是例行的必要性。我们发现，使用ESGF的数据传送率往往慢于通常可用于美国住宅，有显着的空间来改善的ESGF门户网站和数据中心的数据传输能力，无论是在工作流机制和数据传输性能。我们相信，使用当前的最佳实践，至少一个数量级的性能改进是在技术范围内的，正如我们在两个高性能计算设施之间传输完整的原始数据集所实现的性能所示。为了实现这些性能改进，我们建议：（例如科学DMZ模型）应用于ESGF数据中心的数据服务器和网络;在ESGF数据中心投入足够的财政和人力资源用于系统和网络工程任务，以支持高性能数据移动;建立、定期测试和公布ESGF数据中心与用于气候数据分析的主要计算设施之间数据传输的性能指标。
英文摘要：
We document the data transfer workflow, data transfer performance, and other aspects of staging approximately 56 terabytes of climate model output data from the distributed Coupled Model Intercomparison Project (CMIP5) archive to the National Energy Research Supercomputing Center (NERSC) at the Lawrence Berkeley National Laboratory required for tracking and characterizing extratropical storms, a phenomena of importance in the mid-latitudes. We present this analysis to illustrate the current challenges in assembling multi-model data sets at major computing facilities for large-scale studies of CMIP5 data. Because of the larger archive size of the upcoming CMIP6 phase of model intercomparison, we expect such data transfers to become of increasing importance, and perhaps of routine necessity. We find that data transfer rates using the ESGF are often slower than what is typically available to US residences and that there is significant room for improvement in the data transfer capabilities of the ESGF portal and data centers both in terms of workflow mechanics and in data transfer performance. We believe performance improvements of at least an order of magnitude are within technical reach using current best practices, as illustrated by the performance we achieved in transferring the complete raw data set between two high performance computing facilities. To achieve these performance improvements, we recommend: that current best practices (such as the Science DMZ model) be applied to the data servers and networks at ESGF data centers; that sufficient financial and human resources be devoted at the ESGF data centers for systems and network engineering tasks to support high performance data movement; and that performance metrics for data transfer between ESGF data centers and major computing facilities used for climate data analysis be established, regularly tested, and published.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
