基于网格数据场的全球快速文件复制
Worldwide Fast File Replication on Grid Datafarm
摘要：
网格数据农场体系结构是为全球千万亿次数据密集型计算而设计的。它提供了一个具有在线千万亿次存储、可扩展I/O带宽和可扩展并行处理的全局并行文件系统，并且可以在具有数万个节点的集群网格中利用本地I/O。其特点之一是在文件系统元数据中管理文件副本，以实现容错和负载均衡。本文讨论和评价了几种支持远程快速文件复制的技术。Grid Datafarm将一组分级的文件作为一个Gfarm文件进行管理，每个文件（称为Gfarm文件片段）存储在一个文件系统节点上，或复制到多个文件系统节点上。每个Gfarm文件片段都是独立复制的，并使用具有网络条带化的速率控制的HighSpeed TCP并行复制。在距离为10，000 km的美日试验床上，我们使用每侧2个节点实现419 Mbps，使用两个跨太平洋网络的893 Mbps中的4个节点实现741 Mbps。
英文摘要：
The Grid Datafarm architecture is designed for global petascale data-intensive computing. It provides a global parallel filesystem with online petascale storage, scalable I/O bandwidth, and scalable parallel processing, and it can exploit local I/O in a grid of clusters with tens of thousands of nodes. One of features is that it manages file replicas in filesystem metadata for fault tolerance and load balancing. This paper discusses and evaluates several techniques to support long-distance fast file replication. The Grid Datafarm manages a ranked group of files as a Gfarm file, each file, called a Gfarm file fragment, being stored on a filesystem node, or replicated on several filesystem nodes. Each Gfarm file fragment is replicated independently and in parallel using rate-controlled HighSpeed TCP with network striping. On a US-Japan testbed with 10,000 km distance, we achieve 419 Mbps using 2 nodes on each side, and 741 Mbps using 4 nodes out of 893 Mbps with two transpacific networks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
