RapidRAID：分布式存储系统中用于快速数据归档的流水线擦除码
RapidRAID: Pipelined Erasure Codes for Fast Data Archival in Distributed Storage Systems
摘要：
为了在分布式存储系统中实现可靠性，数据通常在不同的节点上复制。然而，要存储的数据量的增加促使了纠删码的引入，纠删码是复制的存储高效替代方案，特别适合于数据中心中的存档，其中旧数据集（很少访问）可以被纠删编码，而副本仅针对最新数据进行维护。许多最近的作品考虑新的存储为中心的擦除码的设计，以提高可修复性。相反，本文解决了从复制到编码的迁移：传统上，纠删编码是一种原子操作，其中具有整个对象的单个节点编码并上传所有编码片段。虽然大型数据集可以通过在不同节点之间分布单个对象编码来并行存档，但由于这种原子性，单个节点的网络和计算能力限制了存档过程。我们提出了一种新的流水线编码策略，在不同的节点之间分配网络和计算负载的单对象编码，这也加快了多个对象存档。我们进一步提出RapidRAID代码，一个明确的家庭的流水线擦除码，提供快速归档，而不损害数据的可靠性或存储开销。最后，我们提供了RapidRAID代码的实际实现，并使用50个节点的集群和一组Amazon EC2实例对其性能进行了基准测试。实验表明，RapidRAID编码减少了单个对象的编码时间高达90%，而当多个对象并发编码时，减少高达20%。
英文摘要：
To achieve reliability in distributed storage systems, data has usually been replicated across different nodes. However the increasing volume of data to be stored has motivated the introduction of erasure codes, a storage efficient alternative to replication, particularly suited for archival in data centers, where old datasets (rarely accessed) can be erasure encoded, while replicas are maintained only for the latest data. Many recent works consider the design of new storage-centric erasure codes for improved repairability. In contrast, this paper addresses the migration from replication to encoding: traditionally erasure coding is an atomic operation in that a single node with the whole object encodes and uploads all the encoded pieces. Although large datasets can be concurrently archived by distributing individual object encodings among different nodes, the network and computing capacity of individual nodes constrain the archival process due to such atomicity. We propose a new pipelined coding strategy that distributes the network and computing load of single-object encodings among different nodes, which also speeds up multiple object archival. We further present RapidRAID codes, an explicit family of pipelined erasure codes which provides fast archival without compromising either data reliability or storage overheads. Finally, we provide a real implementation of RapidRAID codes and benchmark its performance using both a cluster of 50 nodes and a set of Amazon EC2 instances. Experiments show that RapidRAID codes reduce a single object's coding time by up to 90%, while when multiple objects are encoded concurrently, the reduction is up to 20%.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
