We present the construction of a new family of erasure correcting codes for distributed storage that yield low repair bandwidth and low repair complexity. The construction is based on two classes of parity symbols. The primary goal of the first class of symbols is to provide good erasure correcting capability, while the second class facilitates node repair, reducing the repair bandwidth and the repair complexity. We compare the proposed codes with other codes proposed in the literature.
Despite their exceptional error-correcting properties, Reed-Solomon (RS) codes have been overlooked in distributed storage applications due to the common belief that they have poor repair bandwidth: A naive repair approach would require the whole file to be reconstructed in order to recover a single erased codeword symbol. In a recent work, Guruswami and Wootters (STOC'16) proposed a single-erasure repair method for RS codes that achieves the optimal repair bandwidth amongst all linear encoding schemes. We extend their trace collection technique to cope with two erasures.
Reliability is essential for storing files in many applications of distributed storage systems. To maintain reliability, when a storage node fails, a new node should be regenerated by a repair process. Most of the previous results on the repair problem assume perfect (error-free) links in the networks. However, in practice, especially in a wireless network, the transmitted packets (for repair) may be lost due to, e.g., link failure or buffer overflow. We study the repair problem of distributed storage systems in packet erasure networks, where a packet loss is modeled as an erasure. The minimum…
Despite their exceptional error-correcting properties, Reed-Solomon codes have been overlooked in distributed storage applications due to the common belief that they have poor repair bandwidth: A naive repair approach would require the whole file to be reconstructed in order to recover a single erased codeword symbol. In a recent work, Guruswami and Wootters (STOC'16) proposed a single-erasure repair method for Reed-Solomon codes that achieves the optimal repair bandwidth amongst all linear encoding schemes. Their key idea is to recover the erased symbol by collecting a sufficiently large numb…
We study the repair problem of distributed storage systems in erasure networks where the packets transmitted from surviving nodes to the new node might be lost. The fundamental storage-bandwidth tradeoff is calculated by multicasting analysis in erasure networks. The optimal tradeoff bound can be asymptotically achieved when the number of transmission (packets) goes to infinity. For a limited number of transmission, we study the probability of successful regenerating. Then, we investigate two approaches of increasing the probability of successful regenerating, namely, by connecting more surviv…
A new class of exact-repair regenerating codes is constructed by combining two layers of erasure correction codes together with combinatorial block designs, e.g., Steiner systems, balanced incomplete block designs and t-designs. The proposed codes have the "uncoded repair" property where the nodes participating in the repair simply transfer part of the stored data directly, without performing any computation. The layered error correction structure makes the decoding process rather straightforward, and in general the complexity is low. We show that this construction is able to achieve performan…
The repair bandwidth of a code is the minimum amount of data required to repair one or several failed nodes (erasures). For MDS codes, the repair bandwidth is bounded below by the so-called cut-set bound, and codes that meet this bound with equality are said to support optimal repair of one or multiple failed nodes. We consider the problem of repairing multiple failed nodes of Reed-Solomon (RS) codes. In a recent work with I. Tamo (Proc. IEEE FOCS 2017), we gave the first explicit construction of RS codes with optimal repair of any single failed node from any subset of helper nodes. In this pa…
Locally repairable codes (LRC) for distribute storage allow two approaches to locally repair multiple failed nodes: 1) parallel approach, by which each newcomer access a set of $r$ live nodes $(r$ is the repair locality$)$ to download data and recover the lost packet; and 2) sequential approach, by which the newcomers are properly ordered and each newcomer access a set of $r$ other nodes, which can be either a live node or a newcomer ordered before it. An $[n,k]$ linear code with locality $r$ and allows local repair for up to $t$ failed nodes by sequential approach is called an $(n,k,r,t)$-exa…
Motivated by iterative decoding techniques for the binary erasure channel Hollmann and Tolhuizen introduced and studied the notion of generic erasure correcting sets for linear codes. A generic $(r,s)$--erasure correcting set generates for all codes of codimension $r$ a parity check matrix that allows iterative decoding of all correctable erasure patterns of size $s$ or less. The problem is to derive bounds on the minimum size $F(r,s)$ of generic erasure correcting sets and to find constructions for such sets. In this paper we continue the study of these sets. We derive better lower and upper …
In this paper we study array-based codes over graphs for correcting multiple node failures. These codes have applications to neural networks, associative memories, and distributed storage systems. We assume that the information is stored on the edges of a complete undirected graph and a node failure is the event where all the edges in the neighborhood of a given node have been erased. A code over graphs is called $ρ$-node-erasure-correcting if it allows to reconstruct the erased edges upon the failure of any $ρ$ nodes or less. We present a binary optimal construction for double-node-erasure co…
In this paper, we show that the single deletion correcting Varshamov-Tenengolts code, with minor modifications, can also correct an ordered deletion-erasure pattern where one deletion and at most one erasure occur and the deletion always occurs before the erasure. For large code lengths, the constructed code has the same logarithmic redundancy as optimal codes.
This paper proposes an erasure correcting code and its systematic form for the distributed storage system. The proposed codes are encoded by exclusive OR and bit-level shift operation. By the shift operation, the encoded packets are slightly longer than the source packets. This paper evaluates the extra length of encoded packets, called overhead, and shows that the proposed codes have smaller overheads than the zigzag decodable code, which is an existing code using exclusive OR and bit-level shift operation.
We study the problem of centralized exact repair of multiple failures in distributed storage. We describe constructions that achieve a new set of interior points under exact repair. The constructions build upon the layered code construction by Tian et al., designed for exact repair of single failure. We firstly improve upon the layered construction for general system parameters. Then, we extend the improved construction to support the repair of multiple failures, with varying number of helpers. In particular, we prove the optimality of one point on the functional repair tradeoff of multiple fa…
When a node in a distributed storage system fails, it needs to be promptly repaired to maintain system integrity. While typical erasure codes can provide a significant storage advantage over replication, they suffer from poor repair efficiency. Locally repairable codes (LRCs) tackle this issue by reducing the number of nodes participating in the repair process (locality), at the cost of reduced minimum distance. In this paper, we study the tradeoff between locality and minimum distance of LRCs with local codes that have arbitrary distance requirements. Unlike existing methods where both the lo…
Locally repairable convolutional codes (LRCCs) for distributed storage systems (DSSs) are introduced in this work. They enable local repair, for a single node erasure (or more generally, $ \partial - 1 $ erasures per local group), and sliding-window global repair, which can correct erasure patterns with up to $ {\rm d}^c_j - 1 $ erasures in every window of $ j+1 $ consecutive blocks of $ n $ nodes, where $ {\rm d}^c_j $ is the $ j $th column distance of the code. The parameter $ j $ can be adjusted, for a fixed LRCC, according to different catastrophic erasure patterns, requiring only to conta…
Motivated by modern network communication applications which require low latency, we study codes that correct erasures with low decoding delay. We provide a simple explicit construction that yields convolutional codes that can correct both burst and arbitrary erasures under a maximum decoding delay constraint $T$. Our proposed code has efficient encoding/decoding algorithms and requires a field size that is linear in $T$. We study the performance of our code over the Gilbert-Elliot channel; our simulation results show significant performance gains over low-delay codes existing in the literatur…
In distributed storage systems, erasure codes with locality $r$ is preferred because a coordinate can be recovered by accessing at most $r$ other coordinates which in turn greatly reduces the disk I/O complexity for small $r$. However, the local repair may be ineffective when some of the $r$ coordinates accessed for recovery are also erased. To overcome this problem, we propose the $(r,δ)_c$-locality providing $δ-1$ local repair options for a coordinate. Consequently, the repair locality $r$ can tolerate $δ-1$ erasures in total. We derive an upper bound on the minimum distance $d$ for any line…
Founsure is an open-source software library that implements a multi-dimensional graph-based erasure coding entirely based on fast exclusive OR (XOR) logic. Its implementation utilizes compiler optimizations and multi-threading to generate the right assembly code for the given multi-core CPU architecture with vector processing capabilities. Founsure possesses important features that shall find various applications in modern data storage, communication, and networked computer systems, in which the data needs protection against device, hardware, and node failures. As data size reached unprecedent…
We propose new repair schemes for Reed-Solomon codes that use subspace polynomials and hence generalize previous works in the literature that employ trace polynomials. The Reed-Solomon codes are over $\mathbb{F}_{q^\ell}$ and have redundancy $r = n-k \geq q^m$, $1\leq m\leq \ell$, where $n$ and $k$ are the code length and dimension, respectively. In particular, for one erasure, we show that our schemes can achieve optimal repair bandwidths whenever $n=q^\ell$ and $r = q^m,$ for all $1 \leq m \leq \ell$. For two erasures, our schemes use the same bandwidth per erasure as the single erasure sche…
Cooperative regenerating codes are regenerating codes designed to tradeoff storage for repair bandwidth in case of multiple node failures. Minimum storage cooperative regenerating (MSCR) codes are a class of cooperative regenerating codes which achieve the minimum storage point of the tradeoff. Recently, these codes have been constructed for all possible parameters $(n,k,d,h)$, where $h$ erasures are repaired by contacting any $d$ surviving nodes. However, these constructions have very large sub-packetization. $ε$-MSR codes are a class of codes introduced to tradeoff subpacketization level for…
We focus on erasure codes for distributed storage. The distributed storage setting imposes locality requirements because of easy repair demands on the decoder. We first establish the characterization of various locality properties in terms of the generator matrix of the code. These lead to bounds on locality and notions of optimality. We then examine the locality properties of a family of non-binary codes with simplex structure. We investigate their optimality and design several easy repair decoding methods. In particular, we show that any correctable erasure pattern can be solved by easy repa…
Optimal locally repairable codes with information locality are considered. Optimal codes are constructed, whose length is also order-optimal with respect to a new bound on the code length derived in this paper. The length of the constructed codes is super-linear in the alphabet size, which improves upon the well known pyramid codes, whose length is only linear in the alphabet size. The recoverable erasure patterns are also analyzed for the new codes. Based on the recoverable erasure patterns, we construct generalized sector-disk (GSD) codes, which can recover from disk erasures mixed with sect…
In a distributed storage system, code symbols are dispersed across space in nodes or storage units as opposed to time. In settings such as that of a large data center, an important consideration is the efficient repair of a failed node. Efficient repair calls for erasure codes that in the face of node failure, are efficient in terms of minimizing the amount of repair data transferred over the network, the amount of data accessed at a helper node as well as the number of helper nodes contacted. Coding theory has evolved to handle these challenges by introducing two new classes of erasure codes,…
An interactive error correcting code ($\mathsf{iECC}$) is an interactive protocol with the guarantee that the receiver can correctly determine the sender's message, even in the presence of noise. This generalizes the concept of an error correcting code ($\mathsf{ECC}$), which is a non-interactive $\mathsf{iECC}$ that is known to have erasure resilience capped at $\frac12$. The work of \cite{GuptaTZ21} constructed the first $\mathsf{iECC}$ resilient to $> \frac12$ adversarial erasures. However, their $\mathsf{iECC}$ has communication complexity quadratic in the message size. In our work, we con…
In this paper, a simple, general-purpose and effective tool for the design of low-density parity-check (LDPC) codes for iterative correction of bursts of erasures is presented. The design method consists in starting from the parity-check matrix of an LDPC code and developing an optimized parity-check matrix, with the same performance on the memory-less erasure channel, and suitable also for the iterative correction of single bursts of erasures. The parity-check matrix optimization is performed by an algorithm called pivot searching and swapping (PSS) algorithm, which executes permutations of c…
A generic $(r,m)$-erasure correcting set is a collection of vectors in $\bF_2^r$ which can be used to generate, for each binary linear code of codimension $r$, a collection of parity check equations that enables iterative decoding of all correctable erasure patterns of size at most $m$. That is to say, the only stopping sets of size at most $m$ for the generated parity check equations are the erasure patterns for which there is more than one manner to fill in theerasures to obtain a codeword. We give an explicit construction of generic $(r,m)$-erasure correcting sets of cardinality $\sum_{i=0}…
Erasure codes are an efficient means of storing data across a network in comparison to data replication, as they tend to reduce the amount of data stored in the network and offer increased resilience in the presence of node failures. The codes perform poorly though, when repair of a failed node is called for, as they typically require the entire file to be downloaded to repair a failed node. A new class of erasure codes, termed as regenerating codes were recently introduced, that do much better in this respect. However, given the variety of efficient erasure codes available in the literature, …
Dau et al. recently extend Guruswami and Wootters' scheme (STOC'2016) to cooperatively repair two or three erasures in Reed-Solomon (RS) codes. However, their scheme restricts to either the case that the characteristic of $F$ divides the extension degree $[F\!:\!B]$ or some special failure patterns, where $F$ is the base field of the RS code and $B$ is the subfield of the repair symbols. In this paper, we derive an improved cooperative repair scheme that removes all these restrictions. That is, our scheme applies to any characteristic of $F$ and can repair all failure patterns of two or three …
Recently, locally repairable codes has gained significant interest for their potential applications in distributed storage systems. However, most constructions in existence are over fields with size that grows with the number of servers, which makes the systems computationally expensive and difficult to maintain. Here, we study linear locally repairable codes over the binary field, tolerating multiple local erasures. We derive bounds on the minimum distance on such codes, and give examples of LRCs achieving these bounds. Our main technical tools come from matroid theory, and as a byproduct of …
MDS codes are erasure-correcting codes that can correct the maximum number of erasures for a given number of redundancy or parity symbols. If an MDS code has $r$ parities and no more than $r$ erasures occur, then by transmitting all the remaining data in the code, the original information can be recovered. However, it was shown that in order to recover a single symbol erasure, only a fraction of $1/r$ of the information needs to be transmitted. This fraction is called the repair bandwidth (fraction). Explicit code constructions were given in previous works. If we view each symbol in the code a…
We consider the problem of designing [n; k] linear codes for distributed storage systems (DSS) that satisfy the (r, t)-Local Repair Property, where any t'(<=t) simultaneously failed nodes can be locally repaired, each with locality r. The parameters n, k, r, t are positive integers such that r<k<n and t <= n-k. We consider the functional repair model and the sequential approach for repairing multiple failed nodes. By functional repair, we mean that the packet stored in each newcomer is not necessarily an exact copy of the lost data but a symbol that keep the (r, t)-local repair property. By th…
The repair problem in distributed storage addresses recovery of the data encoded using an erasure code, for instance, a Reed-Solomon (RS) code. We consider the problem of repairing a single node or multiple nodes in RS-coded storage systems using the smallest possible amount of inter-nodal communication. According to the cut-set bound, communication cost of repairing $h\ge 1$ failed nodes for an $(n,k=n-r)$ MDS code using $d$ helper nodes is at least $dhl/(d+h-k),$ where $l$ is the size of the node. Guruswami and Wootters (2016) initiated the study of efficient repair of RS codes, showing that…
Distributed and cloud storage systems are used to reliably store large-scale data. Erasure codes have been recently proposed and used in real-world distributed and cloud storage systems such as Google File System, Microsoft Azure Storage, and Facebook HDFS-RAID, to enhance the reliability. In order to decrease the repair bandwidth and disk I/O, a class of erasure codes called locally repairable codes (LRCs) have been proposed which have small locality compare to other erasure codes. Although LRCs have small locality, they have lower minimum distance compare to the Singleton bound. Hence, seeki…
In a modern distributed storage system, storage nodes are organized in racks, and the cross-rack communication dominates the system bandwidth. In We study the rack-aware storage system where all storage nodes are organized in racks and within each rack the nodes can communicate freely without taxing the system bandwidth. Rack-aware regenerating codes (RRCs) were proposed for minimizing the repair bandwidth for single erasures. In the initial setting of RRCs, the repair of a single node requires the participation of all the remaining nodes in the rack containing the failed node as well as a lar…
The piggybacking framework for designing erasure codes for distributed storage has empirically proven to be very useful, and has been used to design codes with desirable properties, such as low repair bandwidth and complexity. However, the theoretical properties of this framework remain largely unexplored. We address this by adapting a general characterization of repair schemes (previously used for Reed Solomon codes) to analyze piggybacking codes with low substriping. With this characterization, we establish a separation between piggybacking and general erasure codes, and several impossibilit…
We consider the locally repairable codes (LRC), aiming at sequential recovering multiple erasures. We define the (n,k,r,t)-SLRC (Sequential Locally Repairable Codes) as an [n,k] linear code where any t'(>= t) erasures can be sequentially recovered, each one by r (2<=r<k) other code symbols. Sequential recovering means that the erased symbols are recovered one by one, and an already recovered symbol can be used to recover the remaining erased symbols. This important recovering method, in contrast with the vastly studied parallel recovering, is currently far from understanding, say, lacking code…
Network transfer and disk read are the most time consuming operations in the repair process for node failures in erasure-code-based distributed storage systems. Recent developments on Reed-Solomon codes, the most widely used erasure codes in practical storage systems, have shown that efficient repair schemes specifically tailored to these codes can significantly reduce the network bandwidth spent to recover single failures. However, the I/O cost, that is, the number of disk reads performed in these repair schemes remains largely unknown. We take the first step to address this gap in the litera…
In distributed storage systems that employ erasure coding, the issue of minimizing the total {\it communication} required to exactly rebuild a storage node after a failure arises. This repair bandwidth depends on the structure of the storage code and the repair strategies used to restore the lost data. Designing high-rate maximum-distance separable (MDS) codes that achieve the optimum repair communication has been a well-known open problem. In this work, we use Hadamard matrices to construct the first explicit 2-parity MDS storage code with optimal repair properties for all single node failure…
Recently, locally repairable codes (LRCs) with local erasure correction constraints that are unequal and disjoint have been proposed. In this work, we study the same topic and provide some improved and additional results.
Erasure coding is widely used for massive storage in data centers to achieve high fault tolerance and low storage redundancy. Since the cross-rack communication cost is often high, it is critical to design erasure codes that minimize the cross-rack repair bandwidth during failure repair. In this paper, we analyze the optimal trade-off between storage redundancy and cross-rack repair bandwidth specifically for data centers, subject to the condition that the original data can be reconstructed from a sufficient number of any non-failed nodes. We characterize the optimal trade-off curve under func…
An error correcting code ($\mathsf{ECC}$) allows a sender to send a message to a receiver such that even if a constant fraction of the communicated bits are corrupted, the receiver can still learn the message correctly. Due to their importance and fundamental nature, $\mathsf{ECC}$s have been extensively studied, one of the main goals being to maximize the fraction of errors that the $\mathsf{ECC}$ is resilient to. For adversarial erasure errors (over a binary channel) the maximal error resilience of an $\mathsf{ECC}$ is $\frac12$ of the communicated bits. In this work, we break this $\frac12$…
The minimum storage rack-aware regenerating (MSRR) code is a variation of regenerating codes that achieves the optimal repair bandwidth for a single node failure in the rack-aware model. The authors in~\cite{Chen-Barg2019} and~\cite{Zhou-Zhang2021} provided explicit constructions of MSRR codes for all parameters to repair a single failed node. This paper generalizes the results in~\cite{Chen-Barg2019} to the case of multiple node failures. We propose a class of MDS array codes and scalar Reed-Solomon (RS) codes, and show that these codes have optimal repair bandwidth and error resilient capabi…
MDS array codes are widely used in storage systems to protect data against erasures. We address the \emph{rebuilding ratio} problem, namely, in the case of erasures, what is the fraction of the remaining information that needs to be accessed in order to rebuild \emph{exactly} the lost information? It is clear that when the number of erasures equals the maximum number of erasures that an MDS code can correct then the rebuilding ratio is 1 (access all the remaining information). However, the interesting and more practical case is when the number of erasures is smaller than the erasure correcting…
This paper investigates the use of redundancy and self repairing against node failures in distributed storage systems, using various strategies. In replication method, access to one replication node is sufficient to reconstruct a lost node, while in MDS erasure coded systems which are optimal in terms of redundancy-reliability tradeoff, a single node failure is repaired after recovering the entire stored data. Moreover, regenerating codes yield a tradeoff curve between storage capacity and repair bandwidth. The current paper aims at investigating a new storage code. Specifically, we propose a …
In 2007, Martinian and Trott presented codes for correcting a burst of erasures with a minimum decoding delay. Their construction employs [n,k] codes that can correct any burst of erasures (including wrap-around bursts) of length n-k. The raised the question if such [n,k] codes exist for all integers k and n with 1<= k <= n and all fields (in particular, for the binary field). In this note, we answer this question affirmatively by giving two recursive constructions and a direct one.
Partial MDS (PMDS) and sector-disk (SD) codes are classes of erasure correcting codes that combine locality with strong erasure correction capabilities. We construct PMDS and SD codes with local regeneration where each local code is a bandwidth-optimal regenerating MDS code. In the event of a node failure, these codes reduce both, the number of servers that have to be contacted as well as the amount of network traffic required for the repair process. The constructions require significantly smaller field size than the only other construction known in literature. Further, we present a constructi…
Erasure codes have emerged as an efficient technology for providing data redundancy in distributed storage systems. However, it is a challenging task to repair the failed storage nodes in erasure-coded storage systems, which requires large quantities of network resources. In this paper, we study fractional repetition (FR) codes, which enable the minimal repair complexity and also minimum repair bandwidth during node repair. We focus on the duality of FR codes, and investigate the relationship between the supported file size of an FR code and its dual code. Furthermore, we present a dual bound …
Batch codes are a family of codes that represent a distributed storage system (DSS) of $n$ nodes so that any batch of $t$ data symbols can be retrieved by reading at most one symbol from each node. Fractional repetition codes are a family of codes for DSS that enable efficient uncoded repairs of failed nodes. In this work these two families of codes are combined to obtain fractional repetition batch (FRB) codes which provide both uncoded repairs and parallel reads of subsets of stored symbols. In addition, new batch codes which can tolerate node failures are considered. This new family of batc…
The focus of this paper is on linear, binary codes with locality having locality parameter $r$, that are capable of recovering from $t\geq 2$ erasures and that moreover, have short block length. Both sequential and parallel (through orthogonal parity checks) recovery is considered here. In the case of parallel repair, minimum-block-length constructions for general $t$ are discussed. In the case of sequential repair, the results include (a) extending and characterizing minimum-block-length constructions for $t=2$, (b) providing improved bounds on block length for $t=3$ as well as a general cons…
We discuss how subspace codes can be used to simultaneously correct errors and erasures when the network performs random linear network coding and the edges are noisy channels. This is done by combining the subspace code with a classical linear error-correcting code. The classical code then takes care of the errors and the subspace codes takes care of the erasures.
Distributed storage systems for large clusters typically use replication to provide reliability. Recently, erasure codes have been used to reduce the large storage overhead of three-replicated systems. Reed-Solomon codes are the standard design choice and their high repair cost is often considered an unavoidable price to pay for high storage efficiency and high reliability. This paper shows how to overcome this limitation. We present a novel family of erasure codes that are efficiently repairable and offer higher reliability compared to Reed-Solomon codes. We show analytically that our codes a…
This paper presents and analyzes a novel concatenated coding scheme for enabling error resilience in two distributed storage settings: one being storage using existing regenerating codes and the second being storage using locally repairable codes. The concatenated coding scheme brings together a maximum rank distance (MRD) code as an outer code and either a globally regenerating or a locally repairable code as an inner code. Also, error resilience for combination of locally repairable codes with regenerating codes is considered. This concatenated coding system is designed to handle two differe…
MDS array codes are widely used in storage systems due to their computationally efficient encoding and decoding procedures. An MDS code with $r$ redundancy nodes can correct any $r$ node erasures by accessing all the remaining information in the surviving nodes. However, in practice, $e$ erasures is a more likely failure event, for $1\le e<r$. Hence, a natural question is how much information do we need to access in order to rebuild $e$ storage nodes? We define the rebuilding ratio as the fraction of remaining information accessed during the rebuilding of $e$ erasures. In our previous work we …
