$ε$-Minimum Storage Regenerating ($ε$-MSR) codes form a special class of Maximum Distance Separable (MDS) codes, providing mechanisms for exact regeneration of a single code block in their codewords by downloading slighly sub-optimal amount of information from the remaining code blocks. The key advantage of these codes is a significantly lower sub-packetization that grows only logarithmically with the length of the code, while providing optimality in storage and error-correcting capacity. However, from an implementation point of view, these codes require each remaining code block to be available for the repair of any single code block. In this paper, we address this issue by constructing $ε$-MSR codes that can repair a failed code block by contacting a fewer number of available code blocks. When a code block fails, our repair procedure needs to contact a few compulsory code blocks and is free to choose any subset of available code blocks for the remaining choices. Further, our construction requiresa field size linear in code length and ensures load balancing among the contacted code blocks in terms of information downloaded from them for a single repair.
In this paper, we propose two new constructions of exact-repair minimum storage regenerating (exact-MSR) codes. For both constructions, the encoded symbols are obtained by treating the message vector over GF(q) as a linearized polynomial and evaluating it over an extension field GF(q^m). For our exact-MSR codes, data repair does not need matrix inversion, and can be implemented by additions and multiplications over GF$(q)$ as well as cyclic shifts when a normal basis is used. The two constructions assume a base field of GF(q) (q>2) and GF(2), respectively. In contrast to existing constructions…
In this paper distributed storage systems with exact repair are studied. A construction for regenerating codes between the minimum storage regenerating (MSR) and the minimum bandwidth regenerating (MBR) points is given. To the best of author's knowledge, no previous construction of exact-regenerating codes between MBR and MSR points is done except in the work by Tian et al. On contrast to their work, the methods used here are elementary. In this paper it is shown that in the case that the parameters $n$, $k$, and $d$ are close to each other, the given construction is close to optimal when comp…
In a distributed storage systems (DSS) with $k$ systematic nodes, robustness against node failure is commonly provided by storing redundancy in a number of other nodes and performing repair mechanism to reproduce the content of the failed nodes. Efficiency is then achieved by minimizing the storage overhead and the amount of data transmission required for data reconstruction and repair, provided by coding solutions such as regenerating codes [1]. Common explicit regenerating code constructions enable efficient repair through accessing a predefined number, $d$, of arbitrary chosen available nod…
Piggybacking is an efficient method to decrease the repair bandwidth of Maximum Distance Separable (MDS) codes or Minimum Storage Regenerating (MSR) codes. In this paper, for minimizing the repair bandwidth of parity nodes of the known MSR codes with high rate, which is usually the whole size of the original data, i.e., the maximal, a new systematic piggybacking design is proposed through an in-depth analysis of the design of piggybacking. As a result, new MSR codes are obtained with almost optimal repair bandwidth of parity nodes while retaining the optimal repair bandwidth of systematic node…
Cooperative repair model is an available technology to deal with multiple node failures in distributed storage systems. Recently, explicit constructions of cooperative MSR codes were given by Ye (IEEE Transactions on Information Theory, 2020) with sub-packetization level $(d-k+h)(d-k+1)^n$. Specifically, the sub-packetization level is $(h+1)2^n$ when $d=k+1$. In this paper, we propose a new cooperative repair scheme by means of the inter-instance and intra-instance pairing inherited from the perfect code which reduces the sub-packetization to $2^n$ when $(h+1)|2^n$ and $(2\ell+1)2^n$ when $h+1…
Clustered distributed storage models real data centers where intra- and cross-cluster repair bandwidths are different. In this paper, exact-repair minimum-storage-regenerating (MSR) codes achieving capacity of clustered distributed storage are designed. Focus is given on two cases: $ε=0$ and $ε=1/(n-k)$, where $ε$ is the ratio of the available cross- and intra-cluster repair bandwidths, $n$ is the total number of distributed nodes and $k$ is the number of contact nodes in data retrieval. The former represents the scenario where cross-cluster communication is not allowed, while the latter corre…
We present a high-rate $(n,k,d=n-1)$-MSR code with a sub-packetization level that is polynomial in the dimension $k$ of the code. While polynomial sub-packetization level was achieved earlier for vector MDS codes that repair systematic nodes optimally, no such MSR code construction is known. In the low-rate regime (i. e., rates less than one-half), MSR code constructions with a linear sub-packetization level are available. But in the high-rate regime (i. e., rates greater than one-half), the known MSR code constructions required a sub-packetization level that is exponential in $k$. In the pres…
The high repair cost of (n,k) Maximum Distance Separable (MDS) erasure codes has recently motivated a new class of codes, called Regenerating Codes, that optimally trade off storage cost for repair bandwidth. On one end of this spectrum of Regenerating Codes are Minimum Storage Regenerating (MSR) codes that can match the minimum storage cost of MDS codes while also significantly reducing repair bandwidth. In this paper, we describe Exact-MSR codes which allow for any failed nodes (whether they are systematic or parity nodes) to be regenerated exactly rather than only functionally or informatio…
This short note revisits the problem of designing secure minimum storage regenerating (MSR) codes for distributed storage systems. A secure MSR code ensures that a distributed storage system does not reveal the stored information to a passive eavesdropper. The eavesdropper is assumed to have access to the content stored on $\ell_1$ number of storage nodes in the system and the data downloaded during the bandwidth efficient repair of an additional $\ell_2$ number of storage nodes. This note combines the Gabidulin codes based precoding [18] and a new construction of MSR codes (without security r…
The minimum storage rack-aware regenerating (MSRR) code is a variation of regenerating codes that achieves the optimal repair bandwidth for a single node failure in the rack-aware model. The authors in~\cite{Chen-Barg2019} and~\cite{Zhou-Zhang2021} provided explicit constructions of MSRR codes for all parameters to repair a single failed node. This paper generalizes the results in~\cite{Chen-Barg2019} to the case of multiple node failures. We propose a class of MDS array codes and scalar Reed-Solomon (RS) codes, and show that these codes have optimal repair bandwidth and error resilient capabi…
This paper presents an explicit construction for an $((n,k,d=n-1), (α,β))$ regenerating code over a field $\mathbb{F}_Q$ operating at the Minimum Storage Regeneration (MSR) point. The MSR code can be constructed to have rate $k/n$ as close to $1$ as desired, sub-packetization given by $r^{\frac{n}{r}}$, for $r=(n-k)$, field size no larger than $n$ and where all code symbols can be repaired with the same minimum data download. The construction modifies a prior construction by Sasidharan et. al. which required far larger field-size. A building block appearing in the construction is a scalar MDS …
Minimum storage regenerating (MSR) codes are MDS codes which allow for recovery of any single erased symbol with optimal repair bandwidth, based on the smallest possible fraction of the contents downloaded from each of the other symbols. Recently, certain Reed-Solomon codes were constructed which are MSR. However, the sub-packetization of these codes is exponentially large, growing like $n^{Ω(n)}$ in the constant-rate regime. In this work, we study the relaxed notion of $ε$-MSR codes, which incur a factor of $(1+ε)$ higher than the optimal repair bandwidth, in the context of Reed-Solomon codes…
This paper presents a construction for high-rate MDS codes that enable bandwidth-efficient repair of a single node. Such MDS codes are also referred to as the minimum storage regenerating (MSR) codes in the distributed storage literature. The construction presented in this paper generates MSR codes for all possible number of helper nodes $d$ as $d$ is a design parameter in the construction. Furthermore, the obtained MSR codes have polynomial sub-packetization (a.k.a. node size) $α$. The construction is built on the recent code proposed by Sasidharan et al. [1], which works only for $d = n-1$, …
In this paper we study distributed storage systems with exact repair. We give a construction for regenerating codes between the minimum storage regenerating (MSR) and the minimum bandwidth regenerating (MBR) points and show that in the case that the parameters n, k, and d are close to each other our constructions are close to optimal when comparing to the known capacity when only functional repair is required. We do this by showing that when the distances of the parameters n, k, and d are fixed but the actual values approach to infinity, the fraction of the performance of our codes with exact …
Regenerating codes for distributed storage have attracted much research interest in the past decade. Such codes trade the bandwidth needed to repair a failed node with the overall amount of data stored in the network. Minimum storage regenerating (MSR) codes are an important class of optimal regenerating codes that minimize (first) the amount of data stored per node and (then) the repair bandwidth. Specifically, an $[n,k,d]$-$(α)$ MSR code $\mathbb{C}$ over $\mathbb{F}_q$ is defined as follows. Using such a code $\mathbb{C}$, a file $\cal{F}$ consisting of $αk$ symbols over $\mathbb{F}_q$ can …
In modern practical data centers, storage nodes are usually organized into equally sized groups, which is called racks. The cost of cross-rack communication is much more expensive compared with the intra-rack communication cost. The codes for this system are called rack-aware regenerating codes. Similar to standard minimum storage regenerating (MSR) codes, it is a challenging task to construct minimum storage rack-aware regenerating (MSRR) codes achieving the cut-set bound. The known constructions of MSRR codes achieving the cut-set bound give codes with alphabet size $q$ exponential in the co…
We consider the problem of multiple-node repair in distributed storage systems under the cooperative model, where the repair bandwidth includes the amount of data exchanged between any two different storage nodes. Recently, explicit constructions of MDS codes with optimal cooperative repair bandwidth for all possible parameters were given by Ye and Barg (IEEE Transactions on Information Theory, 2019). The node size (or sub-packetization) in this construction scales as $\exp(Θ(n^h))$, where $h$ is the number of failed nodes and $n$ is the code length. In this paper, we give new explicit constru…
Reducing The Sub-packetization Level of Optimal-Access Cooperative MSR Codes
Ministry of Education Key Laboratory of Mathematics and Information Networks, School of Mathematical Sciences, Beijing University of Posts and Telecommunications（北京邮电大学数学与信息技术教育部重点实验室）；School of Information Science and Engineering, Shandong Agricultural University（山东农业大学信息科学与工程学院）
Recently, the research on local repair codes is mainly confined to repair the failed nodes within each repair group. But if the extreme cases occur that the entire repair group has failed, the local code stored in the failed group need to be recovered as a whole. In this paper, local codes with cooperative repair, in which the local codes are constructed based on minimum storage regeneration (MSR) codes, is proposed to achieve repairing the failed groups. Specifically, the proposed local codes with cooperative repair construct a kind of mutual interleaving structure among the parity symbols, t…
The paper is devoted to the problem of erasure coding in distributed storage. We consider a model of storage that assumes that nodes are organized into equally sized groups, called racks, that within each group the nodes can communicate freely without taxing the system bandwidth, and that the only information transmission that counts is the one between the racks. This assumption implies that the nodes within each of the racks can collaborate before providing information to the failed node. The main emphasis of the paper is on code construction for this storage model. We present an explicit fam…
In this paper, we provide explicit constructions for a class of exact-repair regenerating codes that possess a layered structure. These regenerating codes correspond to interior points on the storage-repair-bandwidth tradeoff, and compare very well in comparison to scheme that employs space-sharing between MSR and MBR codes. For the parameter set $(n,k,d=k)$ with $n < 2k-1$, we construct a class of codes with an auxiliary parameter $w$, referred to as canonical codes. With $w$ in the range $n-k < w < k$, these codes operate in the region between the MSR point and the MBR point, and perform sig…
Multi-message private information retrieval (MPIR) is an interesting variation of PIR which allows a user to download multiple messages from the database without revealing the identity of the desired messages. Obviously, the user can repeatly use a single-message PIR scheme, but we wish for a more efficient way to reduce the download cost. In [1], Banawan and Ukulus investigate the multi-message PIR problem with replicated database. In this paper, we consider multi-message PIR schemes where the database is stored using minimum storage regenerating (MSR) or minimum bandwidth regenerating (MBR) …
The newly presented $(k+2,k)$ Hadamard minimum storage regenerating (MSR) code is the first class of high rate storage code with optimal repair property for all single node failures. In this paper, we propose a new simple repair strategy, which can considerably reduces the computation load of the node repair in contrast to the original one.
In large scale distributed storage systems (DSS) deployed in cloud computing, correlated failures resulting in simultaneous failure (or, unavailability) of blocks of nodes are common. In such scenarios, the stored data or a content of a failed node can only be reconstructed from the available live nodes belonging to the available blocks. To analyze the resilience of the system against such block failures, this work introduces the framework of Block Failure Resilient (BFR) codes, wherein the data (e.g., a file in DSS) can be decoded by reading out from a same number of codeword symbols (nodes) …
Private information retrieval (PIR) is the problem of privately retrieving one out of $M$ original files from $N$ severs, i.e., each individual server learns nothing about the file that the user is requesting. Usually, the $M$ files are replicated or encoded by a maximum distance separable (MDS) code and then stored across the $N$ servers. Compared to mere replication, MDS coded servers can significantly reduce the storage overhead. Particularly, PIR from minimum storage regenerating (MSR) coded servers can simultaneously reduce the repair bandwidth when repairing failed servers. Existing PIR …
Decoding and repair schemes are proposed for shift-exclusive-or (shift-XOR) product-matrix (PM) regenerating codes, which outperform the existing schemes in terms of both communication and computation costs. In particular, for the shift-XOR minimum bandwidth regenerating (MBR) codes, our decoding and repair schemes have the optimal transmission bandwidth and can be implemented in-place without extra storage space for intermediate XOR results. Technically, our schemes involve an in-place algorithm for solving a system of shift-XOR equations, called \emph{shift-XOR elimination}, which does not h…
We study the exact and optimal repair of multiple failures in codes for distributed storage. More particularly, we provide an explicit construction of exact minimum bandwidth coordinated regenerating codes (MBCR) for n=d+t,k,d >= k,t >= 1. Our construction differs from existing constructions by allowing both t>1 (i.e., repair of multiple failures) and d>k (i.e., contacting more than k devices during repair).
Private Information Retrieval (PIR) schemes allow a user to retrieve a record from the server without revealing any information on which record is being downloaded. In this paper, we consider PIR schemes where the database is stored using Minimum Storage Regenerating (MSR) codes which is a class of optimal regenerating codes providing efficient repair when a node failure occurs in the system. We analyse the relationship between the costs of privacy, storage and repair, and also construct an explicit PIR scheme that uses the MSR codes from [3] to achieve the optimal curve of the trade-off.
Reed-Solomon (RS) codes are widely used in distributed storage systems. In this paper, we study the repair bandwidth and sub-packetization size of RS codes. The repair bandwidth is defined as the amount of transmitted information from surviving nodes to a failed node. The RS code can be viewed as a polynomial over a finite field $GF(q^\ell)$ evaluated at a set of points, where $\ell$ is called the sub-packetization size. Smaller bandwidth reduces the network traffic in distributed storage, and smaller $\ell$ facilitates the implementation of RS codes with lower complexity. Recently, Guruswami …
We present two lower bounds on sub-packetization level $α$ of MSR codes with parameters $(n, k, d=n-1, α)$ where $n$ is the block length, $k$ dimension, $d$ number of helper nodes contacted during single node repair and $α$ the sub-packetization level. The first bound we present is for any MSR code and is given by $α\ge e^{\frac{(k-1)(r-1)}{2r^2}}$. The second bound we present is for the case of optimal-access MSR codes and the bound is given by $α\ge \min \{ r^{\frac{n-1}{r}}, r^{k-1} \}$. There exist optimal-access MSR constructions that achieve the second sub-packetization level bound with …
Distributed storage codes have important applications in the design of modern storage systems. In a distributed storage system, every storage node has a probability to fail and once an individual storage node fails, it must be reconstructed using data stored in the surviving nodes. Computation load and network bandwidth are two important issues we need to concern when repairing a failed node. The traditional maximal distance separable (MDS) storage codes have low repair complexity but high repair bandwidth. On the contrary, minimal storage regenerating (MSR) codes have low repair bandwidth but…
This paper considers capacity-achieving coding for the clustered form of distributed storage that reflects practical storage networks. To reflect the clustered structure with limited cross-cluster communication bandwidths, nodes in the same cluster are set to communicate $β_I$ symbols, while nodes in other clusters can communicate $β_c \leq β_I$ symbols with one another. We provide two types of exact regenerating codes which achieve the capacity of clustered distributed storage: the minimum-bandwidth-regenerating (MBR) codes and the minimum-storage-regenerating (MSR) codes. First, we construct…
In this paper, we revisit the problem of characterizing the secrecy capacity of minimum storage regenerating (MSR) codes under the passive $(l_1,l_2)$-eavesdropper model, where the eavesdropper has access to data stored on $l_1$ nodes and the repair data for an additional $l_2$ nodes. We study it from the information-theoretic perspective. First, some general properties of MSR codes as well as a simple and generally applicable upper bound on secrecy capacity are given. Second, a new concept of \emph{stable} MSR codes is introduced, where the stable property is shown to be closely linked with s…
This paper studies the design of codes for distributed storage systems (DSS) that enable local repair in the event of node failure. This paper presents locally repairable codes based on low degree multivariate polynomials. Its code construction mechanism extends work on Noisy Interpolating Set by Dvir et al. \cite{dvir2011}. The paper presents two classes of codes that allow node repair to be performed by contacting 2 and 3 surviving nodes respectively. It further shows that both classes are good in terms of their rate and minimum distance, and allow their rate to be bartered for greater flexi…
Key Laboratory of Cryptologic Technology and Information Security, Ministry of Education（教育部密码技术与信息安全重点实验室）；School of Cyber Science and Technology, Shandong University（山东大学网络空间安全学院）；Quan Cheng Laboratory（泉城实验室）
Data storage in Distributed Storage Systems (DSSs) is a multidimensional optimization problem. Using network coding, one wants to provide reliability, scalability, security, reduced storage overhead, reduced bandwidth for repair and minimal disk I/O etc. in such systems. Regenerating codes have been used to optimize some of these parameters, where a file can be reconstructed by contacting any k nodes in the system and in case of node failure it can be repaired by using any d nodes in the system. This was further generalized to Fractional repetition (FR) codes (a smart replication of encoded pa…
Regenerating codes based on the approach of interference alignment for wireless interference channel achieve the cut-set bound for distributed storage systems. These codes provide data reliability, and perform efficient exact node repair when some node fails. Interference alignment as a concept is especially important to improve the repair efficiency of a failed node in a minimum storage regenerating (MSR) code. In addition it can improve the stored data security in presence of passive intruders. In this paper we construct a new code resilient against a threat model where a passive eavesdroppe…
The distributed data storage systems are constructed by large number of nodes which are interconnected over a network. Each node in such peer-to-peer network is vulnerable and at a potential risk for attack. The attackers can eavesdrop the nodes and possibly modify their data. Hence distributed storage systems should be secure apart from satisfying the reconstruction and repair requirements. We constructed a distributed storage system, Twin MDS code framework which is more efficient than the regenerating codes based storage systems. We prove that this Twin MDS code framework gives better perfo…
Minimum-Storage Regenerating (MSR) codes have emerged as a viable alternative to Reed-Solomon (RS) codes as they minimize the repair bandwidth while they are still optimal in terms of reliability and storage overhead. Although several MSR constructions exist, so far they have not been practically implemented mainly due to the big number of I/O operations. In this paper, we analyze high-rate MDS codes that are simultaneously optimized in terms of storage, reliability, I/O operations, and repair-bandwidth for single and multiple failures of the systematic nodes. The codes were recently introduce…
Regenerating codes are a class of distributed storage codes that optimally trade the bandwidth needed for repair of a failed node with the amount of data stored per node of the network. Minimum Storage Regenerating (MSR) codes minimize first, the amount of data stored per node, and then the repair bandwidth, while Minimum Bandwidth Regenerating (MBR) codes carry out the minimization in the reverse order. An [n, k, d] regenerating code permits the data to be recovered by connecting to any k of the n nodes in the network, while requiring that repair of a failed node be made possible by connectin…
We consider a set up where a file of size M is stored in n distributed storage nodes, using an (n,k) minimum storage regenerating (MSR) code, i.e., a maximum distance separable (MDS) code that also allows efficient exact-repair of any failed node. The problem of interest in this paper is to minimize the repair bandwidth B for exact regeneration of a single failed node, i.e., the minimum data to be downloaded by a new node to replace the failed node by its exact replica. Previous work has shown that a bandwidth of B=[M(n-1)]/[k(n-k)] is necessary and sufficient for functional (not exact) regene…
Regenerating codes are a class of recently developed codes for distributed storage that, like Reed-Solomon codes, permit data recovery from any arbitrary k of n nodes. However regenerating codes possess in addition, the ability to repair a failed node by connecting to any arbitrary d nodes and downloading an amount of data that is typically far less than the size of the data file. This amount of download is termed the repair bandwidth. Minimum storage regenerating (MSR) codes are a subclass of regenerating codes that require the least amount of network storage; every such code is a maximum dis…
An explicit construction of systematic MDS codes, called HashTag+ codes, with arbitrary sub-packetization level for all-node repair is proposed. It is shown that even for small sub-packetization levels, HashTag+ codes achieve the optimal MSR point for repair of any parity node, while the repair bandwidth for a single systematic node depends on the sub-packetization level. Compared to other codes in the literature, HashTag+ codes provide from 20% to 40% savings in the average amount of data accessed and transferred during repair.
Optimal-Access Cooperative MSR Codes: Parity-Check Matrix Construction And a Unified Transformation
Ministry of Education Key Laboratory of Mathematics and Information Networks, School of Mathematical Sciences, Beijing University of Posts and Telecommunications（北京邮电大学数学与信息技术教育部重点实验室）；School of Information Science and Engineering, Shandong Agricultural University（山东农业大学信息科学与工程学院）
We present a precise characterization of linear functional-repair storage codes in terms of {\em admissible states/}, with each state made up from a collection of vector spaces over some fixed finite field. To illustrate the usefulness of our characterization, we provide several applications. We first describe a simple construction of functional-repair storage codes for a family of code parameters meeting the cutset bound outside the MBR and MSR points; these codes are conjectured to have optimal rate with respect to their repair locality. Then, we employ our characterization to develop a cons…
In large scale distributed storage systems (DSS) deployed in cloud computing, correlated failures resulting in simultaneous failure (or, unavailability) of blocks of nodes are common. In such scenarios, the stored data or a content of a failed node can only be reconstructed from the available live nodes belonging to available blocks. To analyze the resilience of the system against such block failures, this work introduces the framework of Block Failure Resilient (BFR) codes, wherein the data (e.g., file in DSS) can be decoded by reading out from a same number of codeword symbols (nodes) from e…
