Minimum cost distributed source coding over a network
Aditya Ramamoorthy
备注：
First version apppeared in the Proceedings of the 2007 IEEE International Symposium on Information Theory, Nice, France, June 24 - 29, 2007. The second version is an expanded journal submission under consideration at the IEEE Transactions on Information Theory
This work considers the problem of transmitting multiple compressible sources over a network at minimum cost. The aim is to find the optimal rates at which the sources should be compressed and the network flows using which they should be transmitted so that the cost of the transmission is minimal. We consider networks with capacity constraints and linear cost functions. The problem is complicated by the fact that the description of the feasible rate region of distributed source coding problems typically has a number of constraints that is exponential in the number of sources. This renders general purpose solvers inefficient. We present a framework in which these problems can be solved efficiently by exploiting the structure of the feasible rate regions coupled with dual decomposition and optimization techniques such as the subgradient method and the proximal bundle method.
The distributed index coding problem is studied, whereby multiple messages are stored at different servers to be broadcast to receivers with side information. First, the existing composite coding scheme is enhanced for the centralized (single-server) index coding problem, which is then merged with fractional partitioning of servers to yield a new coding scheme for distributed index coding. New outer bounds on the capacity region are also established. For 213 out of 218 non-isomorphic distributed index coding problems with four messages the achievable sum-rate of the proposed distributed compos…
We consider a distributed source coding system in which several observations are communicated to the decoder using limited transmission rate. The observations must be separately coded. We introduce a robust distributed coding scheme which flexibly trades off between system robustness and compression efficiency. The optimality of this coding scheme is proved for various special cases.
A novel distributed space-time block code (STBC) for the next generation mobile TV broadcasting is proposed. The new code provides efficient performance within a wide range of power imbalance showing strong adaptivity to the single frequency network (SFN) broadcasting deployments. The new code outperforms existing STBCs with equivalent decoding complexity and approaches those with much higher complexities.
Minimum energy (ME) source coding is an effective technique for efficient communication with energy-constrained devices, such as sensor network nodes. In this paper, the principles of generalized ME source coding is developed that is broadly applicable. Two scenarios - fixed and variable length codewords - are analyzed. The application of this technique to RFID systems where ME source coding is particularly advantageous due to the asymmetric nature of data communications is demonstrated, a first to the best of our knowledge.
In this paper, we study the capacity region of the general distributed index coding. In contrast to the traditional centralized index coding where a single server contains all $n$ messages requested by the receivers, in the distributed index coding there are $2^n-1$ servers, each containing a unique non-empty subset $J$ of the messages and each is connected to all receivers via a noiseless independent broadcast link with an arbitrary capacity $C_J \ge 0$. First, we generalize the existing polymatroidal outer bound on the capacity region of the centralized problem to the distributed case. Next,…
Distributed arithmetic coding (DAC) has been shown to be effective for Slepian-Wolf coding, especially for short data blocks. In this letter, we propose to use the DAC to compress momery-correlated sources. More specifically, the correlation between sources is modeled as a hidden Markov process. Experimental results show that the performance is close to the theoretical Slepian-Wolf limit.
Index coding is often studied with the assumption that a single source has all the messages requested by the receivers. We refer to this as \emph{centralized} index coding. In contrast, this paper focuses on \emph{distributed} index coding and addresses the following question: How does the availability of messages at distributed sources (storage nodes) affect the solutions and achievable rates of index coding? An extension to the work of Arbabjolfaei et al. in ISIT 2013 is presented when distributed sources communicate via a semi-deterministic multiple access channel (MAC) to simultaneous rece…
In this paper, we give a distributed joint source channel coding scheme for arbitrary correlated sources for arbitrary point in the Slepian-Wolf rate region, and arbitrary link capacities using LDPC codes. We consider the Slepian-Wolf setting of two sources and one destination, with one of the sources derived from the other source by some correlation model known at the decoder. Distributed encoding and separate decoding is used for the two sources. We also give a distributed source coding scheme when the source correlation has memory to achieve any point in the Slepian-Wolf rate achievable reg…
A network coding-based scheme is proposed to improve the energy efficiency of distributed storage systems in WSNs (wireless sensor networks), which mainly focuses on two problems: firstly, consideration is given to effective distributed storage technology in WSNs; secondly, we address how to repair the data in failed storage nodes with less resource. For the first problem, we propose a method to obtain a sparse generator matrix to construct network codes, and this sparse generator matrix is proven to be the sparsest. Benefiting from the sparse generator matrix, the energy consumption required …
We show that polar codes can be used to achieve the rate-distortion functions in the problem of hierarchical source coding also known as the successive refinement problem. We also analyze the distributed version of this problem, constructing a polar coding scheme that achieves the rate distortion functions for successive refinement with side information.
A subspace-based approach for rate-adaptive distributed source coding (DSC) based on discrete Fourier transform (DFT) codes is developed. Punctured DFT codes can be used to implement rate-adaptive source coding, however they perform poorly after even moderate puncturing since the performance of the subspace error localization degrades severely. The proposed subspace-based error localization extends and improves the existing one, based on additional syndrome, and is naturally suitable for rate-adaptive distributed source coding architecture.
Traditional distributed source coding rarely considers the possible link between separate encoders. However, the broadcast nature of wireless communication in sensor networks provides a free gossip mechanism which can be used to simplify encoding/decoding and reduce transmission power. Using this broadcast advantage, we present a new two-encoder scheme which imitates the ping-pong game and has a successive approximation structure. For the quadratic Gaussian case, we prove that this scheme is successively refinable on the {sum-rate, distortion pair} surface, which is characterized by the rate-d…
Distributed source coding schemes are typically based on the use of channels codes as source codes. In this paper we propose a new paradigm, termed "distributed arithmetic coding", which exploits the fact that arithmetic codes are good source as well as channel codes. In particular, we propose a distributed binary arithmetic coder for Slepian-Wolf coding with decoder side information, along with a soft joint decoder. The proposed scheme provides several advantages over existing Slepian-Wolf coders, especially its good performance at small block lengths, and the ability to incorporate arbitrary…
In distributed multilevel diversity coding, $K$ correlated sources (each with $K$ components) are encoded in a distributed manner such that, given the outputs from any $α$ encoders, the decoder can reconstruct the first $α$ components of each of the corresponding $α$ sources. For this problem, the optimality of a multilayer Slepian-Wolf coding scheme based on binning and superposition is established when $K\leq 3$. The same conclusion is shown to hold for general $K$ under a certain symmetry condition, which generalizes a celebrated result by Yeung and Zhang.
A distortion-noise profile is a function indicating the maximum allowed source distortion value for each noise level in the channel. In this paper, the minimum energy required to achieve a distortion noise profile is studied for Gaussian sources which are transmitted robustly over Gaussian channels. We provide improved lower and upper bounds for the minimum energy behavior of the square-law profile using a family of lower bounds and our proposed coding scheme.
We introduce a distributed source coding scheme called successive Wyner-Ziv coding. We show that any point in the rate region of the quadratic Gaussian CEO problem can be achieved via the successive Wyner-Ziv coding. The concept of successive refinement in the single source coding is generalized to the distributed source coding scenario, which we refer to as distributed successive refinement. For the quadratic Gaussian CEO problem, we establish a necessary and sufficient condition for distributed successive refinement, where the successive Wyner-Ziv coding scheme plays an important role.
In this paper, we introduce a model of a single-failure locally recoverable distributed storage system. This model appears to give rise to a problem seemingly dual of the well-studied index coding problem. The relation between the dimensions of an optimal index code and optimal distributed storage code of our model has been established in this paper. We also show some extensions to vector codes.
In this paper we propose a new cooperative packet transmission scheme that allows independent sources to superimpose over-the-air their packet transmissions. Relay nodes are used and cooperative diversity is combined with distributed space-time block coding (STBC). With the proposed scheme the participating relays create a ST code for the over-the-air superimposed symbols that are received locally and without proceeding to any decoding step beforehand. The advantage of the proposed scheme is that communication is completed in fewer transmission slots because of the concurrent packet transmissi…
In this paper, we propose a lattice-based robust distributed source coding system for two correlated sources and provide a detailed performance analysis under the high resolution assumption. It is shown, among other things, that, in the asymptotic regime where 1) the side distortion approaches 0 and 2) the ratio between the central and side distortions approaches 0, our scheme is capable of achieving the information-theoretic limit of quadratic multiple description coding when the two sources are identical, whereas a variant of the random coding scheme by Chen and Berger with Gaussian codes ha…
In this paper, we consider general two-way relay networks (TWRNs) with two source and N relay nodes. A distributed differential space time coding with analog network coding (DDSTC-ANC) scheme is proposed. A simple blind estimation and a differential signal detector are developed to recover the desired signal at each source. The pairwise error probability (PEP) and block error rate (BLER) of the DDSTC-ANC scheme are analyzed. Exact and simplified PEP expressions are derived. To improve the system performance, the optimum power allocation (OPA) between the source and relay nodes is determined ba…
This paper investigates the distributed space-time (ST) coding proposals for the future Digital Video Broadcasting--Next Generation Handheld (DVB-NGH) standard. We first theoretically show that the distributed MIMO scheme is the best broadcasting scenario in terms of channel capacity. Consequently we evaluate the performance of several ST coding proposals for DVB-NGH with practical system specifications and channel conditions. Simulation results demonstrate that the 3D code is the best ST coding solution for broadcasting in the distributed MIMO scenario.
Motivated by applications in distributed storage and distributed computation, we introduce embedded index coding (EIC). EIC is a type of distributed index coding in which nodes in a distributed system act as both senders and receivers of information. We show how embedded index coding is related to index coding in general, and give characterizations and bounds on the communication costs of optimal embedded index codes. We also define task-based EIC, in which each sending node encodes and sends data blocks independently of the other nodes. Task-based EIC is more computationally tractable and has…
In this paper we provide sufficient conditions for lossy transmission of functions of correlated data over a multiple access channel (MAC). The conditions obtained can be shown as generalized version of Yamamoto's result. We also obtain efficient joint source-channel coding schemes for transmission of discrete and continuous alphabet sources to recover the function values. Keywords: Joint source-channel coding, Graph coloring, Lipschitz functions, Correlated sources.
The advent of network coding presents promising opportunities in many areas of communication and networking. It has been recently shown that network coding technique can significantly increase the overall throughput of wireless networks by taking advantage of their broadcast nature. In wireless networks, each transmitted packet is broadcasted within a certain area and can be overheard by the neighboring nodes. When a node needs to transmit packets, it employs the opportunistic coding approach that uses the knowledge of what the node's neighbors have heard in order to reduce the number of trans…
This paper studies the optimization of zero-delay analog mappings in a network setting that involves distributed coding. The cost surface is known to be non-convex, and known greedy methods tend to get trapped in poor locally optimal solutions that depend heavily on initialization. We derive an optimization algorithm based on the principles of "deterministic annealing", a powerful global optimization framework that has been successfully employed in several disciplines, including, in our recent work, to a simple zero-delay analog communications problem. We demonstrate strict superiority over th…
In this work, we propose a distributed rate allocation algorithm that minimizes the average decoding delay for multimedia clients in inter-session network coding systems. We consider a scenario where the users are organized in a mesh network and each user requests the content of one of the available sources. We propose a novel distributed algorithm where network users determine the coding operations and the packet rates to be requested from the parent nodes, such that the decoding delay is minimized for all the clients. A rate allocation problem is solved by every user, which seeks the rates t…
In index coding, a server broadcasts multiple messages to their respective receivers, each with some side information that can be utilized to reduce the amount of communication from the server. Distributed index coding is an extension of index coding in which the messages are broadcast from multiple servers, each storing different subsets of the messages. In this paper, the optimal tradeoff among the message rates and the server broadcast rates, which is defined formally as the capacity region, is studied for a general distributed index coding problem. Inner and outer bounds on the capacity re…
Unitary space-time modulation is known to be an efficient means to communicate over non-coherent Multiple Input Multiple Output (MIMO) channels. In this letter, differential unitary space-time coding and non-coherent space-time coding for the training based approach of Kim and Tarokh are addressed. For this approach, necessary and sufficient conditions for multi-group decodability are derived in a simple way assuming a Generalized Likelihood Ratio Test receiver and a unitary codebook. Extending Kim and Tarokh's approach for colocated MIMO systems, a novel training based approach to distributed…
The distributed source coding problem is considered when the sensors, or encoders, are under Byzantine attack; that is, an unknown number of sensors have been reprogrammed by a malicious intruder to undermine the reconstruction at the fusion center. Three different forms of the problem are considered. The first is a variable-rate setup, in which the decoder adaptively chooses the rates at which the sensors transmit. An explicit characterization of the variable-rate minimum achievable sum rate is stated, given by the maximum entropy over the set of distributions indistinguishable from the true …
We show how real-number codes can be used to compress correlated sources, and establish a new framework for lossy distributed source coding, in which we quantize compressed sources instead of compressing quantized sources. This change in the order of binning and quantization blocks makes it possible to model correlation between continuous-valued sources more realistically and correct quantization error when the sources are completely correlated. The encoding and decoding procedures are described in detail, for discrete Fourier transform (DFT) codes. Reconstructed signal, in the mean squared er…
We obtain novel index coding schemes and show that they provably outperform all previously known graph theoretic bounds proposed so far. Further, we establish a rather strong negative result: all known graph theoretic bounds are within a logarithmic factor from the chromatic number. This is in striking contrast to minrank since prior work has shown that it can outperform the chromatic number by a polynomial factor in some cases. The conclusion is that all known graph theoretic bounds are not much stronger than the chromatic number.
Low-density parity-check (LDPC) codes with the parity-based approach for distributed joint source channel coding (DJSCC) with decoder side information is described in this paper. The parity-based approach is theoretical limit achievable. Different edge degree distributions are used for source variable nodes and parity variable nodes. Particularly, the codeword-averaged density evolution (CADE) is presented for asymmetrically correlated nonuniform sources over the asymmetric memoryless transmission channel. Extensive simulations show that the splitting of variable nodes can improve the coding e…
Based on the impressive features that network coding and compressed sensing paradigms have separately brought, the idea of bringing them together in practice will result in major improvements and influence in the upcoming 5G networks. In this context, this paper aims to evaluate the effectiveness of these key techniques in a cluster-based wireless sensor network, in the presence of temporal and spatial correlations. Our goal is to achieve better compression gains by scaling down the total payload carried by applying temporal compression as well as reducing the total number of transmissions in …
Minimum energy required to achieve a distortion-noise profile, i.e., a function indicating the maximum allowed distortion value for each channel noise level. In this paper, the minimum energy required to achieve a distortion noise profile is studied for Gaussian sources which are transmitted robustly over Gaussian channels. We provide upper bound for the minimum energy behavior of the staircase profile using our proposed coding scheme. Conversely, utilizing a family of lower bounds originally derived for broadcast channels with power constraints, the minimum required energy is lower bounded fo…
We examine an error-correcting coding framework in which each coded symbol is constrained to be a function of a fixed subset of the message symbols. With an eye toward distributed storage applications, we seek to design systematic codes with good minimum distance that can be decoded efficiently. On this note, we provide theoretical bounds on the minimum distance of such a code based on the coded symbol constraints. We refine these bounds in the case where we demand a systematic linear code. Finally, we provide conditions under which each of these bounds can be achieved by choosing our code to …
In the distributed coding of correlated sources, the problem of characterizing the joint probability distribution of a pair of random variables satisfying an n-letter Markov chain arises. The exact solution of this problem is intractable. In this paper, we seek a single-letter necessary condition for this n-letter Markov chain. To this end, we propose a new data processing inequality on a new measure of correlation by means of spectrum analysis. Based on this new data processing inequality, we provide a single-letter necessary condition for the required joint probability distribution. We apply…
Superdense coding proved that entanglement-assisted quantum communications can improve the data transmission rates compared to classical systems. It allows sending 2 classical bits between the parties in exchange of 1 quantum bit and a pre-shared entangled Bell pair. This paper introduces a new protocol which is intended for distributed communication. Using a pre-shared entangled Bell pair and 1 classical bit 2,5 classical bits can be transmitted in average. This means not only valuable increase in capacity but the two-way distributed operation opens new fields of investigation.
Coded distributed computing was recently introduced to mitigate the effect of stragglers on distributed computing. This paper combines ideas of approximate computing with coded computing to further accelerate computation. We propose successive approximation coding (SAC) techniques that realize a tradeoff between accuracy and speed, allowing the distributed computing system to produce approximations that increase in accuracy over time. If a sufficient number of compute nodes finish their tasks, SAC exactly recovers the desired computation. We theoretically provide design guidelines for our SAC …
Content distribution over networks is often achieved by using mirror sites that hold copies of files or portions thereof to avoid congestion and delay issues arising from excessive demands to a single location. Accordingly, there are distributed storage solutions that divide the file into pieces and place copies of the pieces (replication) or coded versions of the pieces (coding) at multiple source nodes. We consider a network which uses network coding for multicasting the file. There is a set of source nodes that contains either subsets or coded versions of the pieces of the file. The cost of…
We consider distributed gradient descent in the presence of stragglers. Recent work on \em gradient coding \em and \em approximate gradient coding \em have shown how to add redundancy in distributed gradient descent to guarantee convergence even if some workers are \em stragglers\em---that is, slow or non-responsive. In this work we propose an approximate gradient coding scheme called \em Stochastic Gradient Coding \em (SGC), which works when the stragglers are random. SGC distributes data points redundantly to workers according to a pair-wise balanced design, and then simply ignores the strag…
This paper considers the problem of minimum cost communication of correlated sources over a network with multiple sinks, which consists of distributed source coding followed by routing. We introduce a new routing paradigm called dispersive information routing, wherein the intermediate nodes are allowed to `split' a packet and forward subsets of the received bits on each of the forward paths. This paradigm opens up a rich class of research problems which focus on the interplay between encoding and routing in a network. Unlike conventional routing methods such as in [1], dispersive information r…
We devise an analytically simple as well as invertible approximate expression, which describes the relation between the minimum distance of a binary code and the corresponding maximum attainable code-rate. For example, for a rate-(1/4), length-256 binary code the best known bounds limit the attainable minimum distance to 65<d(n=256,k=64)<90, while our solution yields d(n=256,k=64)=74.4. The proposed formula attains the approximation accuracy within the rounding error for ~97% of (n,k) scenarios, where the exact value of the minimum distance is known. The results provided may be utilized for th…
Distributed source coding is traditionally viewed in the block coding context -- all the source symbols are known in advance at the encoders. This paper instead considers a streaming setting in which iid source symbol pairs are revealed to the separate encoders in real time and need to be reconstructed at the decoder with some tolerable end-to-end delay using finite rate noiseless channels. A sequential random binning argument is used to derive a lower bound on the error exponent with delay and show that both ML decoding and universal decoding achieve the same positive error exponents inside t…
Network coding is a highly efficient data dissemination mechanism for wireless networks. Since network coded information can only be recovered after delivering a sufficient number of coded packets, the resulting decoding delay can become problematic for delay-sensitive applications such as real-time media streaming. Motivated by this observation, we consider several algorithms that minimize the decoding delay and analyze their performance by means of simulation. The algorithms differ both in the required information about the state of the neighbors' buffers and in the way this knowledge is use…
Three science and engineering problems of recent interests -index coding, locally recoverable distributed storage, and guessing games on graphs- are discussed and the connection between their optimal solutions is elucidated. By generalizing recent results by Shanmugam and Dimakis and by Mazumdar on the complementarity between the optimal broadcast rate of an index coding problem on a directed graph and the normalized rate of a locally recoverable distributed storage problem on the same graph, it is shown that the capacity region and the optimal rate region of these two problems are complementa…
In this paper, we propose a new distributed coding structure with a soft input soft output (SISO) relay encoder for error-prone parallel relay channels. We refer to it as the distributed soft coding (DISC). In the proposed scheme, each relay first uses the received noisy signals to calculate the soft bit estimate (SBE) of the source symbols. A simple SISO encoder is developed to encode the SBEs of source symbols based on a constituent code generator matrix. The SISO encoder outputs at different relays are then forwarded to the destination and form a distributed codeword. The performance of the…
We consider the problem of transmission of several distributed sources over a multiple access channel (MAC) with side information at the sources and the decoder. Source-channel separation does not hold for this channel. Sufficient conditions are provided for transmission of sources with a given distortion. The source and/or the channel could have continuous alphabets (thus Gaussian sources and Gaussian MACs are special cases). Various previous results are obtained as special cases. We also provide several good joint source-channel coding schemes for a discrete/continuous source and discrete/co…
Consider a mobile edge computing system in which users wish to obtain the result of a linear inference operation on locally measured input data. Unlike the offloaded input data, the model weight matrix is distributed across wireless Edge Nodes (ENs). ENs have non-deterministic computing times, and they can transmit any shared computed output back to the users cooperatively. This letter investigates the potential advantages obtained by coding model information prior to ENs' storage. Through an information-theoretic analysis, it is concluded that, while generally limiting cooperation opportuniti…
We consider a distributed source coding problem of $L$ correlated Gaussian observations $Y_i, i=1,2,...,L$. We assume that the random vector $Y^{L}={}^{\rm t} (Y_1,Y_2,$ $...,Y_L)$ is an observation of the Gaussian random vector $X^K={}^{\rm t}(X_1,X_2,...,X_K)$, having the form $Y^L=AX^K+N^L ,$ where $A$ is a $L\times K$ matrix and $N^L={}^{\rm t}(N_1,N_2,...,N_L)$ is a vector of $L$ independent Gaussian random variables also independent of $X^K$. The estimation error on $X^K$ is measured by the distortion covariance matrix. The rate distortion region is defined by a set of all rate vectors f…
The current BigData era routinely requires the processing of large scale data on massive distributed computing clusters. Such large scale clusters often suffer from the problem of "stragglers", which are defined as slow or failed nodes. The overall speed of a computational job on these clusters is typically dominated by stragglers in the absence of a sophisticated assignment of tasks to the worker nodes. In recent years, approaches based on coding theory (referred to as "coded computation") have been effectively used for straggler mitigation. Coded computation offers significant benefits for s…
We consider the problem of data collection in a two-layer network consisting of (1) links between $N$ distributed agents and a remote sink node; (2) a sparse network formed by these distributed agents. We study the effect of inter-agent communications on the overall energy consumption. Despite the sparse connections between agents, we provide an in-network coding scheme that reduces the overall energy consumption by a factor of $Θ(\log N)$ compared to a naive scheme which neglects inter-agent communications. By providing lower bounds on both the energy consumption and the sparseness (number of…
Differential distributed space-time coding (D-DSTC) has been considered to improve both diversity and data-rate in cooperative communications in the absence of channel information. However, conventionally, it is assumed that relays are perfectly synchronized in the symbol level. In practice, this assumption is easily violated due to the distributed nature of the relay networks. This paper proposes a new differential encoding and decoding process for D-DSTC systems with two relays. The proposed method is robust against synchronization errors and does not require any channel information at the d…
Peer-to-peer distributed storage systems provide reliable access to data through redundancy spread over nodes across the Internet. A key goal is to minimize the amount of bandwidth used to maintain that redundancy. Storing a file using an erasure code, in fragments spread across nodes, promises to require less redundancy and hence less maintenance bandwidth than simple replication to provide the same level of reliability. However, since fragments must be periodically replaced as nodes fail, a key question is how to generate a new fragment in a distributed way while transferring as little data …
An adaptive delay-tolerant distributed space-time coding (DSTC) scheme that exploits feedback is proposed for two-hop cooperative MIMO networks. Maximum likelihood (ML) receivers and adjustable code matrices are considered subject to a power constraint with a decode-and-forward (DF) cooperation strategy. In the proposed delay-tolerant DSTC scheme, an adjustable code matrix is employed to transform the space-time coded matrix at the relay nodes. Least-squares (LS) algorithms are developed with reduced computational complexity to adjust the parameters of the codes. Simulation results show that t…
The multicast capacity of a directed network is closely related to the $s$-$t$ maximum flow, which is equal to the $s$-$t$ minimum cut capacity due to the max-flow min-cut theorem. If the topology of a network (or link capacities) is dynamically changing or have stochastic nature, it is not so trivial to predict statistical properties on the maximum flow. In this paper, we present a coding theoretic approach for evaluating the accumulate distribution of the minimum cut capacity of weighted random graphs. The main feature of our approach is to utilize the correspondence between the cut space of…
We wish to minimize the resources used for network coding while achieving the desired throughput in a multicast scenario. We employ evolutionary approaches, based on a genetic algorithm, that avoid the computational complexity that makes the problem NP-hard. Our experiments show great improvements over the sub-optimal solutions of prior methods. Our new algorithms improve over our previously proposed algorithm in three ways. First, whereas the previous algorithm can be applied only to acyclic networks, our new method works also with networks with cycles. Second, we enrich the set of components…
