We consider a variation of the Wyner-Ziv problem pertaining to lossy compression of individual sequences using finite-state encoders and decoders. There are two main results in this paper. The first characterizes the relationship between the performance of the best $M$-state encoder-decoder pair to that of the best block code of size $\ell$ for every input sequence, and shows that the loss of the latter relative to the former (in terms of both rate and distortion) never exceeds the order of $(\log M)/\ell$, independently of the input sequence. Thus, in the limit of large $M$, the best rate-distortion performance of every infinite source sequence can be approached universally by a sequence of block codes (which are also implementable by finite-state machines). While this result assumes an asymptotic regime where the number of states is fixed, and only the length $n$ of the input sequence grows without bound, we then consider the case where the number of states $M=M_n$ is allowed to grow concurrently with $n$. Our second result is then about the critical growth rate of $M_n$ such that the rate-distortion performance of $M_n$-state encoder-decoder pairs can still be matched by a universal code. We show that this critical growth rate of $M_n$ is linear in $n$.
We introduce a distributed source coding scheme called successive Wyner-Ziv coding. We show that any point in the rate region of the quadratic Gaussian CEO problem can be achieved via the successive Wyner-Ziv coding. The concept of successive refinement in the single source coding is generalized to the distributed source coding scenario, which we refer to as distributed successive refinement. For the quadratic Gaussian CEO problem, we establish a necessary and sufficient condition for distributed successive refinement, where the successive Wyner-Ziv coding scheme plays an important role.
We describe and analyze sparse graphical code constructions for the problems of source coding with decoder side information (the Wyner-Ziv problem), and channel coding with encoder side information (the Gelfand-Pinsker problem). Our approach relies on a combination of low-density parity check (LDPC) codes and low-density generator matrix (LDGM) codes, and produces sparse constructions that are simultaneously good as both source and channel codes. In particular, we prove that under maximum likelihood encoding/decoding, there exist low-density codes (i.e., with finite degrees) from our construct…
Distributed source coding (DSC) addresses the compression of correlated sources without communication links among them. This paper is concerned with the Wyner-Ziv problem: coding of an information source with side information available only at the decoder in the form of a noisy version of the source. Both the theoretical analysis and code design are addressed in the framework of multi-dimensional nested lattice coding (NLC). For theoretical analysis, accurate computation of the rate-distortion function is given under the high-resolution assumption, and a new upper bound using the derivative of…
We investigate the Wyner-Ziv coding in which the statistics of the principal source is known but the statistics of the channel generating the side-information is unknown except that it is in a certain class. The class consists of channels such that the distortion between the principal source and the side-information is smaller than a threshold, but channels may be neither stationary nor ergodic. In this situation, we define a new rate-distortion function as the minimum rate such that there exists a Wyner-Ziv code that is universal for every channel in the class. Then, we show an upper bound an…
An $l$-link binary CEO problem is considered in this paper. We present a practical encoding and decoding scheme for this problem employing the graph-based codes. A successive coding scheme is proposed for converting an $l$-link binary CEO problem to the $(2l-1)$ single binary Wyner-Ziv (WZ) problems. By using the compound LDGM-LDPC codes, the theoretical bound of each binary WZ is asymptotically achievable. Our proposed decoder successively decodes the received data by employing the well-known Sum-Product (SP) algorithm and leverages them to reconstruct the source. The sum-rate distortion perf…
We consider the role of Wyner-Ziv binning in compress-forward for relay channels. In the one-way relay channel, we analyze a compress-forward scheme without Wyner- Ziv binning but with joint decoding of both the message and compression index. It achieves the same rate as the original compress-forward scheme with binning and successive decoding. Therefore, binning helps reduce decoding complexity by allowing successive decoding, but has no impact on achievable rate for the one-way relay channel. On the other hand, no binning simplifies relay operation. By extending compress-forward without binn…
We consider both channel coding and source coding, with perfect past feedback/feedforward, in the presence of side information. It is first observed that feedback does not increase the capacity of the Gel'fand-Pinsker channel, nor does feedforward improve the achievable rate-distortion performance in the Wyner-Ziv problem. We then focus on the Gaussian case showing that, as in the absence of side information, feedback/feedforward allows to efficiently attain the respective performance limits. In particular, we derive schemes via variations on that of Schalkwijk and Kailath. These variants, whi…
The $L$-link binary Chief Executive Officer (CEO) problem under logarithmic loss is investigated in this paper. A quantization splitting technique is applied to convert the problem under consideration to a $(2L-1)$-step successive Wyner-Ziv (WZ) problem, for which a practical coding scheme is proposed. In the proposed scheme, low-density generator-matrix (LDGM) codes are used for binary quantization while low-density parity-check (LDPC) codes are used for syndrome generation; the decoder performs successive decoding based on the received syndromes and produces a soft reconstruction of the remo…
The combination of source coding with decoder side-information (Wyner-Ziv problem) and channel coding with encoder side-information (Gel'fand-Pinsker problem) can be optimally solved using the separation principle. In this work we show an alternative scheme for the quadratic-Gaussian case, which merges source and channel coding. This scheme achieves the optimal performance by a applying modulo-lattice modulation to the analog source. Thus it saves the complexity of quantization and channel decoding, and remains with the task of "shaping" only. Furthermore, for high signal-to-noise ratio (SNR),…
In problems of lossy source/noisy channel coding with side information, the theoretical bounds are achieved using "good" source/channel codes that can be partitioned into "good" channel/source codes. A scheme that achieves optimality in channel coding with side information at the encoder using independent channel and source codes was outlined in previous works. In practice, the original problem is transformed into a multiple-access problem in which the superposition of the two independent codes can be decoded using successive interference cancellation. Inspired by this work, we analyze the sup…
Wyner-Ziv coding (WZC) is a compression technique using decoder side information, which is unknown at the encoder, to help the reconstruction. In this paper, we propose and implement a new WZC structure, called residual WZC, for the quadratic-Gaussian Wyner-Ziv problem where side information can be arbitrarily distributed. In our two-stage residual WZC, the source is quantized twice and the input of the second stage is the quantization error (residue) of the first stage. The codebook of the first stage quantizer must be simultaneously good for source and channel coding, since it also acts as a…
In 1985 Kaspi provided a single-letter characterization of the sum-rate-distortion function for a two-way lossy source coding problem in which two terminals send multiple messages back and forth with the goal of reproducing each other's sources. Yet, the question remained whether more messages can strictly improve the sum-rate-distortion function. Viewing the sum-rate as a functional of the distortions and the joint source distribution and leveraging its convex-geometric properties, we construct an example which shows that two messages can strictly improve the one-message (Wyner-Ziv) rate-dist…
We provide a complete characterization of the rate-distortion region for the multistage successive refinement of the Wyner-Ziv source coding problem with degraded side informations at the decoder. Necessary and sufficient conditions for a source to be successively refinable along a distortion vector are subsequently derived. A source-channel separation theorem is provided when the descriptions are sent over independent channels for the multistage case. Furthermore, we introduce the notion of generalized successive refinability with multiple degraded side informations. This notion captures whet…
We show how real-number codes can be used to compress correlated sources and establish a new framework for distributed lossy source coding, in which we quantize compressed sources instead of compressing quantized sources. This change in the order of binning and quantization blocks makes it possible to model correlation between continuous-valued sources more realistically and compensate for the quantization error when the sources are completely correlated. We focus on the asymmetric case, i.e., lossy source coding with side information at the decoder, also known as Wyner-Ziv coding. The encodin…
This paper addresses lossy transmission of a common source over a broadcast channel when there is correlated side information at the receivers, with emphasis on the quadratic Gaussian and binary Hamming cases. A digital scheme that combines ideas from the lossless version of the problem, i.e., Slepian-Wolf coding over broadcast channels, and dirty paper coding, is presented and analyzed. This scheme uses layered coding where the common layer information is intended for both receivers and the refinement information is destined only for one receiver. For the quadratic Gaussian case, a quantity c…
We consider a two-user state-dependent multiaccess channel in which the states of the channel are known non-causally to one of the encoders and only strictly causally to the other encoder. Both encoders transmit a common message and, in addition, the encoder that knows the states non-causally transmits an individual message. We find explicit characterizations of the capacity region of this communication model in both discrete memoryless and memoryless Gaussian cases. The analysis also reveals optimal ways of exploiting the knowledge of the state only strictly causally at the encoder that sends…
National University of Singapore（新加坡国立大学）；Indian Institute of Science（印度科学学院）；Google Research（谷歌研究院）
Communication efficient distributed mean estimation is an important primitive that arises in many distributed learning and optimization scenarios such as federated learning. Without any probabilistic assumptions on the underlying data, we study the problem of distributed mean estimation where the server has access to side information. We propose \emph{Wyner-Ziv estimators}, which are communication and computationally efficient and near-optimal when an upper bound for the distance between the side information and the data is known. As a corollary, we also show that our algorithms provide effici…
This paper investigates the joint source-channel coding problem of sending a memoryless source over a memoryless broadcast channel. An inner bound and several outer bounds on the admissible distortion region are derived, which respectively generalize and unify several existing bounds. As a consequence, we also obtain an inner bound and an outer bound for the degraded broadcast channel case. When specialized to the Gaussian or binary source broadcast, the inner bound and outer bound not only recover the best known inner bound and outer bound in the literature, but also generate some new results…
Noisy network coding is recently proposed for the general multi-source network by Lim, Kim, El Gamal and Chung. This scheme builds on compress-forward (CF) relaying but involves three new ideas, namely no Wyner-Ziv binning, relaxed simultaneous decoding and message repetition. In this paper, using the two-way relay channel as the underlining example, we analyze the impact of each of these ideas on the achievable rate region of relay networks. First, CF without binning but with joint decoding of both the message and compression index can achieve a larger rate region than the original CF scheme …
In this paper we provide a new inequality useful for the proofs of strong converse theorems in the multiterminal information theory. We apply this inequality to the recent work by Tyagi and Watanabe on the strong converse theorem for the Wyner-Ziv source coding problem to obtain a new strong converse outer bound. This outer bound deviates from the Wyner-Ziv rate distortion region with the order $O\left(\frac{1}{\sqrt{n}}\right)$ on the length $n$ of source outputs.
We present a nested lattice-code-based strategy that achieves the random-coding based Compress-and-Forward (CF) rate for the three node Gaussian relay channel. To do so, we first outline a lattice-based strategy for the $(X+Z_1,X+Z_2)$ Wyner-Ziv lossy source-coding with side-information problem in Gaussian noise, a re-interpretation of the nested lattice-code-based Gaussian Wyner-Ziv scheme presented by Zamir, Shamai, and Erez. We use the notation $(X+Z_1,X+Z_2)$ Wyner-Ziv to mean that the source is of the form $X+ Z_1$ and the side-information at the receiver is of the form $X+ Z_2$, for inde…
We consider the rate distortion problem with side information at the decoder posed and investigated by Wyner and Ziv. The rate distortion function indicating the trade-off between the rate on the data compression and the quality of data obtained at the decoder was determined by Wyner and Ziv. In this paper, we study the error probability of decoding at rates below the rate distortion function. We evaluate the probability of decoding such that the estimation of source outputs by the decoder has a distortion not exceeding a prescribed distortion level. We prove that when the rate of the data com…
It is well known that lossless compression of a discrete memoryless source with near-uniform encoder output is possible at a rate above its entropy if and only if the encoder is randomized. This work focuses on deriving conditions for near-uniform encoder output(s) in the Wyner-Ziv and the distributed lossy compression problems. We show that in the Wyner-Ziv problem, near-uniform encoder output and operation close to the WZ-rate limit is simultaneously possible, whereas in the distributed lossy compression problem, jointly near-uniform outputs is achievable in the interior of the distributed l…
Advancing over up-to-date information theoretic results that assume symmetric correlation models, in this work we consider the problem of lossy binary source coding with side information, where the correlation is expressed by a generic binary asymmetric channel. Specifically, we present an in-depth analysis of rate distortion with side information available to both the encoder and decoder (conventional predictive), as well as the Wyner-Ziv problem for this particular setup. Prompted by our recent results for the Z-channel correlation case, we evaluate the rate loss between the Wyner-Ziv and th…
Indian Institute of Technology, Kharagpur（印度理工学院卡拉格普尔分校）；Azure Software Systems Ltd.（Azure软件系统有限公司）
In this paper, we propose a new feedback-channel-free Distributed Video Coding (DVC) algorithm using Local Rank Transform (LRT). The encoder computes LRT by considering selected neighborhood pixels of Wyner-Ziv frame. The ranks from the modified LRT are merged, and their positions are entropy coded and sent to the decoder. In addition, means of each block of Wyner-Ziv frame are also transmitted to assist motion estimation. Using these measurements, the decoder generates side information (SI) by implementing motion estimation and compensation in LRT domain. An iterative algorithm is executed on…
In the first paper of this two part communication, we solved in a unified framework a variety of two terminal source coding problems with noncooperative encoders, thereby consolidating works of Shannon, Slepian-Wolf, Wyner, Ahlswede-Körner, Wyner-Ziv, Berger {\em et al.} and Berger-Yeung. To achieve such unification we made use of a fundamental principle that dissociates bulk of the analysis from the distortion criterion at hand (if any) and extends the typicality arguments of Shannon and Wyner-Ziv. In this second paper, we generalize the fundamental principle for any number of sources and on …
This paper deals with a coding problem called complementary delivery, where messages from two correlated sources are jointly encoded and each decoder reproduces one of two messages using the other message as the side information. Both lossless and lossy universal complementary delivery coding schemes are investigated. In the lossless case, it is demonstrated that a universal complementary delivery code can be constructed by only combining two Slepian-Wolf codes. Especially, it is shown that a universal lossless complementary delivery code, for which error probability is exponentially tight, ca…
We consider a source coding problem with a network scenario in mind, and formulate it as a remote vector Gaussian Wyner-Ziv problem under covariance matrix distortions. We define a notion of minimum for two positive-definite matrices based on which we derive an explicit formula for the rate-distortion function (RDF). We then study the special cases and applications of this result. We show that two well-studied source coding problems, i.e. remote vector Gaussian Wyner-Ziv problems with mean-squared error and mutual information constraints are in fact special cases of our results. Finally, we ap…
A multiterminal lossy coding problem, which includes various problems such as the Wyner-Ziv problem and the complementary delivery problem as special cases, is considered. It is shown that any point in the achievable rate-distortion region can be attained even if the source statistics are not known.
Since the publication of Shannon's theory of one terminal source coding, a number of interesting extensions have been derived by researchers such as Slepian-Wolf, Wyner, Ahlswede-Körner, Wyner-Ziv and Berger-Yeung. Specifically, the achievable rate or rate-distortion region has been described by a first order information-theoretic functional of the source statistics in each of the above cases. At the same time several problems have also remained unsolved. Notable two terminal examples include the joint distortion problem, where both sources are reconstructed under a combined distortion criteri…
In this paper, we introduce new lower bounds on the distortion of scalar fixed-rate codes for lossy compression with side information available at the receiver. These bounds are derived by presenting the relevant random variables as a Markov chain and applying generalized data processing inequalities a la Ziv and Zakai. We show that by replacing the logarithmic function with other functions, in the data processing theorem we formulate, we obtain new lower bounds on the distortion of scalar coding with side information at the decoder. The usefulness of these results is demonstrated for uniform …
We show that the duality between channel capacity and data compression is retained when state information is available to the sender, to the receiver, to both, or to neither. We present a unified theory for eight special cases of channel capacity and rate distortion with state information, which also extends existing results to arbitrary pairs of independent and identically distributed (i.i.d.) correlated state information available at the sender and at the receiver, respectively. In particular, the resulting general formula for channel capacity assumes the same form as the generalized Wyner Z…
In this work, a likelihood encoder is studied in the context of lossy source compression. The analysis of the likelihood encoder is based on a soft-covering lemma. It is demonstrated that the use of a likelihood encoder together with the soft-covering lemma gives alternative achievability proofs for classical source coding problems. The case of the rate-distortion function with side information at the decoder (i.e. the Wyner-Ziv problem) is carefully examined and an application of the likelihood encoder to the multi-terminal source coding inner bound (i.e. the Berger-Tung region) is outlined.
We study communication systems over band-limited Additive White Gaussian Noise (AWGN) channels in which the transmitter output is constrained to be symmetric binary (bi-polar). In this work we improve the original Ozarov-Wyner-Ziv (OWZ) lower bound on capacity by introducing new achievability schemes with two advantages over the studied OWZ scheme which is based on peak-power constrained pulse-amplitude modulation. Our schemes achieve a moderately improved information rate and do so with much less sign transitions of the binary signal. The gap between the known upper-bound based on spectral co…
The Wyner-Ahlswede-Körner (WAK) empirical-coordination problem where the encoders cooperate via a finite-capacity one-sided link is considered. The coordination-capacity region is derived by combining several source coding techniques, such as Wyner-Ziv (WZ) coding, binning and superposition coding. Furthermore, a semi-deterministic (SD) broadcast channel (BC) with one-sided decoder cooperation is considered. Duality principles relating the two problems are presented, and the capacity region for the SD-BC setting is derived. The direct part follows from an achievable region for a general BC tha…
The two most prevalent notions of common information (CI) are due to Wyner and Gacs-Korner and both the notions can be stated as two different characteristic points in the lossless Gray-Wyner region. Although the information theoretic characterizations for these two CI quantities can be easily evaluated for random variables with infinite entropy (eg., continuous random variables), their operational significance is applicable only to the lossless framework. The primary objective of this paper is to generalize these two CI notions to the lossy Gray-Wyner network, which hence extends the theoreti…
In this work, we consider a distributed source coding problem with a joint distortion criterion depending on the sources and the reconstruction. This includes as a special case the problem of computing a function of the sources to within some distortion and also the classic Slepian-Wolf problem, Berger-Tung problem, Wyner-Ziv problem, Yeung-Berger problem and the Ahlswede-Korner-Wyner problem. While the prevalent trend in information theory has been to prove achievability results using Shannon's random coding arguments, using structured random codes offer rate gains over unstructured random co…
A secrecy system with side information at the decoders is studied in the context of lossy source compression over a noiseless broadcast channel. The decoders have access to different side information sequences that are correlated with the source. The fidelity of the communication to the legitimate receiver is measured by a distortion metric, as is traditionally done in the Wyner-Ziv problem. The secrecy performance of the system is also evaluated under a distortion metric. An achievable rate-distortion region is derived for the general case of arbitrarily correlated side information. Exact bou…
Consider the two-way rate-distortion problem in which a helper sends a common limited-rate message to both users based on side information at its disposal. We characterize the region of achievable rates and distortions where a Markov form (Helper)-(User 1)-(User 2) holds. The main insight of the result is that in order to achieve the optimal rate, the helper may use a binning scheme, as in Wyner-Ziv, where the side information at the decoder is the "further" user, namely, User 2. We derive these regions explicitly for the Gaussian sources with square error distortion, analyze a trade-off betwe…
The source coding problem with action-dependent side information at the decoder has recently been introduced to model data acquisition in resource-constrained systems. In this paper, an efficient algorithm for numerical computation of the rate-distortion-cost function for this problem is proposed, and a convergence proof is provided. Moreover, a two-stage code design based on multiplexing is put forth, whereby the first stage encodes the actions and the second stage is composed of an array of classical Wyner-Ziv codes, one for each action. Specific coding/decoding strategies are designed based…
We present novel non-asymptotic or finite blocklength achievability bounds for three side-information problems in network information theory. These include (i) the Wyner-Ahlswede-Korner (WAK) problem of almost-lossless source coding with rate-limited side-information, (ii) the Wyner-Ziv (WZ) problem of lossy source coding with side-information at the decoder and (iii) the Gel'fand-Pinsker (GP) problem of channel coding with noncausal state information available at the encoder. The bounds are proved using ideas from channel simulation and channel resolvability. Our bounds for all three problems…
We introduce the idea of distortion side information, which does not directly depend on the source but instead affects the distortion measure. We show that such distortion side information is not only useful at the encoder, but that under certain conditions, knowing it at only the encoder is as good as knowing it at both encoder and decoder, and knowing it at only the decoder is useless. Thus distortion side information is a natural complement to the signal side information studied by Wyner and Ziv, which depends on the source but does not involve the distortion measure. Furthermore, when both…
A likelihood encoder is studied in the context of lossy source compression. The analysis of the likelihood encoder is based on the soft-covering lemma. It is demonstrated that the use of a likelihood encoder together with the soft-covering lemma yields simple achievability proofs for classical source coding problems. The cases of the point-to-point rate-distortion function, the rate-distortion function with side information at the decoder (i.e. the Wyner-Ziv problem), and the multi-terminal source coding inner bound (i.e. the Berger-Tung problem) are examined in this paper. Furthermore, a non-…
In analogy to the well-known notion of finite--state compressibility of individual sequences, due to Lempel and Ziv, we define a similar notion of "finite-state encryptability" of an individual plaintext sequence, as the minimum asymptotic key rate that must be consumed by finite-state encrypters so as to guarantee perfect secrecy in a well-defined sense. Our main basic result is that the finite-state encryptability is equal to the finite-state compressibility for every individual sequence. This is in parallelism to Shannon's classical probabilistic counterpart result, asserting that the minim…
In this work, we establish a full single-letter characterization of the rate-distortion region of an instance of the Gray-Wyner model with side information at the decoders. Specifically, in this model an encoder observes a pair of memoryless, arbitrarily correlated, sources $(S^n_1,S^n_2)$ and communicates with two receivers over an error-free rate-limited link of capacity $R_0$, as well as error-free rate-limited individual links of capacities $R_1$ to the first receiver and $R_2$ to the second receiver. Both receivers reproduce the source component $S^n_2$ losslessly; and Receiver $1$ also r…
Polar lattices, which are constructed from polar codes, have recently been proved to be able to achieve the capacity of the additive white Gaussian noise (AWGN) channel. In this work, we propose a new construction of polar lattices to solve the dual problem, i.e., achieving the rate-distortion bound of a memoryless Gaussian source, which means that polar lattices can also be good for the lossy compression of continuous sources. The structure of the proposed polar lattices enables us to integrate the post-entropy coding process into the lattice quantizer, which simplifies the quantization proce…
