5 pages, 2 figures. It is presented at the 54th Annual Conference on Information Sciences and Systems (CISS), Princeton, NJ, USA, March 18-20, 2020. arXiv admin note: text overlap with arXiv:1912.07083
We consider the problem of source coding subject to a fidelity criterion for the Gray-Wyner network that connects a single source with two receivers via a common channel and two private channels. The pareto-optimal trade-offs between the sum-rate of the private channels and the rate of the common channel is completely characterized for jointly Gaussian sources subject to the mean-squared error criterion, leveraging convex duality and an argument involving the factorization of convex envelopes. Specifically, it is attained by selecting the auxiliary random variable to be jointly Gaussian with the sources.
We show a reduction method to construct a code for the Gray-Wyner (GW) network from a given code for the Wyner-Ahlswede-Körner (WAK) network. By combining this reduction with a converse bound on the GW network, we derive a converse bound on the WAK network. The derived bound gives an alternative proof of the strong converse theorem for the WAK network.
The Gray-Wyner network subject to a fidelity criterion is studied. Upper and lower bounds for the trade-offs between the private sum-rate and the common rate are obtained for arbitrary sources subject to mean-squared error distortion. The bounds meet exactly, leading to the computation of the rate region, when the source is jointly Gaussian. They meet partially when the sources are modeled via an additive Gaussian "channel". The bounds are inspired from the Shannon bounds on the rate-distortion problem.
In this paper, the rate-distortion theory of the Gray-Wyner lossy source coding system is investigated. For the case of jointly Gaussian distributed sources, we establish an expression for the rate-distortion function under the constraint of quadratic distortion. Using the proposed rate-distortion function, any corner point on the rate-distortion region can be conveniently calculated. We take Wyner's common information as an example and provide a general and simple method to solve this problem. Through the analysis of the rate-distortion function, the rate on each layer and covariance matrix o…
We establish the rate region of an extended Gray-Wyner system for 2-DMS $(X,Y)$ with two additional decoders having complementary causal side information. This extension is interesting because in addition to the operationally significant extreme points of the Gray-Wyner rate region, which include Wyner's common information, G{á}cs-K{ö}rner common information and information bottleneck, the rate region for the extended system also includes the K{ö}rner graph entropy, the privacy funnel and excess functional information, as well as three new quantities of potential interest, as extreme points. T…
Explicit constructions of polar codes and polar lattices for both lossless and lossy Gray-Wyner problems are studied. Polar codes are employed to extract Wyner's common information of doubly symmetric binary source; polar lattices are then extended to extract that of a pair of Gaussian sources or multiple Gaussian sources. With regard to the discrete sources, the entire best-known region of the lossless Gray-Wyner problem are achieved by specifying the test channels to construct polar codes without time-sharing. As a result, we are able to give an interpretation that the Wyner's common informa…
The coding problem over the Gray-Wyner network is studied from the second-order coding rates perspective. A tilted information density for this network is introduced in the spirit of Kostina-Verdú, and, under a certain regularity condition, the second-order region is characterized in terms of the variance of this tilted information density and the tangent vector of the first-order region. The second-order region is proved by the type method: the achievability part is proved by the type-covering argument, and the converse part is proved by a refinement of the perturbation approach that was used…
Wyner's common information was originally defined for a pair of dependent discrete random variables. Its significance is largely reflected in, hence also confined to, several existing interpretations in various source coding problems. This paper attempts to both generalize its definition and to expand its practical significance by providing a new operational interpretation. The generalization is two-folded: the number of dependent variables can be arbitrary, so are the alphabet of those random variables. New properties are determined for the generalized Wyner's common information of N dependen…
The problem of preserving privacy when a multivariate source is required to be revealed partially to multiple users is modeled as a Gray-Wyner source coding problem with K correlated sources at the encoder and K decoders in which the kth decoder, k = 1, 2, ...,K, losslessly reconstructs the kth source via a common link and a private link. The privacy requirement of keeping each decoder oblivious of all sources other than the one intended for it is introduced via an equivocation constraint at each decoder such that the total equivocation summed over all decoders is E. The set of achievable rate…
In decentralized decision-making problems, communicating agents choose their actions based on locally available information and knowledge about decision rules or strategies of other agents. In this work, we consider a strategic communication game between an informed encoder and two decoders communicating via a Gray-Wyner network. All three agents are assumed to be rational and endowed with distinct objectives captured by non-aligned cost functions. The encoder selects and announces beforehand the compression scheme to be implemented. Then, it transmits three signals: a public signal, and a pri…
The problem of distributed binary hypothesis testing in the Gray-Wyner network with side information is studied in this paper. An observer has access to a discrete memoryless and stationary source and describes its observation to two detectors via one common and two private channels. The channels are considered error-free but rate-limited. Each detector also has access to its own discrete memoryless and stationary source, i.e., the side information. The goal is to perform two distinct binary hypothesis testings on the joint distribution of observations at detectors. Additionally, the observer …
The two most prevalent notions of common information (CI) are due to Wyner and Gacs-Korner and both the notions can be stated as two different characteristic points in the lossless Gray-Wyner region. Although the information theoretic characterizations for these two CI quantities can be easily evaluated for random variables with infinite entropy (eg., continuous random variables), their operational significance is applicable only to the lossless framework. The primary objective of this paper is to generalize these two CI notions to the lossy Gray-Wyner network, which hence extends the theoreti…
Examined in this paper, is the Gray and Wyner achievable lossy rate region for a tuple of correlated multivariate Gaussian random variables (RVs) $X_1 : Ω\rightarrow {\mathbb R}^{p_1}$ and $X_2 : Ω\rightarrow {\mathbb R}^{p_2}$ with respect to square-error distortions at the two decoders. It is shown that among all joint distributions induced by a triple of RVs $(X_1,X_2, W)$, such that $W : Ω\rightarrow {\mathbb W} $ is the auxiliary RV taking continuous, countable, or finite values, the Gray and Wyner achievable rate region is characterized by jointly Gaussian RVs $(X_1,X_2, W)$ such that $W…
Rate-Distortion-Perception Tradeoff for the Gray-Wyner Problem
School of Automation and Intelligent Manufacturing, Southern University of Science and Technology（南方科技大学自动化与智能制造学院）；School of Cyber Science and Technology, Beihang University（北京航空航天大学网络空间科学学院）
In this paper, we revisit the discrete lossy Gray-Wyner problem. In particular, we derive its optimal second-order coding rate region, its error exponent (reliability function) and its moderate deviations constant under mild conditions on the source. To obtain the second-order asymptotics, we extend some ideas from Watanabe's work (2015). In particular, we leverage the properties of an appropriate generalization of the conditional distortion-tilted information density, which was first introduced by Kostina and Verdú (2012). The converse part uses a perturbation argument by Gu and Effros (2009)…
We analyse the common information problem for the generalised Gray-Wyner problem. We aim to explore the problem and solution in relation to the non-orthogonality among the source decoders' components. We consider a simple networked control system consisting of 2 groups of users: (i) one sender or Observer named Alice; and (ii) a group of multiple receivers or Controllers, named Bobs. In order to tackle the possible risk arisen from the common information among Bobs, Alice provides a redundancy creating some virtual messages which are in the null of each specific Bob, but not for others. The af…
In this work, we investigate two source coding models, a \emph{Helper} problem and a \emph{Gray-Wyner} problem, under equivocation constraints. Specifically, in the Helper problem, an encoder communicates with a legitimate receiver through noise-free rate-limited public and private links; and an external passive eavesdropper intercepts every information that is sent on the public link. We study two classes of this model: i) when a pair of arbitrarily correlated discrete memoryless sources is to be encoded such that one component has to be recovered lossily at the legitimate receiver while the …
In this work, we establish a full single-letter characterization of the rate-distortion region of an instance of the Gray-Wyner model with side information at the decoders. Specifically, in this model an encoder observes a pair of memoryless, arbitrarily correlated, sources $(S^n_1,S^n_2)$ and communicates with two receivers over an error-free rate-limited link of capacity $R_0$, as well as error-free rate-limited individual links of capacities $R_1$ to the first receiver and $R_2$ to the second receiver. Both receivers reproduce the source component $S^n_2$ losslessly; and Receiver $1$ also r…
It is well known that lossless compression of a discrete memoryless source with near-uniform encoder output is possible at a rate above its entropy if and only if the encoder is randomized. This work focuses on deriving conditions for near-uniform encoder output(s) in the Wyner-Ziv and the distributed lossy compression problems. We show that in the Wyner-Ziv problem, near-uniform encoder output and operation close to the WZ-rate limit is simultaneously possible, whereas in the distributed lossy compression problem, jointly near-uniform outputs is achievable in the interior of the distributed l…
Wyner's Common Information and a natural relaxation are studied in the special case of Gaussian random variables. The relaxation replaces conditional independence by a bound on the conditional mutual information. The main contribution is the proof that Gaussian auxiliaries are optimal, leading to a closed-form formula. As a corollary, the proof technique also establishes the optimality of Gaussian auxiliaries for the Gaussian Gray-Wyner network, a long-standing open problem.
The classical Gray and Wyner source coding for a simple network for sources that generate a tuple of multivariate, correlated Gaussian random variables $(Y_1,Y_2)$ is re-examined using the geometric approach of Gaussian random variables, and the weak stochastic realization of correlated Gaussian random variables. New results are: (1) The formulation, methods and algorithms to parametrize all random variables $W : Ω\rightarrow {\mathbb R}^n $ which make the two components of the tuple $(Y_1,Y_2)$ conditionally independent, according to the weak stochastic realization of $(Y_1, Y_2)$. (2) The tr…
An important notion of common information between two random variables is due to Wyner. In this paper, we derive a lower bound on Wyner's common information for continuous random variables. The new bound improves on the only other general lower bound on Wyner's common information, which is the mutual information. We also show that the new lower bound is tight for the so-called "Gaussian channels" case, namely, when the joint distribution of the random variables can be written as the sum of a single underlying random variable and Gaussian noises. We motivate this work from the recent variations…
We show how real-number codes can be used to compress correlated sources and establish a new framework for distributed lossy source coding, in which we quantize compressed sources instead of compressing quantized sources. This change in the order of binning and quantization blocks makes it possible to model correlation between continuous-valued sources more realistically and compensate for the quantization error when the sources are completely correlated. We focus on the asymmetric case, i.e., lossy source coding with side information at the decoder, also known as Wyner-Ziv coding. The encodin…
In problems of lossy source/noisy channel coding with side information, the theoretical bounds are achieved using "good" source/channel codes that can be partitioned into "good" channel/source codes. A scheme that achieves optimality in channel coding with side information at the encoder using independent channel and source codes was outlined in previous works. In practice, the original problem is transformed into a multiple-access problem in which the superposition of the two independent codes can be decoded using successive interference cancellation. Inspired by this work, we analyze the sup…
We consider lossy source compression of a binary symmetric source using polar codes and the low-complexity successive encoding algorithm. It was recently shown by Arikan that polar codes achieve the capacity of arbitrary symmetric binary-input discrete memoryless channels under a successive decoding strategy. We show the equivalent result for lossy source compression, i.e., we show that this combination achieves the rate-distortion bound for a binary symmetric source. We further show the optimality of polar codes for various problems including the binary Wyner-Ziv and the binary Gelfand-Pinske…
In 1985 Kaspi provided a single-letter characterization of the sum-rate-distortion function for a two-way lossy source coding problem in which two terminals send multiple messages back and forth with the goal of reproducing each other's sources. Yet, the question remained whether more messages can strictly improve the sum-rate-distortion function. Viewing the sum-rate as a functional of the distortions and the joint source distribution and leveraging its convex-geometric properties, we construct an example which shows that two messages can strictly improve the one-message (Wyner-Ziv) rate-dist…
In this work, a likelihood encoder is studied in the context of lossy source compression. The analysis of the likelihood encoder is based on a soft-covering lemma. It is demonstrated that the use of a likelihood encoder together with the soft-covering lemma gives alternative achievability proofs for classical source coding problems. The case of the rate-distortion function with side information at the decoder (i.e. the Wyner-Ziv problem) is carefully examined and an application of the likelihood encoder to the multi-terminal source coding inner bound (i.e. the Berger-Tung region) is outlined.
We presented assisted common information as a generalization of Gács-Körner (GK) common information at ISIT 2010. The motivation for our formulation was to improve upperbounds on the efficiency of protocols for secure two-party sampling (which is a form of secure multi-party computation). Our upperbound was based on a monotonicity property of a rate-region (called the assisted residual information region) associated with the assisted common information formulation. In this note we present further results. We explore the connection of assisted common information with the Gray-Wyner system. We s…
This paper generalizes Wyner's definition of common information of a pair of random variables to that of $N$ random variables. We prove coding theorems that show the same operational meanings for the common information of two random variables generalize to that of $N$ random variables. As a byproduct of our proof, we show that the Gray-Wyner source coding network can be generalized to $N$ source squences with $N$ decoders. We also establish a monotone property of Wyner's common information which is in contrast to other notions of the common information, specifically Shannon's mutual informatio…
This paper addresses lossy transmission of a common source over a broadcast channel when there is correlated side information at the receivers, with emphasis on the quadratic Gaussian and binary Hamming cases. A digital scheme that combines ideas from the lossless version of the problem, i.e., Slepian-Wolf coding over broadcast channels, and dirty paper coding, is presented and analyzed. This scheme uses layered coding where the common layer information is intended for both receivers and the refinement information is destined only for one receiver. For the quadratic Gaussian case, a quantity c…
The Gray and Wyner lossy source coding for a simple network for sources that generate a tuple of jointly Gaussian random variables (RVs) $X_1 : Ω\rightarrow {\mathbb R}^{p_1}$ and $X_2 : Ω\rightarrow {\mathbb R}^{p_2}$, with respect to square-error distortion at the two decoders is re-examined using (1) Hotelling's geometric approach of Gaussian RVs-the canonical variable form, and (2) van Putten's and van Schuppen's parametrization of joint distributions ${\bf P}_{X_1, X_2, W}$ by Gaussian RVs $W : Ω\rightarrow {\mathbb R}^n $ which make $(X_1,X_2)$ conditionally independent, and the weak sto…
In this paper, we introduce new lower bounds on the distortion of scalar fixed-rate codes for lossy compression with side information available at the receiver. These bounds are derived by presenting the relevant random variables as a Markov chain and applying generalized data processing inequalities a la Ziv and Zakai. We show that by replacing the logarithmic function with other functions, in the data processing theorem we formulate, we obtain new lower bounds on the distortion of scalar coding with side information at the decoder. The usefulness of these results is demonstrated for uniform …
A likelihood encoder is studied in the context of lossy source compression. The analysis of the likelihood encoder is based on the soft-covering lemma. It is demonstrated that the use of a likelihood encoder together with the soft-covering lemma yields simple achievability proofs for classical source coding problems. The cases of the point-to-point rate-distortion function, the rate-distortion function with side information at the decoder (i.e. the Wyner-Ziv problem), and the multi-terminal source coding inner bound (i.e. the Berger-Tung problem) are examined in this paper. Furthermore, a non-…
A multiterminal lossy coding problem, which includes various problems such as the Wyner-Ziv problem and the complementary delivery problem as special cases, is considered. It is shown that any point in the achievable rate-distortion region can be attained even if the source statistics are not known.
Polar lattices, which are constructed from polar codes, have recently been proved to be able to achieve the capacity of the additive white Gaussian noise (AWGN) channel. In this work, we propose a new construction of polar lattices to solve the dual problem, i.e., achieving the rate-distortion bound of a memoryless Gaussian source, which means that polar lattices can also be good for the lossy compression of continuous sources. The structure of the proposed polar lattices enables us to integrate the post-entropy coding process into the lattice quantizer, which simplifies the quantization proce…
Advancing over up-to-date information theoretic results that assume symmetric correlation models, in this work we consider the problem of lossy binary source coding with side information, where the correlation is expressed by a generic binary asymmetric channel. Specifically, we present an in-depth analysis of rate distortion with side information available to both the encoder and decoder (conventional predictive), as well as the Wyner-Ziv problem for this particular setup. Prompted by our recent results for the Z-channel correlation case, we evaluate the rate loss between the Wyner-Ziv and th…
We consider a two-user state-dependent multiaccess channel in which the states of the channel are known non-causally to one of the encoders and only strictly causally to the other encoder. Both encoders transmit a common message and, in addition, the encoder that knows the states non-causally transmits an individual message. We find explicit characterizations of the capacity region of this communication model in both discrete memoryless and memoryless Gaussian cases. The analysis also reveals optimal ways of exploiting the knowledge of the state only strictly causally at the encoder that sends…
This paper investigates the problem of secure lossy source coding in the presence of an eavesdropper with arbitrary correlated side informations at the legitimate decoder (referred to as Bob) and the eavesdropper (referred to as Eve). This scenario consists of an encoder that wishes to compress a source to satisfy the desired requirements on: (i) the distortion level at Bob and (ii) the equivocation rate at Eve. It is assumed that the decoders have access to correlated sources as side information. For instance, this problem can be seen as a generalization of the well-known Wyner-Ziv problem ta…
Consider the following information theoretic setup wherein independent codebooks of N correlated random variables are generated according to their respective marginals. The problem of determining the conditions on the rates of codebooks to ensure the existence of at least one codeword tuple which is jointly typical with respect to a given joint density (called the multivariate covering lemma) has been studied fairly well and the associated rate regions have found applications in several source coding scenarios. However, several multiterminal source coding applications, such as the general mult…
We study error exponents for source coding with side information. Both achievable exponents and converse bounds are obtained for the following two cases: lossless source coding with coded information (SCCSI) and lossy source coding with full side information (Wyner-Ziv). These results recover and extend several existing results on source-coding error exponents and are tight in some circumstances. Our bounds have a natural interpretation as a two-player game between nature and the code designer, with nature seeking to minimize the exponent and the code designer seeking to maximize it. In the Wy…
We present a nested lattice-code-based strategy that achieves the random-coding based Compress-and-Forward (CF) rate for the three node Gaussian relay channel. To do so, we first outline a lattice-based strategy for the $(X+Z_1,X+Z_2)$ Wyner-Ziv lossy source-coding with side-information problem in Gaussian noise, a re-interpretation of the nested lattice-code-based Gaussian Wyner-Ziv scheme presented by Zamir, Shamai, and Erez. We use the notation $(X+Z_1,X+Z_2)$ Wyner-Ziv to mean that the source is of the form $X+ Z_1$ and the side-information at the receiver is of the form $X+ Z_2$, for inde…
This paper takes a rate-distortion approach to understanding the information-theoretic laws governing cache-aided communications systems. Specifically, we characterise the optimal tradeoffs between the delivery rate, cache capacity and reconstruction distortions for a single-user problem and some special cases of a two-user problem. Our analysis considers discrete memoryless sources, expected- and excess-distortion constraints, and separable and f-separable distortion functions. We also establish a strong converse for separable-distortion functions, and we show that lossy versions of common in…
We study the secure lossy transmission of a vector Gaussian source to a legitimate user in the presence of an eavesdropper, where both the legitimate user and the eavesdropper have vector Gaussian side information. The aim of the transmitter is to describe the source to the legitimate user in a way that the legitimate user can reconstruct the source within a certain distortion level while the eavesdropper is kept ignorant of the source as much as possible as measured by the equivocation. We obtain an outer bound for the rate, equivocation and distortion region of this secure lossy transmission…
The problem of reliable transmission of correlated sources over the broadcast channel, originally studied by Han and Costa, is revisited. An alternative characterization of their sufficient condition for reliable transmission is given, which includes results of Marton for channel coding over broadcast channels and of Gray and Wyner for distributed source coding. A ``minimalistic'' coding scheme is presented, which is based on joint typicality encoding and decoding, without requiring the use of Cover's superposition coding, random hashing, and common part between two sources. The analysis of th…
A secrecy system with side information at the decoders is studied in the context of lossy source compression over a noiseless broadcast channel. The decoders have access to different side information sequences that are correlated with the source. The fidelity of the communication to the legitimate receiver is measured by a distortion metric, as is traditionally done in the Wyner-Ziv problem. The secrecy performance of the system is also evaluated under a distortion metric. An achievable rate-distortion region is derived for the general case of arbitrarily correlated side information. Exact bou…
Source coding with a side information "vending machine" is a recently proposed framework in which the statistical relationship between the side information and the source, instead of being given and fixed as in the classical Wyner-Ziv problem, can be controlled by the decoder. This control action is selected by the decoder based on the message encoded by the source node. Unlike conventional settings, the message can thus carry not only information about the source to be reproduced at the decoder, but also control information aimed at improving the quality of the side information. In this paper…
