We consider the problem of lossy source coding with a mismatched distortion measure. That is, we investigate what distortion guarantees can be made with respect to distortion measure $\tildeρ$, for a source code designed such that it achieves distortion less than $D$ with respect to distortion measure $ρ$. We find a single-letter characterization of this mismatch distortion and study properties of this quantity. These results give insight into the robustness of lossy source coding with respect to modeling errors in the distortion measure. They also provide guidelines on how to choose a good tractable approximation of an intractable distortion measure.
Arıkan's polar coding, is by now a well studied technique that allows achieving the symmetric capacity of binary input memoryless channels with low complexity encoding and decoding, provided that the polar decoding architecture is used and the decoding metric is matched to the true channel. In this paper, we analyze communication rates that are achievable when the polar coding/decoding architecture is used with the decoder using an incorrect model of the channel. We define the `polar mismatched capacity' as an analogue of the classical mismatched capacity, give an expression for it, and derive…
This paper studies multiuser random coding techniques for channel coding with a given (possibly suboptimal) decoding rule. For the mismatched discrete memoryless multiple-access channel, an error exponent is obtained that is tight with respect to the ensemble average, and positive within the interior of Lapidoth's achievable rate region. This exponent proves the ensemble tightness of the exponent of Liu and Hughes in the case of maximum-likelihood decoding. An equivalent dual form of Lapidoth's achievable rate region is given, and the latter is shown to extend immediately to channels with infi…
In this paper, we consider the mismatched rate-distortion problem, in which the encoding is done using a codebook, and the encoder chooses the minimum-distortion codeword according to a mismatched distortion function that differs from the true one. For the case of discrete memoryless sources, we establish achievable rate-distortion bounds using multi-user coding techniques, namely, superposition coding and expurgated parallel coding. We study examples where these attain the matched rate-distortion trade-off but a standard ensemble with independent codewords fails to do so. On the other hand, i…
We consider a joint source channel coding (JSCC) problem in which we desire to transmit an arbitrary memoryless source over an arbitrary additive channel. We propose a mismatched coding architecture that consists of Gaussian codebooks for both the source reproduction sequences and channel codewords. The natural nearest neighbor encoder and decoder, however, need to be judiciously modified to obtain the highest communication rates at finite blocklength. In particular, we consider a unequal error protection (UEP) scheme in which all sources are partitioned into disjoint power type classes. We al…
This paper considers the problem of channel coding with a given (possibly suboptimal) maximum-metric decoding rule. A cost-constrained random-coding ensemble with multiple auxiliary costs is introduced, and is shown to achieve error exponents and second-order coding rates matching those of constant-composition random coding, while being directly applicable to channels with infinite or continuous alphabets. The number of auxiliary costs required to match the error exponents and second-order rates of constant-composition coding is studied, and is shown to be at most two. For i.i.d. random coding…
The problem of mismatched decoding for discrete memoryless channels is addressed. A mismatched cognitive multiple-access channel is introduced, and an inner bound on its capacity region is derived using two alternative encoding methods: superposition coding and random binning. The inner bounds are derived by analyzing the average error probability of the code ensemble for both methods and by a tight characterization of the resulting error exponents. Random coding converse theorems are also derived. A comparison of the achievable regions shows that in the matched case, random binning performs a…
We introduce a universal quantization scheme based on random coding, and we analyze its performance. This scheme consists of a source-independent random codebook (typically_mismatched_ to the source distribution), followed by optimal entropy-coding that is_matched_ to the quantized codeword distribution. A single-letter formula is derived for the rate achieved by this scheme at a given distortion, in the limit of large codebook dimension. The rate reduction due to entropy-coding is quantified, and it is shown that it can be arbitrarily large. In the special case of "almost uniform" codebooks (…
We consider a multiterminal source coding problem in which a source is estimated at a central processing unit from lossy-compressed remote observations. Each lossy-encoded observation is produced by a remote sensor which obtains a noisy version of the source and compresses this observation minimizing a local distortion measure which depends only on the marginal distribution of its observation. The central node, on the other hand, has knowledge of the joint distribution of the source and all the observations and produces the source estimate which minimizes a different distortion measure between…
Hardware impairments in radio-frequency components of a wireless system cause unavoidable distortions to transmission that are not captured by the conventional linear channel model. In this paper, a 'binoisy' single-user multiple-input multiple-output (SU-MIMO) relation is considered where the additional distortions are modeled via an additive noise term at the transmit side. Through this extended SU-MIMO channel model, the effects of transceiver hardware impairments on the achievable rate of multi-antenna point-to-point systems are studied. Channel input distributions encompassing practical d…
This paper studies channel coding for the discrete memoryless multiple-access channel with a given (possibly suboptimal) decoding rule. A multi-letter successive decoding rule depending on an arbitrary non-negative decoding metric is considered, and achievable rate regions and error exponents are derived both for the standard MAC (independent codebooks), and for the cognitive MAC (one user knows both messages) with superposition coding. In the cognitive case, the rate region and error exponent are shown to be tight with respect to the ensemble average. The rate regions are compared with those …
This paper studies the mismatched decoding problem for binary-input discrete memoryless channels. An example is provided for which an achievable rate based on superposition coding exceeds the LM rate (Hui, 1983; Csiszár-Körner, 1981), thus providing a counter-example to a previously reported converse result (Balakirsky, 1995). Both numerical evaluations and theoretical results are used in establishing this claim.
Information-Theoretic Foundations of Mismatched Decoding
National University of Singapore（新加坡国立大学）；Institució Catalana de Recerca i Estudis Avançats (ICREA)（加泰罗尼亚高级研究学院）；Universitat Pompeu Fabra（庞培法布拉大学）；University of Cambridge（剑桥大学）；Bar-Ilan University（巴伊兰大学）
This paper investigates a lossy source coding problem in which two decoders can access their side-information respectively. The correlated sources are a product of two component correlated sources, and we exclusively investigate the case such that each component is degraded. We show the rate-distortion function for that case, and give the following observations. When the components are degraded in matched order, the rate distortion function of the product sources is equal to the sum of the component-wise rate distortion functions. On the otherhand, the former is strictly smaller than the latte…
For a number of lossy source coding problems it is shown that even if the usual single-letter sum-rate-distortion expressions may become invalid for non-infinite distortion functions, they can be approached, to any desired accuracy, via the usual valid expressions for appropriately truncated finite versions of the distortion functions.
In this paper, we consider a class of multiterminal source coding problems, each subject to distortion constraints computed using a specific, entropy-based, distortion measure. We provide the achievable rate distortion region for two cases and, in so doing, we demonstrate a relationship between the lossy multiterminal source coding problems with our specific distortion measure and (1) the canonical Slepian-Wolf lossless distributed source coding network, and (2) the Ahlswede-Körner-Wyner source coding with side information problem in which only one of the sources is recovered losslessly.
Blau and Michaeli recently introduced a novel concept for inverse problems of signal processing, that is, the perception-distortion tradeoff. We introduce their tradeoff into the rate distortion theory of variable-length lossy source coding in information theory, and clarify the tradeoff among information rate, distortion and perception for general information sources. We also discuss the fixed-length coding with average distortion criterion that was missing in the previous letter.
We approach index coding as a special case of rate-distortion with multiple receivers, each with some side information about the source. Specifically, using techniques developed for the rate-distortion problem, we provide two upper bounds and one lower bound on the optimal index coding rate. The upper bounds involve specific choices of the auxiliary random variables in the best existing scheme for the rate-distortion problem. The lower bound is based on a new lower bound for the general rate-distortion problem. The bounds are shown to coincide for a number of (groupcast) index coding instances…
We consider the variable-to-fixed length lossy source coding (VFSC) problem. The optimal compression rate of the average length of variable-to-fixed source coding, allowing a non-vanishing probability of excess-distortion $\varepsilon$, is shown to be equal to $(1-\varepsilon)R(D)$, where $R(D)$ is the rate-distortion function of the source. In comparison to the related results of Koga and Yamamoto as well as Kostina, Polyanskiy, and Verdú for fixed-to-variable length source coding, our results demonstrate an interesting feature that variable-to-fixed length source coding has the same first-or…
This paper considers the problem of covert communication with mismatched decoding, in which a sender wishes to reliably communicate with a receiver whose decoder is fixed and possibly sub-optimal, and simultaneously to ensure that the communication is covert with respect to a warden. We present single-letter lower and upper bounds on the information-theoretically optimal throughput as a function of the given decoding metric, channel laws, and the desired level of covertness. These bounds match for a variety of scenarios of interest, such as (i) when the channel between the sender and receiver …
Shannon proved that if we can transmit bits reliably at rates larger than the rate distortion function $R(D)$, then we can transmit this source to within a distortion $D$. We answer the converse question ``If we can transmit a source to within a distortion $D$, can we transmit bits reliably at rates less than the rate distortion function?'' in the affirmative. This can be viewed as a direct converse of the rate distortion theorem.
This paper presents a one shot analysis of the lossy compression problem under average distortion constraints. We calculate the exact expected distortion of a random code. The result is given as an integral formula using a newly defined functional $\tilde{D}(z,Q_Y)$ where $Q_Y$ is the random coding distribution and $z\in [0,1]$. When we plug in the code distribution as $Q_Y$, this functional produces the average distortion of the code, thus provide a converse result utilizing the same functional. Two alternative formulas are provided for $\tilde{D}(z,Q_Y)$, the first involves a supremum over s…
Minimum energy required to achieve a distortion-noise profile, i.e., a function indicating the maximum allowed distortion value for each channel noise level. In this paper, the minimum energy required to achieve a distortion noise profile is studied for Gaussian sources which are transmitted robustly over Gaussian channels. We provide upper bound for the minimum energy behavior of the staircase profile using our proposed coding scheme. Conversely, utilizing a family of lower bounds originally derived for broadcast channels with power constraints, the minimum required energy is lower bounded fo…
A single-letter lower bound on the sum rate of multiple description coding with tree-structured distortion constraints is established by generalizing Ozarow's celebrated converse argument through the introduction of auxiliary random variables that form a Markov tree. For the quadratic vector Gaussian case, this lower bound is shown to be achievable by an extended version of the El Gamal-Cover scheme, yielding a complete sum-rate characterization.
In this paper, we present iterative algorithms that numerically compute the rate-distortion regions of two problems: the two-encoder multiterminal source coding problem and the Chief Executive Officer (CEO) problem, both under logarithmic loss distortion measure.With the clear connection of these models with the distributed information bottleneck method, the proposed algorithms may find usefulness in a variety of applications, such as clustering, pattern recognition and learning. We illustrate the efficiency of our algorithms through some numerical examples.
A multiple-descriptions (MD) coding strategy is proposed and an inner bound to the achievable rate-distortion region is derived. The scheme utilizes linear codes. It is shown in two different MD set-ups that the linear coding scheme achieves a larger rate-distortion region than previously known random coding strategies. Furthermore, it is shown via an example that the best known random coding scheme for the set-up can be improved by including additional randomly generated codebooks.
We consider a source coding problem with a network scenario in mind, and formulate it as a remote vector Gaussian Wyner-Ziv problem under covariance matrix distortions. We define a notion of minimum for two positive-definite matrices based on which we derive an explicit formula for the rate-distortion function (RDF). We then study the special cases and applications of this result. We show that two well-studied source coding problems, i.e. remote vector Gaussian Wyner-Ziv problems with mean-squared error and mutual information constraints are in fact special cases of our results. Finally, we ap…
We consider the rate distortion problem with side information at the decoder posed and investigated by Wyner and Ziv. The rate distortion function indicating the trade-off between the rate on the data compression and the quality of data obtained at the decoder was determined by Wyner and Ziv. In this paper, we study the error probability of decoding at rates below the rate distortion function. We evaluate the probability of decoding such that the estimation of source outputs by the decoder has a distortion not exceeding a prescribed distortion level. We prove that when the rate of the data com…
We consider the classic joint source-channel coding problem of transmitting a memoryless source over a memoryless channel. The focus of this work is on the long-standing open problem of finding the rate of convergence of the smallest attainable expected distortion to its asymptotic value, as a function of blocklength $n$. Our main result is that in general the convergence rate is not faster than $n^{-1/2}$. In particular, we show that for the problem of transmitting i.i.d uniform bits over a binary symmetric channels with Hamming distortion, the smallest attainable distortion (bit error rate) …
An information-spectrum approach is applied to solve the multiterminal source coding problem for correlated general sources, where sources may be nonstationary and/or nonergodic, and the distortion measure is arbitrary and may be nonadditive. A general formula for the rate-distortion region of the multiterminal source coding problem with the maximum distortion criterion under fixed-length coding is shown in this correspondence.
We consider lossy source coding when side information affecting the distortion measure may be available at the encoder, decoder, both, or neither. For example, such distortion side information can model reliabilities for noisy measurements, sensor calibration information, or perceptual effects like masking and sensitivity to context. When the distortion side information is statistically independent of the source, we show that in many cases (e.g, for additive or multiplicative distortion side information) there is no penalty for knowing the side information only at the encoder, and there is no …
We consider the problem of multiple descriptions (MD) source coding and propose new coding strategies involving both unstructured and structured coding layers. Previously, the most general achievable rate-distortion (RD) region for the $l$-descriptions problem was the Combinatorial Message Sharing with Binning (CMSB) region. The CMSB scheme utilizes unstructured quantizers and unstructured binning. In the first part of the paper, we show that this strategy can be improved upon using more general unstructured quantizers and a more general unstructured binning method. In the second part, structu…
This paper characterizes the second-order coding rates for lossy source coding with side information available at both the encoder and the decoder. We first provide non-asymptotic bounds for this problem and then specialize the non-asymptotic bounds for three different scenarios: discrete memoryless sources, Gaussian sources, and Markov sources. We obtain the second-order coding rates for these settings. It is interesting to observe that the second-order coding rate for Gaussian source coding with Gaussian side information available at both the encoder and the decoder is the same as that for G…
We investigate the Wyner-Ziv coding in which the statistics of the principal source is known but the statistics of the channel generating the side-information is unknown except that it is in a certain class. The class consists of channels such that the distortion between the principal source and the side-information is smaller than a threshold, but channels may be neither stationary nor ergodic. In this situation, we define a new rate-distortion function as the minimum rate such that there exists a Wyner-Ziv code that is universal for every channel in the class. Then, we show an upper bound an…
A randomized covering-packing duality between source and channel coding will be discussed by considering the source coding problem of coding a source with a certain distortion level and by considering a channel which communicates the source within a certain distortion level. An operational view of source-channel separation for communication with a fidelity criterion will be discussed in brief.
In this paper, we consider a distributed remote source coding problem, where a sequence of observations of source vectors is available at the encoder. The problem is to specify the optimal rate for encoding the observations subject to a covariance matrix distortion constraint and in the presence of side information at the decoder. For this problem, we derive lower and upper bounds on the rate-distortion function (RDF) for the Gaussian case, which in general do not coincide. We then provide some cases, where the RDF can be derived exactly. We also show that previous results on specific instance…
This work investigates functional source coding problems with maximal distortion, motivated by approximate function computation in many modern applications. The maximal distortion treats imprecise reconstruction of a function value as good as perfect computation if it deviates less than a tolerance level, while treating reconstruction that differs by more than that level as a failure. Using a geometric understanding of the maximal distortion, we propose a hypergraph-based source coding scheme for function computation that is constructive in the sense that it gives an explicit procedure for fin…
We consider a novel variant of $d$-semifaithful lossy coding in which the distortion measure is revealed only to the encoder and only at run-time, as well as an extension of it in which the distortion constraint $d$ is also revealed at run-time. Two forms of rate redundancy are used to analyze the performance, and achievability results of both a pointwise and minimax nature are demonstrated. The first coding scheme uses ideas from VC dimension and growth functions, the second uses appropriate quantization of the space of distortion measures, and the third relies on a random coding argument.
Consider a generalized multiterminal source coding system, where $\ell\choose m$ encoders, each observing a distinct size-$m$ subset of $\ell$ ($\ell\geq 2$) zero-mean unit-variance symmetrically correlated Gaussian sources with correlation coefficient $ρ$, compress their observations in such a way that a joint decoder can reconstruct the sources within a prescribed mean squared error distortion based on the compressed data. The optimal rate-distortion performance of this system was previously known only for the two extreme cases $m=\ell$ (the centralized case) and $m=1$ (the distributed case)…
Minimum power required to achieve a distortion-noise profile, i.e., a function indicating the maximum allowed distortion value for each noise level, is studied for the transmission of Gaussian sources over Gaussian channels under a regime of bandwidth approaching zero. A simple but instrumental lower bound to the minimum required power for a given profile is presented. For an upper bound, a dirty-paper based coding scheme is proposed and its power-distortion tradeoff is analyzed. Finally, upper and lower bounds to the minimum power is analyzed and compared for specific distortion-noise profile…
The source-coding problem with side information at the decoder is studied subject to a constraint that the encoder---to whom the side information is unavailable---be able to compute the decoder's reconstruction sequence to within some distortion. For discrete memoryless sources and finite single-letter distortion measures, an expression is given for the minimal description rate as a function of the joint law of the source and side information and of the allowed distortions at the encoder and at the decoder. The minimal description rate is also computed for a memoryless Gaussian source with squ…
In this paper, distributed (or multiterminal) source coding with one distortion criterion and correlated messages is considered. This problem can be also called ``Berger-Yeung problem with correlated messages''. It corresponds to the source coding part of the graph-based framework for transmission of a pair of correlated sources over the multiple-access channel (MAC) where one is lossless and the other is lossy. As a result, the achievable rate-distortion region for this problem is provided. It is an information-theoretic characterization of the rate of exponential growth (as a function of the…
This paper investigates a source coding problem in which two terminals communicating through a relay wish to estimate one another's source within some distortion constraint. The relay has access to side information that is correlated with the sources. Two different schemes based on the order of communication, \emph{distributed source coding/delivery} and \emph{two cascaded rounds}, are proposed and inner and outer bounds for the resulting rate-distortion regions are provided. Examples are provided to show that neither rate-distortion region includes the other one.
In this work we investigate the behavior of the minimal rate needed in order to guarantee a given probability that the distortion exceeds a prescribed threshold, at some fixed finite quantization block length. We show that the excess coding rate above the rate-distortion function is inversely proportional (to the first order) to the square root of the block length. We give an explicit expression for the proportion constant, which is given by the inverse Q-function of the allowed excess distortion probability, times the square root of a constant, termed the excess distortion dispersion. This re…
This paper quantifies the fundamental limits of variable-length transmission of a general (possibly analog) source over a memoryless channel with noiseless feedback, under a distortion constraint. We consider excess distortion, average distortion and guaranteed distortion ($d$-semifaithful codes). In contrast to the asymptotic fundamental limit, a general conclusion is that allowing variable-length codes and feedback leads to a sizable improvement in the fundamental delay-distortion tradeoff. In addition, we investigate the minimum energy required to reproduce $k$ source samples with a given f…
In steganography, we always hope to maximize the embedding payload subject to an upper-bounded distortion. We need suitable distortion measurement to evaluate the embedding impact. However, different distortion functions exposes different levels of distortion evaluation, implying that we have different optimization distributions by applying different distortion functions. In applications, the embedding distortion is caused by a certain number of embedding operations. Instead of a predefined distortion, we actually utilize a number of modifications to embed as many message bits as possible as l…
In this work, we consider a distributed source coding problem with a joint distortion criterion depending on the sources and the reconstruction. This includes as a special case the problem of computing a function of the sources to within some distortion and also the classic Slepian-Wolf problem, Berger-Tung problem, Wyner-Ziv problem, Yeung-Berger problem and the Ahlswede-Korner-Wyner problem. While the prevalent trend in information theory has been to prove achievability results using Shannon's random coding arguments, using structured random codes offer rate gains over unstructured random co…
Let (S1,i, S2,i), distributed according to i.i.d p(s1, s2), i = 1, 2, . . . be a memoryless, correlated partial side information sequence. In this work we study channel coding and source coding problems where the partial side information (S1, S2) is available at the encoder and the decoder, respectively, and, additionally, either the encoder's or the decoder's side information is increased by a limited-rate description of the other's partial side information. We derive six special cases of channel coding and source coding problems and we characterize the capacity and the rate-distortion functi…
We show that polar codes can be used to achieve the rate-distortion functions in the problem of hierarchical source coding also known as the successive refinement problem. We also analyze the distributed version of this problem, constructing a polar coding scheme that achieves the rate distortion functions for successive refinement with side information.
We introduce the idea of distortion side information, which does not directly depend on the source but instead affects the distortion measure. We show that such distortion side information is not only useful at the encoder, but that under certain conditions, knowing it at only the encoder is as good as knowing it at both encoder and decoder, and knowing it at only the decoder is useless. Thus distortion side information is a natural complement to the signal side information studied by Wyner and Ziv, which depends on the source but does not involve the distortion measure. Furthermore, when both…
A distortion-noise profile is a function indicating the maximum allowed source distortion value for each noise level in the channel. In this paper, the minimum energy required to achieve a distortion noise profile is studied for Gaussian sources which are transmitted robustly over Gaussian channels. We provide improved lower and upper bounds for the minimum energy behavior of the square-law profile using a family of lower bounds and our proposed coding scheme.
We consider the classical two-encoder multiterminal source coding problem where distortion is measured under logarithmic loss. We provide a single-letter characterization of the achievable rate distortion region for arbitrarily correlated sources with finite alphabets. In doing so, we also give the rate distortion region for the $m$-encoder CEO problem (also under logarithmic loss). Several applications and examples are given.
In this paper, we study the asymptotic performance of Abelian group codes for the lossy source coding problem for arbitrary discrete (finite alphabet) memoryless sources as well as the channel coding problem for arbitrary discrete (finite alphabet) memoryless channels. For the source coding problem, we derive an achievable rate-distortion function that is characterized in a single-letter information-theoretic form using the ensemble of Abelian group codes. When the underlying group is a field, it simplifies to the symmetric rate-distortion function. Similarly, for the channel coding problem, w…
In this work, the problem of transmitting an i.i.d Gaussian source over an i.i.d Gaussian wiretap channel with an i.i.d Gaussian side information available at the intended receiver is considered. The intended receiver is assumed to have a certain minimum SNR and the eavesdropper is assumed to have a strictly lower SNR, compared to the intended receiver. The objective is to minimize the distortion of source reconstruction at the intended receiver. In this work, it is shown that the source-channel separation coding scheme is optimum in the sense of achieving minimum distortion. Two hybrid digita…
