Information-Theoretic Foundations of Mismatched Decoding
National University of Singapore（新加坡国立大学）；Institució Catalana de Recerca i Estudis Avançats (ICREA)（加泰罗尼亚高级研究学院）；Universitat Pompeu Fabra（庞培法布拉大学）；University of Cambridge（剑桥大学）；Bar-Ilan University（巴伊兰大学）
Jonathan Scarlett, Albert Guillén i Fàbregas, Anelia Somekh-Baruch, Alfonso Martinez
备注：
Published in Foundations and Trends in Communications and Information Theory (Volume 17, Issue 2-3). arXiv v5 fixes several minor typos/errors and also two more substantial issues in Sections 5.6 and 3.8.3 (see the Acknowledgments section for details)
Shannon's channel coding theorem characterizes the maximal rate of information that can be reliably transmitted over a communication channel when optimal encoding and decoding strategies are used. In many scenarios, however, practical considerations such as channel uncertainty and implementation constraints rule out the use of an optimal decoder. The mismatched decoding problem addresses such scenarios by considering the case that the decoder cannot be optimized, but is instead fixed as part of the problem statement. This problem is not only of direct interest in its own right, but also has close connections with other long-standing theoretical problems in information theory. In this monograph, we survey both classical literature and recent developments on the mismatched decoding problem, with an emphasis on achievable random-coding rates for memoryless channels. We present two widely-considered achievable rates known as the generalized mutual information (GMI) and the LM rate, and overview their derivations and properties. In addition, we survey several improved rates via multi-user coding techniques, as well as recent developments and challenges in establishing upper bounds on the mismatch capacity, and an analogous mismatched encoding problem in rate-distortion theory. Throughout the monograph, we highlight a variety of applications and connections with other prominent information theory problems.
We derive an upper bound on the reliability function of mismatched decoding for zero-rate codes. The bound is based on a result by Komlós that shows the existence of a subcode with certain symmetry properties. The bound is shown to coincide with the expurgated exponent at rate zero for a broad family of channel-decoding metric pairs.
Shaping gain is attained in schemes where a shaped subcode is chosen from a larger codebook by a codeword selection process. This includes the popular method of Trellis Shaping (TS), originally proposed by Forney for average power reduction. The decoding process of such schemes is mismatched, since it is aware of only the large codebook. This study models such schemes by a random code construction and derives achievable bounds on the transmission rate under matched and mismatched decoding. For matched decoding the bound is obtained using a modified asymptotic equipartition property (AEP) theor…
This paper studies the mismatched decoding problem for binary-input discrete memoryless channels. An example is provided for which an achievable rate based on superposition coding exceeds the LM rate (Hui, 1983; Csiszár-Körner, 1981), thus providing a counter-example to a previously reported converse result (Balakirsky, 1995). Both numerical evaluations and theoretical results are used in establishing this claim.
We consider an ensemble of constant composition codes that are subsets of linear codes: while the encoder uses only the constant-composition subcode, the decoder operates as if the full linear code was used, with the motivation of simultaneously benefiting both from the probabilistic shaping of the channel input and from the linear structure of the code. We prove that the codebook mismatch can be fully compensated by using a mismatched additive decoding metric that achieves the random coding error exponent of (non-linear) constant composition codes. As the coding rate tends to the mutual infor…
We derive a sphere-packing error exponent for coded transmission over discrete memoryless channels with a fixed decoding metric. By studying the error probability of the code over an auxiliary channel, we find a lower bound to the probability of error of mismatched decoding. The bound is shown to decay exponentially for coding rates smaller than a new upper bound to the mismatch capacity which is established in this paper. For rates higher than the new upper bound, the error probability is shown to be bounded away from zero. The new upper bound is shown to improve over previous upper bounds to…
This paper studies multiuser random coding techniques for channel coding with a given (possibly suboptimal) decoding rule. For the mismatched discrete memoryless multiple-access channel, an error exponent is obtained that is tight with respect to the ensemble average, and positive within the interior of Lapidoth's achievable rate region. This exponent proves the ensemble tightness of the exponent of Liu and Hughes in the case of maximum-likelihood decoding. An equivalent dual form of Lapidoth's achievable rate region is given, and the latter is shown to extend immediately to channels with infi…
Arıkan's polar coding, is by now a well studied technique that allows achieving the symmetric capacity of binary input memoryless channels with low complexity encoding and decoding, provided that the polar decoding architecture is used and the decoding metric is matched to the true channel. In this paper, we analyze communication rates that are achievable when the polar coding/decoding architecture is used with the decoder using an incorrect model of the channel. We define the `polar mismatched capacity' as an analogue of the classical mismatched capacity, give an expression for it, and derive…
Hardware impairments in radio-frequency components of a wireless system cause unavoidable distortions to transmission that are not captured by the conventional linear channel model. In this paper, a 'binoisy' single-user multiple-input multiple-output (SU-MIMO) relation is considered where the additional distortions are modeled via an additive noise term at the transmit side. Through this extended SU-MIMO channel model, the effects of transceiver hardware impairments on the achievable rate of multi-antenna point-to-point systems are studied. Channel input distributions encompassing practical d…
This paper studies channel coding for the discrete memoryless multiple-access channel with a given (possibly suboptimal) decoding rule. A multi-letter successive decoding rule depending on an arbitrary non-negative decoding metric is considered, and achievable rate regions and error exponents are derived both for the standard MAC (independent codebooks), and for the cognitive MAC (one user knows both messages) with superposition coding. In the cognitive case, the rate region and error exponent are shown to be tight with respect to the ensemble average. The rate regions are compared with those …
We revisit the information-theoretic analysis of bit-interleaved coded modulation (BICM) by modeling the BICM decoder as a mismatched decoder. The mismatched decoding model is well-defined for finite, yet arbitrary, block lengths, and naturally captures the channel memory among the bits belonging to the same symbol. We give two independent proofs of the achievability of the BICM capacity calculated by Caire et al. where BICM was modeled as a set of independent parallel binary-input channels whose output is the bitwise log-likelihood ratio. Our first achievability proof uses typical sequences, …
The problem of mismatched decoding for discrete memoryless channels is addressed. A mismatched cognitive multiple-access channel is introduced, and an inner bound on its capacity region is derived using two alternative encoding methods: superposition coding and random binning. The inner bounds are derived by analyzing the average error probability of the code ensemble for both methods and by a tight characterization of the resulting error exponents. Random coding converse theorems are also derived. A comparison of the achievable regions shows that in the matched case, random binning performs a…
This paper considers the problem of covert communication with mismatched decoding, in which a sender wishes to reliably communicate with a receiver whose decoder is fixed and possibly sub-optimal, and simultaneously to ensure that the communication is covert with respect to a warden. We present single-letter lower and upper bounds on the information-theoretically optimal throughput as a function of the given decoding metric, channel laws, and the desired level of covertness. These bounds match for a variety of scenarios of interest, such as (i) when the channel between the sender and receiver …
We extend the notion of list decoding to {\em ratio list decoding} which involves a list decoder whose list size is specified as a function of the number of messages $M_n$ and the block length $n$. We present necessary and sufficient conditions on $M_n$ for the existence of code sequences which enable reliable list decoding with respect to the desired list size $L(M_n,n)$. It is shown that the ratio-capacity, defined as the supremum of achievable normalized logarithms of the ratio $r(M_n,n)=M_n/L(M_n,n)$ is equal to the Shannon channel capacity $C$, for both stochastic and deterministic encodi…
Matched and Euclidean-Mismatched Decoding on Fourier-Curve Constellations with Tangent Noise
RPTU University Kaiserslautern-Landau（莱布尼茨普法尔茨技术大学凯泽斯劳滕-兰道校区）；ShanghaiTech University（上海科技大学）；Tsinghua University（清华大学）；Princeton University（普林斯顿大学）；German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心）
This paper considers the problem of channel coding with a given (possibly suboptimal) maximum-metric decoding rule. A cost-constrained random-coding ensemble with multiple auxiliary costs is introduced, and is shown to achieve error exponents and second-order coding rates matching those of constant-composition random coding, while being directly applicable to channels with infinite or continuous alphabets. The number of auxiliary costs required to match the error exponents and second-order rates of constant-composition coding is studied, and is shown to be at most two. For i.i.d. random coding…
Achievable and converse bounds for general channels and mismatched decoding are derived. The direct (achievable) bound is derived using random coding and the analysis is tight up to factor 2. The converse is given in term of the achievable bound and the factor between them is given. This gives performance of the best rate-R code with possible mismatched decoding metric over a general channel, up to the factor that is identified. In the matched case we show that the converse equals the minimax meta-converse of Polyanskiy et al.
The problem of mismatched decoding with an additive metric $q$ for a discrete memoryless channel $W$ is addressed. The "product-space" improvement of the random coding lower bound on the mismatch capacity, $C_q^{(\infty)}(W)$, was introduced by Csiszár and Narayan. We study two kinds of decoders. The {\it $δ$-margin mismatched decoder} outputs a message whose metric with the channel output exceeds that of all the other codewords by at least $δ$. The {\it $τ$-threshold decoder} outputs a single message whose metric with the channel output exceeds a threshold $τ$. Both decoders declare an error …
It is well known that for Gaussian channels, a nearest neighbor decoding rule, which seeks the minimum Euclidean distance between a codeword and the received channel output vector, is the maximum likelihood solution and hence capacity-achieving. Nearest neighbor decoding remains a convenient and yet mismatched solution for general channels, and the key message of this paper is that the performance of the nearest neighbor decoding can be improved by generalizing its decoding metric to incorporate channel state dependent output processing and codeword scaling. Using generalized mutual informatio…
Two channels are equivalent if their maximum likelihood (ML) decoders coincide for every code. We show that this equivalence relation partitions the space of channels into a generalized hyperplane arrangement. With this, we define a coding distance between channels in terms of their ML-decoders which is meaningful from the decoding point of view, in the sense that the closer two channels are, the larger is the probability of them sharing the same ML-decoder. We give explicit formulas for these probabilities.
We derive a single-letter upper bound to the mismatched-decoding capacity for discrete memoryless channels. The bound is expressed as the mutual information of a transformation of the channel, such that a maximum-likelihood decoding error on the translated channel implies a mismatched-decoding error in the original channel. In particular, a strong converse is shown to hold for this upper-bound: if the rate exceeds the upper-bound, the probability of error tends to 1 exponentially when the block-length tends to infinity. We also show that the underlying optimization problem is a convex-concave …
The rising interest in applications requiring the transmission of small amounts of data has recently lead to the development of accurate performance bounds and of powerful channel codes for the transmission of short-data packets over the AWGN channel. Much less is known about the interaction between error control coding and channel estimation at short blocks when transmitting over channels with states (e.g., fading channels, phase-noise channels, etc...) for the setup where no a priori channel state information (CSI) is available at the transmitter and the receiver. In this paper, we use the m…
This paper considers the problem of channel coding over Gaussian intersymbol interference (ISI) channels with a given metric decoding rule. Specifically, it is assumed that the mismatched decoder has an incorrect assumption on the impulse response function. The mismatch capacity is the highest achievable rate for a given decoding rule. Existing lower bounds to the mismatch capacity for channels and decoding metrics with memory (as in our model) are presented only in the form of multi-letter expressions that have not been calculated in practice. Consequently, they provide little insight on the …
A Lagrange-dual (Gallager-style) lower bound is derived for the error exponent function of the typical random code (TRC) pertaining to the i.i.d. random coding ensemble and mismatched stochastic likelihood decoding. While the original expression, derived from the method of types (the Csiszar-style expression) involves minimization over probability distributions defined on the channel input--output alphabets, the new Lagrange-dual formula involves optimization of five parameters, independently of the alphabet sizes. For both stochastic and deterministic mismatched decoding (including maximum li…
We develop a novel framework for proving converse theorems for channel coding, which is based on the analysis technique of multicast transmission with an additional auxiliary receiver, which serves as a genie to the original receiver. The genie provides the original receiver a certain narrowed list of codewords to choose from that includes the transmitted one. This technique is used to derive upper bounds on the mismatch capacity of discrete memoryless channels as well as the reliability function with a mismatched decoding metric. Unlike previous works, our bounding technique exploits also the…
We consider a joint source channel coding (JSCC) problem in which we desire to transmit an arbitrary memoryless source over an arbitrary additive channel. We propose a mismatched coding architecture that consists of Gaussian codebooks for both the source reproduction sequences and channel codewords. The natural nearest neighbor encoder and decoder, however, need to be judiciously modified to obtain the highest communication rates at finite blocklength. In particular, we consider a unequal error protection (UEP) scheme in which all sources are partitioned into disjoint power type classes. We al…
Reliable communication over delay-constrained block-fading channels with discrete inputs and mismatched (imperfect) channel state information at the transmitter (CSIT) is studied. The CSIT mismatch is modeled as Gaussian random variables, whose variances decay as a power of the signal-to-noise ratio (SNR). A special focus is placed on the large-SNR decay of the outage probability when power control with long-term power constraints is used. Without explicitly characterizing the corresponding power allocation algorithms, we derive the outage exponent as a function of the system parameters, inclu…
The common approach of designing a communication device is to maximize a well-defined objective function, e.g., the channel capacity and the cut-off rate. We propose easy-to-implement solutions for Gaussian channels that approximate the optimal results for these maximization problems. Three topics are addressed. First, we consider the case where the channel output is quantized, and we find the quantization thresholds that maximize the mutual information. The approximation derived from the asymptotic solution has a negligible loss on the entire range of SNR when 2-PAM modulation is used, and it…
The fundamental limits of channels with mismatched decoding are addressed. A general formula is established for the mismatch capacity of a general channel, defined as a sequence of conditional distributions with a general decoding metrics sequence. We deduce an identity between the Verdú-Han general channel capacity formula, and the mismatch capacity formula applied to Maximum Likelihood decoding metric. Further, several upper bounds on the capacity are provided, and a simpler expression for a lower bound is derived for the case of a non-negative decoding metric. The general formula is special…
We consider the problem of universal decoding for arbitrary unknown channels in the random coding regime. For a given random coding distribution and a given class of metric decoders, we propose a generic universal decoder whose average error probability is, within a sub-exponential multiplicative factor, no larger than that of the best decoder within this class of decoders. Since the optimum, maximum likelihood (ML) decoder of the underlying channel is not necessarily assumed to belong to the given class of decoders, this setting suggests a common generalized framework for: (i) mismatched deco…
The error exponent of the typical random code is defined as the asymptotic normalized expectation of the logarithm of the probability of error, as opposed to the traditional definition of the random coding exponent as the normalized logarithm of the expectation of the probability of error with respect to a given ensemble of codes. For a certain ensemble of independent codewords, with a given power spectrum, and a generalized stochastic mismatched decoder, we characterize the error exponent the typical random codes (TRC) for the colored Gaussian channel, with emphasis on the range of low rates,…
This paper studies the subexponential prefactor to the random-coding bound for a given rate. Using a refinement of Gallager's bounding techniques, an alternative proof of a recent result by Altuğ and Wagner is given, and the result is extended to the setting of mismatched decoding.
A challenging problem related to the design of polar codes is "robustness against channel parameter variations" as stated in Arıkan's original work. In this paper, we describe how the problem of robust polar code design can be viewed as a mismatch decoding problem. We propose conditions which ensure a polar encoder/decoder designed for a mismatched B-DMC can be used to communicate reliably. In particular, the analysis shows that the original polar code construction method is robust over the class of binary symmetric channels.
The optimal decoder achieving the outage capacity under imperfect channel estimation is investigated. First, by searching into the family of nearest neighbor decoders, which can be easily implemented on most practical coded modulation systems, we derive a decoding metric that minimizes the average of the transmission error probability over all channel estimation errors. This metric, for arbitrary memoryless channels, achieves the capacity of a composite (more noisy) channel. Next, according to the notion of estimation-induced outage capacity (EIO capacity) introduced in our previous work, we c…
The optimal decoder achieving the outage capacity under imperfect channel estimation is investigated. First, by searching into the family of nearest neighbor decoders, which can be easily implemented on most practical coded modulation systems, we derive a decoding metric that minimizes the average of the transmission error probability over all channel estimation errors. Next, we specialize our general expression to obtain the corresponding decoding metric for fading MIMO channels. According to the notion of estimation-induced outage (EIO) capacity introduced in our previous work and assuming n…
Quantum cryptography via key distribution mechanisms that utilize quantum entanglement between sender-receiver pairs will form the basis of future large-scale quantum networks. A key engineering challenge in such networks will be the ability to correct for decoherence effects in the distributed entanglement resources. It is widely believed that sophisticated quantum error correction codes, such as quantum low-density parity-check (LDPC) codes, will be pivotal in such a role. However, recently the importance of the channel mismatch effect in degrading the performance of deployed quantum LDPC co…
Probabilistic Amplitude Shaping (PAS) is a coded-modulation scheme in which the encoder is a concatenation of a distribution matcher with a systematic Forward Error Correction (FEC) code. For reduced computational complexity the decoder can be chosen as a concatenation of a mismatched FEC decoder and dematcher. This work studies the theoretic limits of PAS. The classical joint source-channel coding (JSCC) setup is modified to include systematic FEC and the mismatched FEC decoder. At each step error exponents and achievable rates for the corresponding setup are derived.
We investigate the problem of strategic point-to-point communication with side information at the decoder, in which the encoder and the decoder have mismatched distortion functions. The decoding process is not supervised, it returns the output sequence that minimizes the decoder's distortion function. The encoding process is designed beforehand and takes into account the decoder's distortion mismatch. When the communication channel is perfect and no side information is available at the decoder, this problem is referred to as the Bayesian persuasion game of Kamenica-Gentzkow in the Economics li…
Generative Decompression: Optimal Lossy Decoding Against Distribution Mismatch
Nokia Bell Laboratories（诺基亚贝尔实验室）；Wireless Intelligence Laboratory（无线智能实验室）；School of Electrical, Computer and Energy Engineering（电气、计算机与能源工程学院）；Arizona State University（亚利桑那州立大学）
Accumulating evidence indicates that the capacity to integrate information in the brain is a prerequisite for consciousness. Integrated Information Theory (IIT) of consciousness provides a mathematical approach to quantifying the information integrated in a system, called integrated information, $Φ$. Integrated information is defined theoretically as the amount of information a system generates as a whole, above and beyond the sum of the amount of information its parts independently generate. IIT predicts that the amount of integrated information in the brain should reflect levels of conscious…
This work identifies information-theoretic quantities that are closely related to the required list size on average for successive cancellation list (SCL) decoding to implement maximum-likelihood decoding over general binary memoryless symmetric (BMS) channels. It also provides upper and lower bounds for these quantities that can be computed efficiently for very long codes. For the binary erasure channel (BEC), we provide a simple method to estimate the mean accurately via density evolution. The analysis shows how to modify, e.g., Reed-Muller codes, to improve the performance when practical li…
We investigate the probability that the peak age of information in a point-to-point communication system operating over a multiantenna wireless fading channel exceeds a predetermined value. The packets are scheduled according to a last-come first-serve policy with preemption in service, and are transmitted over the channel using a simple automatic repetition request protocol. We consider quadrature phase shift keying modulation, pilot-assisted transmission, maximum-likelihood channel estimation, and mismatched scaled nearest-neighbor decoding. Our analysis, which exploits nonasymptotic tools i…
The paper is focused on the tradeoff between performance and decoding complexity per iteration for LDPC codes in terms of their gap (in rate) to capacity. The study of this tradeoff is done via information-theoretic bounds which also enable to get an indication on the sub-optimality of message-passing iterative decoding algorithms (as compared to optimal ML decoding). The bounds are generalized for parallel channels, and are applied to ensembles of punctured LDPC codes where both intentional and random puncturing are addressed. This work suggests an improvement in the tightness of some informa…
The errors occurring in DNA-based storage are correlated in nature, which is a direct consequence of the synthesis and sequencing processes. In this paper, we consider the memory-$k$ nanopore channel model recently introduced by Hamoum et al., which models the inherent memory of the channel. We derive the maximum a posteriori (MAP) decoder for this channel model. The derived MAP decoder allows us to compute achievable information rates for the true DNA storage channel assuming a mismatched decoder matched to the memory-$k$ nanopore channel model, and quantify the loss in performance assuming a…
We consider the problem of guessing the realization of a random variable but under more general Tsallis' non-extensive entropic framework rather than the classical Maxwell-Boltzman-Gibbs-Shannon framework. We consider both the conditional guessing problem in the presence of some related side information, and the unconditional one where no such side-information is available. For both types of the problem, the non-extensive moment bounds of the required number of guesses are derived; here we use the $q$-normalized expectation in place of the usual (linear) expectation to define the non-extensive…
A comparison between the joint and the successive decoding schemes for a two-link case binary Chief Executive Officer (CEO) problem is presented. We utilize the logarithmic loss as the criterion for measuring and comparing the total distortion of the decoding techniques. The binary symmetric channel (BSC) is considered as the test-channel model, which provides a theoretical rate-distortion bound. Simulation results show that the joint decoding scheme provides smaller compression rate and distortion values compared to the successive decoding scheme. On the other hand the successive decoding met…
