We investigate the computation of Csiszar's bounds for the joint source-channel coding (JSCC) error exponent, E_J, of a communication system consisting of a discrete memoryless source and a discrete memoryless channel. We provide equivalent expressions for these bounds and derive explicit formulas for the rates where the bounds are attained. These equivalent representations can be readily computed for arbitrary source-channel pairs via Arimoto's algorithm. When the channel's distribution satisfies a symmetry property, the bounds admit closed-form parametric expressions. We then use our results to provide a systematic comparison between the JSCC error exponent E_J and the tandem coding error exponent E_T, which applies if the source and channel are separately coded. It is shown that E_T <= E_J <= 2E_T. We establish conditions for which E_J > E_T and for which E_J = 2E_T. Numerical examples indicate that E_J is close to 2E_T for many source-channel pairs. This gain translates into a power saving larger than 2 dB for a binary source transmitted over additive white Gaussian noise channels and Rayleigh fading channels with finite output quantization. Finally, we study the computation of the lossy JSCC error exponent under the Hamming distortion measure.
Noiseless coding theorem for finite stationary memoryless information sources is proved by using induction on the number of source symbols and the inequality of geometric and harmonic means.
This paper investigates the first- and second-order maximum achievable rates of codes with/without cost constraints for mixed {channels} whose channel law is characterized by a general mixture of (at most) uncountably many stationary and memoryless discrete channels. These channels are referred to as {mixed memoryless channels with general mixture} and include the class of mixed memoryless channels of finitely or countably memoryless channels as a special case. For mixed memoryless channels with general mixture, the first-order coding theorem which gives a formula for the $\varepsilon$-capacit…
In this paper, we give a distributed joint source channel coding scheme for arbitrary correlated sources for arbitrary point in the Slepian-Wolf rate region, and arbitrary link capacities using LDPC codes. We consider the Slepian-Wolf setting of two sources and one destination, with one of the sources derived from the other source by some correlation model known at the decoder. Distributed encoding and separate decoding is used for the two sources. We also give a distributed source coding scheme when the source correlation has memory to achieve any point in the Slepian-Wolf rate achievable reg…
Existing fixed-length feedback communication schemes are either specialized to particular channels (Schalkwijk--Kailath, Horstein), or apply to general channels but either have high coding complexity (block feedback schemes) or are difficult to analyze (posterior matching). This paper introduces a new fixed-length feedback coding scheme which achieves the capacity for all discrete memoryless channels, has an error exponent that approaches the sphere packing bound as the rate approaches the capacity, and has $O(n\log n)$ coding complexity. These benefits are achieved by judiciously combining fe…
We propose a new coding scheme for the discrete memoryless two-user multi-access channel (MAC) with rate-limited feedback. Our scheme combines ideas from the Venkataramanan-Pradhan scheme for perfect feedback with ideas from the Shaviv-Steinberg scheme for rate-limited feedback. Our achievable region includes the Shaviv-Steinberg achievable region and this inclusion can be strict. For general MACs and for sufficiently large feedback rates, our scheme outperforms the Shaviv-Steinberg scheme as it achieves the same rate region as the Venkataramanan-Pradhan scheme for perfect feedback (which cann…
A solution to the problem of asymptotically optimum perfect universal steganography of finite memoryless sources with a passive warden is provided, which is then extended to contemplate a distortion constraint. The solution rests on the fact that Slepian's Variant I permutation coding implements first-order perfect universal steganography of finite host signals with optimum embedding rate. The duality between perfect universal steganography with asymptotically optimum embedding rate and lossless universal source coding with asymptotically optimum compression rate is evinced in practice by show…
A polar coding scheme is introduced in this paper for the wire-tap channel. It is shown that the provided scheme achieves the entire rate-equivocation region for the case of symmetric and degraded wire-tap channel, where the weak notion of secrecy is assumed. For the particular case of binary erasure wire-tap channel, an alternative proof is given. The case of general non-degraded wire-tap channels is also considered.
Bounds on the reliability function for the discrete memoryless relay channel are derived using the method of types. Two achievable error exponents are derived based on partial decode-forward and compress-forward which are well-known superposition block-Markov coding schemes. The derivations require combinations of the techniques involved in the proofs of Csiszár-Körner-Marton's packing lemma for the error exponent of channel coding and Marton's type covering lemma for the error exponent of source coding with a fidelity criterion. The decode-forward error exponent is evaluated on Sato's relay c…
We recently showed in [1] the superiority of certain structured coding matrices ensembles (such as partial row-orthogonal) for sparse superposition codes when compared with purely random matrices with i.i.d. entries, both information-theoretically and under practical vector approximate message-passing decoding. Here we generalize this result to binary input channels under generalized vector approximate message-passing decoding [2].We focus on specific binary output channels for concreteness but our analysis based on the replica symmetric method from statistical physics applies to any memoryles…
We investigate properties of a channel coding scheme leading to the minimum-possible frame error ratio when transmitting over a memoryless channel with rate R>C. The results are compared to the well-known properties of a channel coding scheme leading to minimum bit error ratio. It is concluded that these two optimization requests are contradicting. A valuable application of the derived results is presented.
A distributed binary hypothesis testing (HT) problem involving two parties, one referred to as the observer and the other as the detector is studied. The observer observes a discrete memoryless source (DMS) and communicates its observations to the detector over a discrete memoryless channel (DMC). The detector observes another DMS correlated with that at the observer, and performs a binary HT on the joint distribution of the two DMS's using its own observed data and the information received from the observer. The trade-off between the type I error probability and the type II error-exponent of …
In this paper, we study the state-dependent two-user interference channel, where the state information is non-causally known at both transmitters but unknown to either of the receivers. We propose two coding schemes for the discrete memoryless case: simultaneous encoding for the sub-messages in the first one and superposition encoding in the second one, both with rate splitting and Gel'fand-Pinsker coding. The corresponding achievable rate regions are established.
We consider a distributed source coding problem of $L$ correlated Gaussian observations $Y_i, i=1,2,...,L$. We assume that the random vector $Y^{L}={}^{\rm t} (Y_1,Y_2,$ $...,Y_L)$ is an observation of the Gaussian random vector $X^K={}^{\rm t}(X_1,X_2,...,X_K)$, having the form $Y^L=AX^K+N^L ,$ where $A$ is a $L\times K$ matrix and $N^L={}^{\rm t}(N_1,N_2,...,N_L)$ is a vector of $L$ independent Gaussian random variables also independent of $X^K$. The estimation error on $X^K$ is measured by the distortion covariance matrix. The rate distortion region is defined by a set of all rate vectors f…
Second-order coding rate of channel coding is discussed for general sequence of channels. The optimum second-order transmission rate with a constant error constraint $ε$ is obtained by using the information spectrum method. We apply this result to the discrete memoryless case, the discrete memoryless case with a cost constraint, the additive Markovian case, and the Gaussian channel case with an energy constraint. We also clarify that the Gallager bound does not give the optimum evaluation in the second-order coding rate.
The problem of variable length and fixed-distortion universal source coding (or D-semifaithful source coding) for stationary and memoryless sources on countably infinite alphabets ($\infty$-alphabets) is addressed in this paper. The main results of this work offer a set of sufficient conditions (from weaker to stronger) to obtain weak minimax universality, strong minimax universality, and corresponding achievable rates of convergences for the worse-case redundancy for the family of stationary memoryless sources whose densities are dominated by an envelope function (or the envelope family) on $…
We develop a low-complexity coding scheme to achieve covert communications over binary-input discrete memoryless channels (BI-DMCs). We circumvent the impossibility of covert communication with linear codes by introducing non-linearity through the use of pulse position modulation (PPM) and multilevel coding (MLC). We show that the MLC-PPM scheme exhibits many appealing properties; in particular, the channel at a given index level remains stationary as the number of level increases, which allows one to use families of channel capacity- and channel resolvability-achieving codes to concretely ins…
Coding schemes for discrete memoryless multicast networks (DM-MN) with rate-limited feedback from the receivers and relays to the transmitter are proposed. The schemes improve over the noisy network coding proposed by Lim et al.. For the single relay channel with relay-transmitter feedback, our coding schemes recover Gabbai and Bross's results, and strictly improve on noisy network coding, distributed decode-forward coding proposed by Lim et al., and all known lower bounds on the achievable rates proposed for the setup without feedback. The coding schemes are based on block-Markov coding, supe…
Physical-layer Network Coding (PNC) can significantly improve the throughput of two-way relay channels. An interesting variant of PNC is Analog Network Coding (ANC). Almost all ANC schemes proposed to date, however, operate in a symbol by symbol manner (memoryless) and cannot exploit the redundant information in channel-coded packets to enhance performance. This paper proposes a non-memoryless ANC scheme. In particular, we design a soft-input soft-output decoder for the relay node to process the superimposed packets from the two end nodes to yield an estimated MMSE packet for forwarding back t…
In this paper we prove the separation of source-network coding and channel coding in wireline networks. For the purposes of this work, a wireline network is any network of independent, memoryless, point-to-point, finite-alphabet channels used to transmit dependent sources either losslessly or subject to a distortion constraint. In deriving this result, we also prove that in a general memoryless network with dependent sources, lossless and zero-distortion reconstruction are equivalent provided that the conditional entropy of each source given the other sources is non-zero. Furthermore, we exten…
In this work, a new upper bound for average error probability of a two-user discrete memoryless (DM) multiple-access channel (MAC) is derived. This bound can be universally obtained for all discrete memoryless MACs with given input and output alphabets. This is the first bound of this type that explicitly uses the method of expurgation. It is shown that the exponent of this bound is greater than or equal to those of previously known bounds.
We study a class of discrete memoryless broadcast interference channels (DM-BICs), where one of the broadcast receivers is subject to the interference from a point-to-point transmission. A general achievable rate region $\mathcal{R}$ based on rate splitting, superposition coding and binning at the broadcast transmitter and rate splitting at the interfering transmitter is derived. Under two partial order broadcast conditions {\em interference-oblivious less noisy} and {\em interference-cognizant less noisy}, a reduced form of $\mathcal{R}$ is shown to be equivalent to the region based on a simp…
Consider a sequence $X^n$ of length $n$ emitted by a Discrete Memoryless Source (DMS) with unknown distribution $p_X$. The objective is to construct a lossless source code that maps $X^n$ to a sequence $\widehat{Y}^m$ of length $m$ that is indistinguishable, in terms of Kullback-Leibler divergence, from a sequence emitted by another DMS with known distribution $p_Y$. The main result is the existence of a coding scheme that performs this task with an optimal ratio $m/n$ equal to $H(X)/H(Y)$, the ratio of the Shannon entropies of the two distributions, as $n$ goes to infinity. The coding scheme …
We study the performance limits of state-dependent discrete memoryless channels with a discrete state available at both the encoder and the decoder. We establish the epsilon-capacity as well as necessary and sufficient conditions for the strong converse property for such channels when the sequence of channel states is not necessarily stationary, memoryless or ergodic. We then seek a finer characterization of these capacities in terms of second-order coding rates. The general results are supplemented by several examples including i.i.d. and Markov states and mixed channels.
We consider transmission of discrete memoryless sources (DMSes) across discrete memoryless channels (DMCs) using variable-length lossy source-channel codes with feedback. The reliability function (optimum error exponent) is shown to be equal to $\max\{0, B(1-R(D)/C)\},$ where $R(D)$ is the rate-distortion function of the source, $B$ is the maximum relative entropy between output distributions of the DMC, and $C$ is the Shannon capacity of the channel. We show that, in this setting and in this asymptotic regime, separate source-channel coding is, in fact, optimal.
We establish an achievable rate region for discrete memoryless interference relay channels that consist of two source-destination pairs and one or more relays. We develop an achievable scheme combining Han-Kobayashi and noisy network coding schemes. We apply our achievability to two cases. First, we characterize the capacity region of a class of discrete memoryless interference relay channels. This class naturally generalizes the injective deterministic discrete memoryless interference channel by El Gamal and Costa and the deterministic discrete memoryless relay channel with orthogonal receive…
We give a unified treatment of some inequalities that are used in the proofs of channel polarization theorems involving a binary-input discrete memoryless channel.
In this paper, we study the asymptotic performance of Abelian group codes for the lossy source coding problem for arbitrary discrete (finite alphabet) memoryless sources as well as the channel coding problem for arbitrary discrete (finite alphabet) memoryless channels. For the source coding problem, we derive an achievable rate-distortion function that is characterized in a single-letter information-theoretic form using the ensemble of Abelian group codes. When the underlying group is a field, it simplifies to the symmetric rate-distortion function. Similarly, for the channel coding problem, w…
The problem of joint universal source coding and identification is considered in the setting of fixed-rate lossy coding of continuous-alphabet memoryless sources. For a wide class of bounded distortion measures, it is shown that any compactly parametrized family of $\R^d$-valued i.i.d. sources with absolutely continuous distributions satisfying appropriate smoothness and Vapnik--Chervonenkis learnability conditions, admits a joint scheme for universal lossy block coding and parameter estimation, such that when the block length $n$ tends to infinity, the overhead per-letter rate and the distort…
The essential interactive capacity of a discrete memoryless channel is defined in this paper as the maximal rate at which the transcript of any interactive protocol can be reliably simulated over the channel, using a deterministic coding scheme. In contrast to other interactive capacity definitions in the literature, this definition makes no assumptions on the order of speakers (which can be adaptive) and does not allow any use of private / public randomness; hence, the essential interactive capacity is a function of the channel model only. It is shown that the essential interactive capacity o…
In this paper, we study the finite blocklength limits of state-dependent discrete memoryless channels where the discrete memoryless state is known noncausally at the encoder. For the point-to-point case, this is known as the Gel'fand-Pinsker channel model. We define the (n,ε)-capacity of the Gel'fand-Pinsker channel as the maximal rate of transmission of a message subject to the condition that the length of the block-code is n and the average error probability is no larger than ε. This paper provides a lower bound for the (n,ε)-capacity of the Gel'fand-Pinsker channel model, and hence an upper…
We consider the classic joint source-channel coding problem of transmitting a memoryless source over a memoryless channel. The focus of this work is on the long-standing open problem of finding the rate of convergence of the smallest attainable expected distortion to its asymptotic value, as a function of blocklength $n$. Our main result is that in general the convergence rate is not faster than $n^{-1/2}$. In particular, we show that for the problem of transmitting i.i.d uniform bits over a binary symmetric channels with Hamming distortion, the smallest attainable distortion (bit error rate) …
We consider Slepian-Wolf code design based on LDPC (low-density parity-check) coset codes for memoryless source-side information pairs. A density evolution formula, equipped with a concentration theorem, is derived for Slepian- Wolf coding based on LDPC coset codes. As a consequence, an intimate connection between Slepian-Wolf coding and channel coding is established. Specifically we show that, under density evolution, design of binary LDPC coset codes for Slepian-Wolf coding of an arbitrary memoryless source-side information pair reduces to design of binary LDPC codes for binary-input output-…
A unified framework to obtain all known lower bounds (random coding, typical random coding and expurgated bound) on the reliability function of a point-to-point discrete memoryless channel (DMC) is presented. By using a similar idea for a two-user discrete memoryless (DM) multiple-access channel (MAC), three lower bounds on the reliability function are derived. The first one (random coding) is identical to the best known lower bound on the reliability function of DM-MAC. It is shown that the random coding bound is the performance of the average code in the constant composition code ensemble. T…
Error probabilities of random codes for memoryless channels are considered in this paper. In the area of communication systems, admissible error probability is very small and it is sometimes more important to discuss the relative gap between the achievable error probability and its bound than to discuss the absolute gap. Scarlett et al. derived a good upper bound of a random coding union bound based on the technique of saddlepoint approximation but it is not proved that the relative gap of their bound converges to zero. This paper derives a new bound on the achievable error probability in this…
This paper studies the mismatched decoding problem for binary-input discrete memoryless channels. An example is provided for which an achievable rate based on superposition coding exceeds the LM rate (Hui, 1983; Csiszár-Körner, 1981), thus providing a counter-example to a previously reported converse result (Balakirsky, 1995). Both numerical evaluations and theoretical results are used in establishing this claim.
This paper considers error probabilities of random codes for memoryless channels in the fixed-rate regime. Random coding is a fundamental scheme to achieve the channel capacity and many studies have been conducted for the asymptotics of the decoding error probability. Gallager derived the exact asymptotics (that is, a bound with asymptotically vanishing relative error) of the error probability below the critical rate. On the other hand, exact asymptotics for fixed rate above the critical rate has been unknown except for symmetric channels (in the strong sense) and strongly nonlattice channels.…
This paper quantifies the fundamental limits of variable-length transmission of a general (possibly analog) source over a memoryless channel with noiseless feedback, under a distortion constraint. We consider excess distortion, average distortion and guaranteed distortion ($d$-semifaithful codes). In contrast to the asymptotic fundamental limit, a general conclusion is that allowing variable-length codes and feedback leads to a sizable improvement in the fundamental delay-distortion tradeoff. In addition, we investigate the minimum energy required to reproduce $k$ source samples with a given f…
In this first part, a computable outer bound is proved for the multiterminal source coding problem, for a setup with two encoders, discrete memoryless sources, and bounded distortion measures.
This work investigates the downlink of a cloud radio access network (C-RAN) in which a central processor communicates with two mobile users through two base stations (BSs). The BSs act as relay nodes and cooperate with each other through error-free rate-limited links. We develop and analyze two coding schemes for this scenario. The first coding scheme is based on Liu-Kang scheme for C-RANs without BS cooperation; and extends it to scenarios allowing conferencing between the BSs. Among few other features, our new coding scheme enables arbitrary correlation among the auxiliary codewords that are…
We consider the problem of modulation and estimation of a random parameter $U$ to be conveyed across a discrete memoryless channel. Upper and lower bounds are derived for the best achievable exponential decay rate of a general moment of the estimation error, $\bE|\hat{U}-U|^ρ$, $ρ\ge 0$, when both the modulator and the estimator are subjected to optimization. These exponential error bounds turn out to be intimately related to error exponents of channel coding and to channel capacity. While in general, there is some gap between the upper and the lower bound, they asymptotically coincide both fo…
The source-coding problem with side information at the decoder is studied subject to a constraint that the encoder---to whom the side information is unavailable---be able to compute the decoder's reconstruction sequence to within some distortion. For discrete memoryless sources and finite single-letter distortion measures, an expression is given for the minimal description rate as a function of the joint law of the source and side information and of the allowed distortions at the encoder and at the decoder. The minimal description rate is also computed for a memoryless Gaussian source with squ…
The generalized Poor-Verdu error lower bound established in [1] for multihypothesis testing is studied in the classical channel coding context. It is proved that for any sequence of block codes sent over the memoryless binary symmetric channel (BSC), the minimum probability of error (under maximum likelihood decoding) has a relative deviation from the generalized bound that grows at most linearly in blocklength. This result directly implies that for arbitrary codes used over the BSC, decoder ties can only affect the subexponential behavior of the minimum probability of error.
A coding scheme is proposed for the memoryless Gaussian broadcast channel with correlated noises and feedback. For all noise correlations other than -1, the gap between the sum-rate the scheme achieves and the full-cooperation bound vanishes as the signal-to-noise ratio tends to infinity. When the correlation coefficient is -1, the gains afforded by feedback are unbounded and the prelog is doubled. When the correlation coefficient is +1 we demonstrate a dichotomy: If the noise variances are equal, then feedback is useless, and otherwise, feedback affords unbounded rate gains and doubles the pr…
We study communication systems over state-dependent channels in the presence of a malicious state-aware jamming adversary. The channel has a memoryless state with an underlying distribution. The adversary introduces a jamming signal into the channel. The state sequence is known non-causally to both the encoder and the adversary. Taking an Arbitrarily Varying Channel (AVC) approach, we consider two setups, namely, the discrete memoryless Gel'fand-Pinsker (GP) AVC and the additive white Gaussian Dirty Paper (DP) AVC. We determine the randomized coding capacity of both the AVCs under a maximum pr…
We consider source coding with fixed lag side information at the decoder. We focus on the special case of perfect side information with unit lag corresponding to source coding with feedforward (the dual of channel coding with feedback) introduced by Pradhan. We use this duality to develop a linear complexity algorithm which achieves the rate-distortion bound for any memoryless finite alphabet source and distortion measure.
The problem of lossless fixed-rate streaming coding of discrete memoryless sources with side information at the decoder is studied. A random time-varying tree-code is used to sequentially bin strings and a Stack Algorithm with a variable bias uses the side information to give a delay-universal coding system for lossless source coding with side information. The scheme is shown to give exponentially decaying probability of error with delay, with exponent equal to Gallager's random coding exponent for sources with side information. The mean of the random variable of computation for the stack deco…
We consider the problem of compression of two memoryless binary sources, the correlation between which is defined by a Hidden Markov Model (HMM). We propose a Decision Feedback (DF) based scheme which when used with low density parity check codes results in compression close to the Slepian Wolf limits.
We consider the problem of compression of two memoryless binary sources, the correlation between which is defined by a Hidden Markov Model (HMM). We propose a Decision Feedback (DF) based scheme which when used with low density parity check codes results in compression close to the Slepian Wolf limits.
For the class of mixed channels decomposed into stationary memoryless channels, single-letter characterizations of the $\varepsilon$-capacity have not been known except for restricted classes of channels such as the regular decomposable channel introduced by Winkelbauer. This paper gives single-letter characterizations of $\varepsilon$-capacity for mixed channels decomposed into at most countably many memoryless channels with a finite input alphabet and a general output alphabet with/without cost constraints. It is shown that a given characterization reduces to the one for the channel capacity…
A new single-letter achievable rate region is proposed for the two-user discrete memoryless multiple-access channel(MAC) with noiseless feedback. The proposed region includes the Cover-Leung rate region [1], and it is shown that the inclusion is strict. The proof uses a block-Markov superposition strategy based on the observation that the messages of the two users are correlated given the feedback. The rates of transmission are too high for each encoder to decode the other's message directly using the feedback, so they transmit correlated information in the next block to learn the message of o…
A lower bound on the minimum required code length of binary codes is obtained. The bound is obtained based on observing a close relation between the Ulam's liar game and channel coding. In fact, Spencer's optimal solution to the game is used to derive this new bound which improves the famous Sphere-Packing Bound.
Channel polarization, originally proposed for binary-input channels, is generalized to arbitrary discrete memoryless channels. Specifically, it is shown that when the input alphabet size is a prime number, a similar construction to that for the binary case leads to polarization. This method can be extended to channels of composite input alphabet sizes by decomposing such channels into a set of channels with prime input alphabet sizes. It is also shown that all discrete memoryless channels can be polarized by randomized constructions. The introduction of randomness does not change the order of …
We consider the two-receiver memoryless broadcast channel with states where each receiver requests both common and private messages, and may know part of the private message requested by the other receiver as receiver message side information (RMSI). We address two categories of the channel (i) channel with states known causally to the transmitter, and (ii) channel with states known non-causally to the transmitter. Starting with the channel without RMSI, we first propose a transmission scheme and derive an inner bound for the causal category. We then unify our inner bound for the causal catego…
Previous works established the capacity region for some special cases of discrete memoryless degraded cognitive interference channel (CIC) with unidirectional destination cooperation (UDC). In this letter, we characterize the capacity region of the general discrete memoryless degraded CIC-UDC. The obtained results imply that the capacity region is achieved by the Gel'fand-Pinsker coding at the cognitive transmitter, superposition coding at the primary transmitter and decode-and-forward at the relay. Furthermore, using this general result and a novel converse analysis, we establish the capacity…
In this paper, we study a model of communication under adversarial noise. In this model, the adversary makes online decisions on whether to corrupt a transmitted bit based on only the value of that bit. Like the usual binary symmetric channel of information theory or the fully adversarial channel of combinatorial coding theory, the adversary can, with high probability, introduce at most a given fraction of error. It is shown that, the capacity (maximum rate of reliable information transfer) of such memoryless adversary is strictly below that of the binary symmetric channel. We give new upper b…
We consider the general discrete memoryless two-way relay channel, where two users exchange messages via a relay, and propose two functional-decode-forward coding strategies for this channel. Functional-decode-forward involves the relay decoding a function of the users' messages rather than the individual messages themselves. This function is then broadcast back to the users, which can be used in conjunction with the user's own message to decode the other user's message. Via a numerical example, we show that functional-decode-forward with linear codes is capable of achieving strictly larger su…
