The Gallager bound is well known in the area of channel coding. However, most discussions about it mainly focus on its applications to memoryless channels. We show in this paper that the bounds obtained by Gallager's method are very tight even for general sources and channels that are defined in the information-spectrum theory. Our method is mainly based on the estimations of error exponents in those bounds, and by these estimations we proved the direct part of the Slepian-Wolf theorem and channel coding theorem for general sources and channels.
We correct an alleged contradiction to Gallager's capacity theorem for waveform channels as presented in a poster at the 2012 IEEE International Symposium on Information Theory.
We describe certain extremalities for Gallager's $E_0$ function evaluated under the uniform input distribution for binary input discrete memoryless channels. The results characterize the extremality of the $E_0(ρ)$ curves of the binary erasure channel and the binary symmetric channel among all the $E_0(ρ)$ curves that can be generated by the class of binary discrete memoryless channels whose $E_0(ρ)$ curves pass through a given point $(ρ_0, e_0)$, for some $ρ_0 > -1$.
In this paper, nested Gallager regions with a single parameter is introduced to exploit Gallager's first bounding technique (GFBT). We present a necessary and sufficient condition on the optimal parameter. We also present a sufficient condition (with a simple geometrical explanation) under which the optimal parameter does not depend on the signal-to-noise ratio (SNR). With this general framework, three existing upper bounds are revisited, including the tangential bound (TB) of Berlekamp, the sphere bound (SB) of Herzberg and Poltyrev, and the tangential-sphere bound (TSB) of Poltyrev. This pap…
The use of multiple antenna arrays in transmission and reception has become an integral part of modern wireless communications. To quantify the performance of such systems, the evaluation of bounds on the error probability of realistic finite length codewords is important. In this paper, we analyze the standard Gallager error bound for both constraints of maximum average power and maximum instantaneous power. Applying techniques from random matrix theory, we obtain analytic expressions of the error exponent when the length of the codeword increases to infinity at a fixed ratio with the antenna…
In this paper, we derive Gallager's random coding error exponent for multiple-input multiple-output (MIMO) channels, assuming no channel-state information (CSI) at the transmitter and perfect CSI at the receiver. This measure gives insight into a fundamental tradeoff between the communication reliability and information rate of MIMO channels, enabling to determine the required codeword length to achieve a prescribed error probability at a given rate below the channel capacity. We quantify the effects of the number of antennas, channel coherence time, and spatial fading correlation on the MIMO …
We propose a novel approach for bounding the probability of error of discrete memoryless channels with a zero-error capacity based on a combination of Lovász' and Gallager's ideas. The obtained bounds are expressed in terms of a function $\vartheta(ρ)$, introduced here, that varies from the cut-off rate of the channel to the Lovázs theta function as $ρ$ varies from 1 to $\infty$ and which is intimately related to Gallager's expurgated coefficient. The obtained bound to the reliability function, though loose in its present form, is finite for all rates larger than the Lovász theta function.
This paper is focused on the performance analysis of binary linear block codes (or ensembles) whose transmission takes place over independent and memoryless parallel channels. New upper bounds on the maximum-likelihood (ML) decoding error probability are derived. The framework of the second version of the Duman and Salehi (DS2) bounds is generalized to the case of parallel channels, along with the derivation of optimized tilting measures. The connection between the generalized DS2 and the 1961 Gallager bounds, known previously for a single channel, is revisited for the case of parallel channel…
This paper is focused on the performance analysis of binary linear block codes (or ensembles) whose transmission takes place over independent and memoryless parallel channels. New upper bounds on the maximum-likelihood (ML) decoding error probability are derived. These bounds are applied to various ensembles of turbo-like codes, focusing especially on repeat-accumulate codes and their recent variations which possess low encoding and decoding complexity and exhibit remarkable performance under iterative decoding. The framework of the second version of the Duman and Salehi (DS2) bounds is genera…
In this paper, parameterized Gallager's first bounding technique (GFBT) is presented by introducing nested Gallager regions, to derive upper bounds on the ML decoding error probability of general codes over AWGN channels. The three well-known bounds, namely, the sphere bound (SB) of Herzberg and Poltyrev, the tangential bound (TB) of Berlekamp, and the tangential-sphere bound (TSB) of Poltyrev, are generalized to general codes without the properties of geometrical uniformity and equal energy. When applied to the binary linear codes, the three generalized bounds are reduced to the conventional …
We study the diversity order vs rate of an additive white Gaussian noise (AWGN) channel in the whole capacity region. We show that for discrete input as well as for continuous input, Gallager's upper bounds on error probability have exponential diversity in low and high rate region but only subexponential in the mid-rate region. For the best available lower bounds and for the practical codes one observes exponential diversity throughout the capacity region. However we also show that performance of practical codes is close to Gallager's upper bounds and the mid-rate subexponential diversity has…
New lower and upper bounds on the reliability function of typewriter channels are given. Our lower bounds improve upon the (multiletter) expurgated bound of Gallager, furnishing a new and simple counterexample to a conjecture made in 1967 by Shannon, Gallager and Berlekamp on its tightness. The only other known counterexample is due to Katsman, Tsfasman and Vlăduţ who used algebraic-geometric codes on a $q$-ary symmetric channels, $q\geq 49$. Here we prove, by introducing dependence between codewords of a random ensemble, that the conjecture is false even for a typewriter channel with $q=4$ in…
We explore the relation between the techniques of statistical mechanics and information theory for assessing the performance of channel coding. We base our study on a framework developed by Gallager in {\em IEEE Trans. Inform. Theory} {\bf 11}, 3 (1965), where the minimum decoding error probability is upper-bounded by an average of a generalized Chernoff's bound over a code ensemble. We show that the resulting bound in the framework can be directly assessed by the replica method, which has been developed in statistical mechanics of disordered systems, whereas in Gallager's original methodology…
The Arimoto algorithm computes the Gallager function $\max_Q {E}_{0}^{}(ρ,Q)$ for a given channel ${P}_{}^{}(y \,|\, x)$ and parameter $ρ$, by means of alternating maximization. Along the way, it generates a sequence of input distributions ${Q}_{1}^{}(x)$, ${Q}_{2}^{}(x)$, ... , that converges to the maximizing input ${Q}_{}^{*}(x)$. We propose a stochastic interpretation for the Arimoto algorithm. We show that for a random (i.i.d.) codebook with a distribution ${Q}_{k}^{}(x)$, the next distribution ${Q}_{k+1}^{}(x)$ in the Arimoto algorithm is equal to the type (${Q}'$) of the feasible transm…
For any channel with a convex constraint set and finite Augustin capacity, existence of a unique Augustin center and associated Erven-Harremoes bound are established. Augustin-Legendre capacity, center, and radius are introduced and proved to be equal to the corresponding Renyi-Gallager entities. Sphere packing bounds with polynomial prefactors are derived for codes on two families of channels: (possibly non-stationary) memoryless channels with multiple additive cost constraints and stationary memoryless channels with convex constraints on the empirical distribution of the input codewords.
The exact order of the optimal sub-exponentially decaying factor in the classical bounds on the error probability of fixed-length codes over a Gallager-symmetric discrete memoryless channel with and without ideal feedback is determined. Regardless of the availability of feedback, it is shown that the order of the optimal sub-exponential factor exhibits a dichotomy. Moreover, the proof technique is used to establish the third-order term in the normal approximation for symmetric channels, where a similar dichotomy is shown to exist.
In this paper, lower bounds on error probability in coding for discrete classical and classical-quantum channels are studied. The contribution of the paper goes in two main directions: i) extending classical bounds of Shannon, Gallager and Berlekamp to classical-quantum channels, and ii) proposing a new framework for lower bounding the probability of error of channels with a zero-error capacity in the low rate region. The relation between these two problems is revealed by showing that Lovász' bound on zero-error capacity emerges as a natural consequence of the sphere packing bound once we move…
We develop several analytical lower bounds on the capacity of binary insertion and deletion channels by considering independent uniformly distributed (i.u.d.) inputs and computing lower bounds on the mutual information between the input and output sequences. For the deletion channel, we consider two different models: independent and identically distributed (i.i.d.) deletion-substitution channel and i.i.d. deletion channel with additive white Gaussian noise (AWGN). These two models are considered to incorporate effects of the channel noise along with the synchronization errors. For the insertio…
Second-order coding rate of channel coding is discussed for general sequence of channels. The optimum second-order transmission rate with a constant error constraint $ε$ is obtained by using the information spectrum method. We apply this result to the discrete memoryless case, the discrete memoryless case with a cost constraint, the additive Markovian case, and the Gaussian channel case with an energy constraint. We also clarify that the Gallager bound does not give the optimum evaluation in the second-order coding rate.
The performance of maximum-likelihood (ML) decoding on the binary erasure channel for finite-length low-density parity-check (LDPC) codes from two random ensembles is studied. The theoretical average spectrum of the Gallager ensemble is computed by using a recurrent procedure and compared to the empirically found average spectrum for the same ensemble as well as to the empirical average spectrum of the Richardson-Urbanke ensemble and spectra of selected codes from both ensembles. Distance properties of the random codes from the Gallager ensemble are discussed. A tightened union-type upper boun…
Sphere packing bounds (SPBs) ---with prefactors that are polynomial in the block length--- are derived for codes on two families of memoryless channels using Augustin's method: (possibly non-stationary) memoryless channels with (possibly multiple) additive cost constraints and stationary memoryless channels with convex constraints on the composition (i.e. empirical distribution, type) of the input codewords. A variant of Gallager's bound is derived in order to show that these sphere packing bounds are tight in terms of the exponential decay rate of the error probability with the block length u…
In this paper, we propose an upper bound on the minimum Bhattacharyya distance of codes for channels with a zero-error capacity. The bound is obtained by combining an extension of the Elias bound introduced by Blahut, with an extension of a bound previously introduced by the author, which builds upon ideas of Gallager, Lovász and Marton.
In this paper, an achievable error exponent for the multiple-access channel with two independent sources is derived. For each user, the source messages are partitioned into two classes and codebooks are generated by drawing codewords from an input distribution depending on the class index of the source message. The partitioning thresholds that maximize the achievable exponent are given by the solution of a system of equations. We also derive both lower and upper bounds for the achievable exponent in terms of Gallager's source and channel functions. Finally, a numerical example shows that using…
This paper considers error probabilities of random codes for memoryless channels in the fixed-rate regime. Random coding is a fundamental scheme to achieve the channel capacity and many studies have been conducted for the asymptotics of the decoding error probability. Gallager derived the exact asymptotics (that is, a bound with asymptotically vanishing relative error) of the error probability below the critical rate. On the other hand, exact asymptotics for fixed rate above the critical rate has been unknown except for symmetric channels (in the strong sense) and strongly nonlattice channels.…
New non-asymptotic random coding theorems (with error probability $ε$ and finite block length $n$) based on Gallager parity check ensemble and Shannon random code ensemble with a fixed codeword type are established for discrete input arbitrary output channels. The resulting non-asymptotic achievability bounds, when combined with non-asymptotic equipartition properties developed in the paper, can be easily computed. Analytically, these non-asymptotic achievability bounds are shown to be asymptotically tight up to the second order of the coding rate as $n$ goes to infinity with either constant o…
We construct a channel coding scheme to achieve the capacity of any discrete memoryless channel based solely on the techniques of polar coding. In particular, we show how source polarization and randomness extraction via polarization can be employed to "shape" uniformly-distributed i.i.d. random variables into approximate i.i.d. random variables distributed ac- cording to the capacity-achieving distribution. We then combine this shaper with a variant of polar channel coding, constructed by the duality with source coding, to achieve the channel capacity. Our scheme inherits the low complexity e…
We give new bounds on the reliability function of a typewriter channel with 5 inputs and crossover probability $1/2$. The lower bound is more of theoretical than practical importance; it improves very marginally the expurgated bound, providing a counterexample to a conjecture on its tightness by Shannon, Gallager and Berlekamp which does not need the construction of algebraic-geometric codes previously used by Katsman, Tsfasman and Vlăduţ. The upper bound is derived by using an adaptation of the linear programming bound and it is essentially useful as a low-rate anchor for the straight line bo…
We propose a technique to derive upper bounds on Gallager's cost-constrained random coding exponent function. Applying this technique to the non-coherent peak-power or average-power limited discrete time memoryless Ricean fading channel, we obtain the high signal-to-noise ratio (SNR) expansion of this channel's cut-off rate. At high SNR the gap between channel capacity and the cut-off rate approaches a finite limit. This limit is approximately 0.26 nats per channel-use for zero specular component (Rayleigh) fading and approaches 0.39 nats per channel-use for very large specular components. We …
The capacity of the heat channel, a linear time-varying (LTV) filter with additive white Gaussian noise (AWGN), is characterized by waterfilling in the time-frequency plane. Similarly, the rate distortion function for a related nonstationary source is characterized by reverse waterfilling in the time-frequency plane. The source is formed by the white Gaussian noise response of the same LTV filter as before. The proofs of both waterfilling theorems rely on a specific Szego theorem for a positive definite operator associated with the filter. An essentially self-contained proof of the Szego theor…
A recent line of work has focused on the use of low-density generator matrix (LDGM) codes for lossy source coding. In this paper, wedevelop a generic technique for deriving lower bounds on the rate-distortion functions of binary linear codes, with particular interest on the effect of bounded degrees. The underlying ideas can be viewing as the source coding analog of the classical result of Gallager, providing bounds for channel coding over the binary symmetric channel using bounded degree LDPC codes. We illustrate this method for different random ensembles of LDGM codes, including the check-re…
This paper presents an extension of the Elias bound on the minimum distance of codes for discrete alphabets with general, possibly infinite-valued, distances. The bound is obtained by combining a previous extension of the Elias bound, introduced by Blahut, with an extension of a bound previously introduced by the author which builds upon ideas of Gallager, Lovász and Marton. The result can in fact be interpreted as a unification of the Elias bound and of Lovász's bound on graph (or zero-error) capacity, both being recovered as particular cases of the one presented here. Previous extensions of …
Probabilistic amplitude shaping (PAS) is a coded modulation strategy in which constellation shaping and channel coding are combined. PAS has attracted considerable attention in both wireless and optical communications. Achievable information rates (AIRs) of PAS have been investigated in the literature using Gallager's error exponent approach. In particular, it has been shown that PAS achieves the capacity of the additive white Gaussian noise channel (Böcherer, 2018). In this work, we revisit the capacity-achieving property of PAS and derive AIRs using weak typicality. Our objective is to provi…
The sphere packing bound, in the form given by Shannon, Gallager and Berlekamp, was recently extended to classical-quantum channels, and it was shown that this creates a natural setting for combining probabilistic approaches with some combinatorial ones such as the Lovász theta function. In this paper, we extend the study to the case of constant composition codes. We first extend the sphere packing bound for classical-quantum channels to this case, and we then show that the obtained result is related to a variation of the Lovász theta function studied by Marton. We then propose a further exten…
A coding theorem is proved for a class of stationary channels with feedback in which the output Y_n = f(X_{n-m}^n, Z_{n-m}^n) is the function of the current and past m symbols from the channel input X_n and the stationary ergodic channel noise Z_n. In particular, it is shown that the feedback capacity is equal to $$ \limp_{n\to\infty} \sup_{p(x^n||y^{n-1})} \frac{1}{n} I(X^n \to Y^n), $$ where I(X^n \to Y^n) = \sum_{i=1}^n I(X^i; Y_i|Y^{i-1}) denotes the Massey directed information from the channel input to the output, and the supremum is taken over all causally conditioned distributions p(x^n…
The error exponent of the typical random code is defined as the asymptotic normalized expectation of the logarithm of the probability of error, as opposed to the traditional definition of the random coding exponent as the normalized logarithm of the expectation of the probability of error with respect to a given ensemble of codes. For a certain ensemble of independent codewords, with a given power spectrum, and a generalized stochastic mismatched decoder, we characterize the error exponent the typical random codes (TRC) for the colored Gaussian channel, with emphasis on the range of low rates,…
The sphere packing bound, in the form given by Shannon, Gallager and Berlekamp, was recently extended to classical-quantum channels, and it was shown that this creates a natural setting for combining probabilistic approaches with some combinatorial ones such as the Lovász theta function. In this paper, we extend the study to the case of constant composition codes. We first extend the sphere packing bound for classical-quantum channels to this case, and we then show that the obtained result is related to a variation of the Lovász theta function studied by Marton. We then propose a further exten…
We develop a new upper bound on the capacity of the relay channel that is tighter than previously known upper bounds. This upper bound is proved using traditional weak converse techniques involving mutual information inequalities and Gallager-type explicit identification of auxiliary random variables. We show that the new upper bound is strictly tighter than all previous bounds for the Gaussian relay channel with non-zero channel gains. When specialized to the relay channel with orthogonal receiver components, the bound resolves a conjecture by Kim on a class of deterministic relay channels. W…
We consider a broadcast channel with a degraded message set, in which a single transmitter sends a common message to two receivers and a private message to one of the receivers only. The main goal of this work is to find new lower bounds to the error exponents of the strong user, the one that should decode both messages, and of the weak user, that should decode only the common message. Unlike previous works, where suboptimal decoders where used, the exponents we derive in this work pertain to optimal decoding and depend on both rates. We take two different approaches. The first approach is bas…
We consider a linear multi-hop network composed of multi-state discrete-time memoryless channels over each hop, with orthogonal time-sharing across hops under a half-duplex relaying protocol. We analyze the probability of error and associated reliability function \cite{Gallager68} over the multi-hop network; with emphasis on random coding and sphere packing bounds, under the assumption of point-to-point coding over each hop. In particular, we define the system reliability function for the multi-hop network and derive lower and upper bounds on this function to specify the reliability-optimal op…
Universally achievable error exponents pertaining to certain families of channels (most notably, discrete memoryless channels (DMC's)), and various ensembles of random codes, are studied by combining the competitive minimax approach, proposed by Feder and Merhav, with Chernoff bound and Gallager's techniques for the analysis of error exponents. In particular, we derive a single--letter expression for the largest, universally achievable fraction $ξ$ of the optimum error exponent pertaining to the optimum ML decoding. Moreover, a simpler single--letter expression for a lower bound to $ξ$ is pres…
We survey coding techniques that enable reliable transmission at rates that approach the capacity of an arbitrary discrete memoryless channel. In particular, we take the point of view of modern coding theory and discuss how recent advances in coding for symmetric channels help provide more efficient solutions for the asymmetric case. We consider, in more detail, three basic coding paradigms. The first one is Gallager's scheme that consists of concatenating a linear code with a non-linear mapping so that the input distribution can be appropriately shaped. We explicitly show that both polar code…
The problem of lossless fixed-rate streaming coding of discrete memoryless sources with side information at the decoder is studied. A random time-varying tree-code is used to sequentially bin strings and a Stack Algorithm with a variable bias uses the side information to give a delay-universal coding system for lossless source coding with side information. The scheme is shown to give exponentially decaying probability of error with delay, with exponent equal to Gallager's random coding exponent for sources with side information. The mean of the random variable of computation for the stack deco…
We derive a new upper bound on the reliability function for channel coding over discrete memoryless channels. Our bounding technique relies on two main elements: (i) adding an auxiliary genie-receiver that reveals to the original receiver a list of codewords including the transmitted one, which satisfy a certain type property, and (ii) partitioning (most of) the list into subsets of codewords that satisfy a certain pairwise-symmetry property, which facilitates lower bounding of the average error probability by the pairwise error probability within a subset. We compare the obtained bound to the…
Motivated by the growing application of wireless multi-access networks with stringent delay constraints, we investigate the Gaussian multiple access channel (MAC) in the finite blocklength regime. Building upon information spectrum concepts, we develop several non-asymptotic inner bounds on channel coding rates over the Gaussian MAC with a given finite blocklength, positive average error probability, and maximal power constraints. Employing Central Limit Theorem (CLT) approximations, we also obtain achievable second-order coding rates for the Gaussian MAC based on an explicit expression for it…
A Lagrange-dual (Gallager-style) lower bound is derived for the error exponent function of the typical random code (TRC) pertaining to the i.i.d. random coding ensemble and mismatched stochastic likelihood decoding. While the original expression, derived from the method of types (the Csiszar-style expression) involves minimization over probability distributions defined on the channel input--output alphabets, the new Lagrange-dual formula involves optimization of five parameters, independently of the alphabet sizes. For both stochastic and deterministic mismatched decoding (including maximum li…
This paper studies expurgated random-coding bounds and exponents for channel coding with a given (possibly suboptimal) decoding rule. Variations of Gallager's analysis are presented, yielding several asymptotic and non-asymptotic bounds on the error probability for an arbitrary codeword distribution. A simple non-asymptotic bound is shown to attain an exponent of Csiszár and Körner under constant-composition coding. Using Lagrange duality, this exponent is expressed in several forms, one of which is shown to permit a direct derivation via cost-constrained coding which extends to infinite and c…
This paper looks into the $K$-user interference channel. Interference Alignment is much likely to be applied with double-layered symbol extensions, either for constant channels in the H$ø$st-Madsen-Nosratinia conjecture or slowly changing channels. In our work, the core idea relies on double-layered symbol extensions to artificially construct equivalent time-variant channels to provide crucial \textit{channel randomness or relativity} required by conventional Cadambe-Jafar scheme in time-variant channels \cite{IA-DOF-Kuser-Interference}.
Some new results are derived concerning random coding error exponents and expurgated exponents for list decoding with a deterministic list size $L$. Two asymptotic regimes are considered, the fixed list-size regime, where $L$ is fixed independently of the block length $n$, and the exponential list-size, where $L$ grows exponentially with $n$. We first derive a general upper bound on the list-decoding average error probability, which is suitable for both regimes. This bound leads to more specific bounds in the two regimes. In the fixed list-size regime, the bound is related to known bounds and …
The error exponent of a discrete memoryless channel is expressed in two forms. One is Gallager's expression with a positive slope parameter and the other is Csiszar and Korner's information-theoretic representation expressed using the mutual information and the relative entropy. They differ in appearance, and existing methods to prove their agreement are not elementary, as they require an evaluation of the KKT conditions that the optimal distribution must satisfy. Similarly, there are two types of expressions for the strong converse exponent. They are Arimoto's expression with a negative slope…
Unveiling a fundamental link between information theory and estimation theory, the I-MMSE relationship by Guo, Shamai and Verdu~\cite{gu05}, together with its numerous extensions, has great theoretical significance and various practical applications. On the other hand, its influences to date have been restricted to channels without feedback or memory, due to the absence of its extensions to such channels. In this paper, we propose extensions of the I-MMSE relationship to discrete-time and continuous-time Gaussian channels with feedback and/or memory. Our approach is based on a very simple obse…
Vector space interference alignment (IA) is known to achieve high degrees of freedom (DoF) with infinite independent channel extensions, but its performance is largely unknown for a finite number of possibly dependent channel extensions. In this paper, we consider a $K$-user $M_t \times M_r$ MIMO interference channel (IC) with arbitrary number of channel extensions $T$ and arbitrary channel diversity order $L$ (i.e., each channel matrix is a generic linear combination of $L$ fixed basis matrices). We study the maximum DoF achievable via vector space IA in the single beam case (i.e. each user s…
A simple lemma is derived that allows to transform a general scalar (non-Gaussian, non-additive) continuous-alphabet channel as well as a general multiple-access channel into a modulo-additive noise channel. While in general the transformation is information lossy, it allows to leverage linear coding techniques and capacity results derived for networks comprised of additive Gaussian nodes to more general networks.
This paper starts by considering the minimization of the Renyi divergence subject to a constraint on the total variation distance. Based on the solution of this optimization problem, the exact locus of the points $\bigl( D(Q\|P_1), D(Q\|P_2) \bigr)$ is determined when $P_1, P_2, Q$ are arbitrary probability measures which are mutually absolutely continuous, and the total variation distance between $P_1$ and $P_2$ is not below a given value. It is further shown that all the points of this convex region are attained by probability measures which are defined on a binary alphabet. This characteriz…
