Error probabilities of random codes for memoryless channels are considered in this paper. In the area of communication systems, admissible error probability is very small and it is sometimes more important to discuss the relative gap between the achievable error probability and its bound than to discuss the absolute gap. Scarlett et al. derived a good upper bound of a random coding union bound based on the technique of saddlepoint approximation but it is not proved that the relative gap of their bound converges to zero. This paper derives a new bound on the achievable error probability in this viewpoint for a class of memoryless channels. The derived bound is strictly smaller than that by Scarlett et al. and its relative gap with the random coding error probability (not a union bound) vanishes as the block length increases for a fixed coding rate.
Universal fixed-to-variable lossless source coding for memoryless sources is studied in the finite blocklength and higher-order asymptotics regimes. Optimal third-order coding rates are derived for general fixed-to-variable codes and for prefix codes. It is shown that the non-prefix Type Size code, in which codeword lengths are chosen in ascending order of type class size, achieves the optimal third-order rate and outperforms classical Two-Stage codes. Converse results are proved making use of a result on the distribution of the empirical entropy and Laplace's approximation. Finally, the fixed…
New non-asymptotic random coding theorems (with error probability $ε$ and finite block length $n$) based on Gallager parity check ensemble and Shannon random code ensemble with a fixed codeword type are established for discrete input arbitrary output channels. The resulting non-asymptotic achievability bounds, when combined with non-asymptotic equipartition properties developed in the paper, can be easily computed. Analytically, these non-asymptotic achievability bounds are shown to be asymptotically tight up to the second order of the coding rate as $n$ goes to infinity with either constant o…
Second order asymptotics of fixed-length source coding and intrinsic randomness is discussed with a constant error constraint. There was a difference between optimal rates of fixed-length source coding and intrinsic randomness, which never occurred in the first order asymptotics. In addition, the relation between uniform distribution and compressed data is discussed based on this fact. These results are valid for general information sources as well as independent and identical distributions. A universal code attaining the second order optimal rate is also constructed.
This paper presents a saddlepoint approximation of the random-coding union bound of Polyanskiy et al. for i.i.d. random coding over discrete memoryless channels. The approximation is single-letter, and can thus be computed efficiently. Moreover, it is shown to be asymptotically tight for both fixed and varying rates, unifying existing achievability results in the regimes of error exponents, second-order coding rates, and moderate deviations. For fixed rates, novel exact-asymptotics expressions are specified to within a multiplicative 1+o(1) term. A numerical example is provided for which the a…
This paper studies the subexponential prefactor to the random-coding bound for a given rate. Using a refinement of Gallager's bounding techniques, an alternative proof of a recent result by Altuğ and Wagner is given, and the result is extended to the setting of mismatched decoding.
This paper considers error probabilities of random codes for memoryless channels in the fixed-rate regime. Random coding is a fundamental scheme to achieve the channel capacity and many studies have been conducted for the asymptotics of the decoding error probability. Gallager derived the exact asymptotics (that is, a bound with asymptotically vanishing relative error) of the error probability below the critical rate. On the other hand, exact asymptotics for fixed rate above the critical rate has been unknown except for symmetric channels (in the strong sense) and strongly nonlattice channels.…
We present novel non-asymptotic or finite blocklength achievability bounds for three side-information problems in network information theory. These include (i) the Wyner-Ahlswede-Korner (WAK) problem of almost-lossless source coding with rate-limited side-information, (ii) the Wyner-Ziv (WZ) problem of lossy source coding with side-information at the decoder and (iii) the Gel'fand-Pinsker (GP) problem of channel coding with noncausal state information available at the encoder. The bounds are proved using ideas from channel simulation and channel resolvability. Our bounds for all three problems…
In this paper, we revisit two multi-terminal lossy source coding problems: the lossy source coding problem with side information available at the encoder and one of the two decoders, which we term as the Kaspi problem (Kaspi, 1994), and the multiple description coding problem with one semi-deterministic distortion measure, which we refer to as the Fu-Yeung problem (Fu and Yeung, 2002). For the Kaspi problem, we first present the properties of optimal test channels. Subsequently, we generalize the notion of the distortion-tilted information density for the lossy source coding problem to the Kas…
This work contains two main contributions concerning the asymmetric broadcast channel. The first is an analysis of the exact random coding error exponents for both users, and the second is the derivation of universal decoders for both users. These universal decoders are certain variants of the maximum mutual information (MMI) universal decoder, which achieve the corresponding random coding exponents of optimal decoding. In addition, we introduce some lower bounds, which involve optimization over very few parameters, unlike the original, exact exponents, which involve minimizations over auxilia…
In this paper, we consider the problem of variable-length source coding allowing errors. The exponential moment of the codeword length is analyzed in the non-asymptotic regime and in the asymptotic regime. Our results show that the smooth Renyi entropy characterizes the optimal exponential moment of the codeword length.
Recently, a new decoding rule called jar decoding was proposed; under jar decoding, a non-asymptotic achievable tradeoff between the coding rate and word error probability was also established for any discrete input memoryless channel with discrete or continuous output (DIMC). Along the path of non-asymptotic analysis, in this paper, it is further shown that jar decoding is actually optimal up to the second order coding performance by establishing new non-asymptotic converse coding theorems, and determining the Taylor expansion of the (best) coding rate $R_n (ε)$ of finite block length for any…
We consider ensembles of channel codes that are partitioned into bins, and focus on analysis of exact random coding error exponents associated with optimum decoding of the index of the bin to which the transmitted codeword belongs. Two main conclusions arise from this analysis: (i) for independent random selection of codewords within a given type class, the random coding exponent of optimal bin index decoding is given by the ordinary random coding exponent function, computed at the rate of the entire code, independently of the exponential rate of the size of the bin. (ii) for this ensemble of …
The exact order of the optimal sub-exponentially decaying factor in the classical bounds on the error probability of fixed-length codes over a Gallager-symmetric discrete memoryless channel with and without ideal feedback is determined. Regardless of the availability of feedback, it is shown that the order of the optimal sub-exponential factor exhibits a dichotomy. Moreover, the proof technique is used to establish the third-order term in the normal approximation for symmetric channels, where a similar dichotomy is shown to exist.
In this paper we provide new compact integral expressions and associated simple asymptotic approximations for converse and achievability bounds in the finite blocklength regime. The chosen converse and random coding union bounds were taken from the recent work of Polyanskyi-Poor-Verdu, and are investigated under parallel AWGN channels, the AWGN channels, the BI-AWGN channel, and the BSC. The technique we use, which is a generalization of some recent results available from the literature, is to map the probabilities of interest into a Laplace integral, and then solve (or approximate) the integr…
Given an independent and identically distributed source $X = \{X_i \}_{i=1}^{\infty}$ with finite Shannon entropy or differential entropy (as the case may be) $H(X)$, the non-asymptotic equipartition property (NEP) with respect to $H(X)$ is established, which characterizes, for any finite block length $n$, how close $-{1\over n} \ln p(X_1 X_2...X_n)$ is to $H(X)$ by determining the information spectrum of $X_1 X_2...X_n $, i.e., the distribution of $-{1\over n} \ln p(X_1 X_2...X_n)$. Non-asymptotic equipartition properties (with respect to conditional entropy, mutual information, and relative …
A solution to the problem of asymptotically optimum perfect universal steganography of finite memoryless sources with a passive warden is provided, which is then extended to contemplate a distortion constraint. The solution rests on the fact that Slepian's Variant I permutation coding implements first-order perfect universal steganography of finite host signals with optimum embedding rate. The duality between perfect universal steganography with asymptotically optimum embedding rate and lossless universal source coding with asymptotically optimum compression rate is evinced in practice by show…
Sparse random linear network coding (SRLNC) used as a class of erasure codes to ensure the reliability of multicast communications has been widely investigated. However, an exact expression for the decoding success probability of SRLNC is still unknown, and existing expressions are either asymptotic or approximate. In this paper, we derive an exact expression for the decoding success probability of SRLNC. The key to achieving this is to propose a criterion that a vector is contained in a subspace. To obtain this criterion, we construct a basis of a subspace, with respect to this basis, the coo…
The zero-error channel capacity is the maximum asymptotic rate that can be reached with error probability exactly zero, instead of a vanishing error probability. The nature of this problem, essentially combinatorial rather than probabilistic, has led to various researches both in Information Theory and Combinatorics. However, the zero-error capacity is still an open problem, for example the capacity of the noisy-typewriter channel with 7 letters is unknown. In this article, we propose a new approach to construct optimal zero-error codes, based on the concatenation of words of variable-length, …
We consider the problem of zero error source coding with limited feedback when side information is present at the receiver. First, we derive an achievable rate region for arbitrary joint distributions on the source and the side information. When all source pairs of source and side information symbols are observable with non-zero probability, we show that this characterization gives the entire rate region. Next, we demonstrate a class of sources for which asymptotically zero feedback suffices to achieve zero-error coding at the rate promised by the Slepian-Wolf bound for asymptotically lossless…
We analyze the exact exponential decay rate of the expected amount of information leaked to the wiretapper in Wyner's wiretap channel setting using wiretap channel codes constructed from both i.i.d. and constant-composition random codes. Our analysis for those sampled from i.i.d. random coding ensemble shows that the previously-known achievable secrecy exponent using this ensemble is indeed the exact exponent for an average code in the ensemble. Furthermore, our analysis on wiretap channel codes constructed from the ensemble of constant-composition random codes leads to an exponent which, in a…
We consider the problem of slotted asynchronous coded communication, where in each time frame (slot), the transmitter is either silent or transmits a codeword from a given (randomly selected) codebook. The task of the decoder is to decide whether transmission has taken place, and if so, to decode the message. We derive the optimum detection/decoding rule in the sense of the best trade-off among the probabilities of decoding error, false alarm, and misdetection. For this detection/decoding rule, we then derive single-letter characterizations of the exact exponential rates of these three probabi…
Compute-and-forward relaying is effective to increase bandwidth efficiency of wireless two-way relay channels. In a compute-and-forward scheme, a relay tries to decode a linear combination composed of transmitted messages from other terminals or relays. Design for error correcting codes and its decoding algorithms suitable for compute-and-forward relaying schemes are still important issue to be studied. In this paper, we will present an asymptotic performance analysis on LDPC codes over two-way relay channels based on density evolution (DE). Because of the asymmetric nature of the channel, we …
In this paper we define critical graphs as minimal graphs that support a given set of rates for the index coding problem, and study them for both the one-shot and asymptotic setups. For the case of equal rates, we find the critical graph with minimum number of edges for both one-shot and asymptotic cases. For the general case of possibly distinct rates, we show that for one-shot and asymptotic linear index coding, as well as asymptotic non-linear index coding, each critical graph is a union of disjoint strongly connected subgraphs (USCS). On the other hand, we identify a non-USCS critical grap…
This paper considers the problem of channel coding with a given (possibly suboptimal) maximum-metric decoding rule. A cost-constrained random-coding ensemble with multiple auxiliary costs is introduced, and is shown to achieve error exponents and second-order coding rates matching those of constant-composition random coding, while being directly applicable to channels with infinite or continuous alphabets. The number of auxiliary costs required to match the error exponents and second-order rates of constant-composition coding is studied, and is shown to be at most two. For i.i.d. random coding…
A coding scheme is proposed for the memoryless Gaussian broadcast channel with correlated noises and feedback. For all noise correlations other than -1, the gap between the sum-rate the scheme achieves and the full-cooperation bound vanishes as the signal-to-noise ratio tends to infinity. When the correlation coefficient is -1, the gains afforded by feedback are unbounded and the prelog is doubled. When the correlation coefficient is +1 we demonstrate a dichotomy: If the noise variances are equal, then feedback is useless, and otherwise, feedback affords unbounded rate gains and doubles the pr…
In this paper, we present an exact model for the analysis of the performance of Random Linear Network Coding (RLNC) in wired erasure networks with finite buffers. In such networks, packets are delayed due to either random link erasures or blocking by full buffers. We assert that because of RLNC, the content of buffers have dependencies which cannot be captured directly using the classical queueing theoretical models. We model the performance of the network using Markov chains by a careful derivation of the buffer occupancy states and their transition rules. We verify by simulations that the pr…
Index coding models broadcast networks in which a sender sends different messages to different receivers simultaneously, where each receiver may know some of the messages a priori. The aim is to find the minimum (normalised) index codelength that the sender sends. This paper considers unicast index coding, where each receiver requests exactly one message, and each message is requested by exactly one receiver. Each unicast index-coding instances can be fully described by a directed graph and vice versa, where each vertex corresponds to one receiver. For any directed graph representing a unicast…
This paper studies expurgated random-coding bounds and exponents for channel coding with a given (possibly suboptimal) decoding rule. Variations of Gallager's analysis are presented, yielding several asymptotic and non-asymptotic bounds on the error probability for an arbitrary codeword distribution. A simple non-asymptotic bound is shown to attain an exponent of Csiszár and Körner under constant-composition coding. Using Lagrange duality, this exponent is expressed in several forms, one of which is shown to permit a direct derivation via cost-constrained coding which extends to infinite and c…
Koetter and Kschischang showed in [R. Koetter and F.R. Kschischang, "Coding for Errors and Erasures in Random Network Coding," IEEE Trans. Inform. Theory, {54(8), 2008] that the network coding counterpart of Gabidulin codes performs asymptotically optimal with respect to the subspace distance. Recently, Silva and Kschischang introduced in [D. Silva and F.R. Kschischang, "On Metrics for Error Correction in Network Coding," To appear in IEEE Trans. Inform. Theory, ArXiv: 0805.3824v4[cs.IT], 2009] the injection distance to give a detailed picture of what happens in noncoherent network coding. We …
This paper focuses on the ultimate limit theory of image compression. It proves that for an image source, there exists a coding method with shapes that can achieve the entropy rate under a certain condition where the shape-pixel ratio in the encoder/decoder is $O({1 \over {\log t}})$. Based on the new finding, an image coding framework with shapes is proposed and proved to be asymptotically optimal for stationary and ergodic processes. Moreover, the condition $O({1 \over {\log t}})$ of shape-pixel ratio in the encoder/decoder has been confirmed in the image database MNIST, which illustrates th…
We derive the optimum second-order coding rates, known as second-order capacities, for erasure and list decoding. For erasure decoding for discrete memoryless channels, we show that second-order capacity is $\sqrt{V}Φ^{-1}(ε_t)$ where $V$ is the channel dispersion and $ε_t$ is the total error probability, i.e., the sum of the erasure and undetected errors. We show numerically that the expected rate at finite blocklength for erasures decoding can exceed the finite blocklength channel coding rate. We also show that the analogous result also holds for lossless source coding with decoder side info…
In practice, since many communication networks are huge in scale or complicated in structure even dynamic, the predesigned network codes based on the network topology is impossible even if the topological structure is known. Therefore, random linear network coding was proposed as an acceptable coding technique. In this paper, we further study the performance of random linear network coding by analyzing the failure probabilities at sink node for different knowledge of network topology and get some tight and asymptotically tight upper bounds of the failure probabilities. In particular, the worst…
Compute-and-forward (CAF) relaying is effective to increase bandwidth efficiency of wireless two-way relay channels. In a CAF scheme, a relay is designed to decode a linear combination composed of transmitted messages from other terminals or relays. Design for error-correcting codes and its decoding algorithms suitable for CAF relaying schemes remain as an important issue to be studied. As described in this paper, we will present an asymptotic performance analysis of LDPC codes over two-way relay channels based on density evolution (DE). Because of the asymmetric characteristics of the channel…
This study considers the unconditional smooth Rényi entropy, the smooth conditional Rényi entropy proposed by Kuzuoka [\emph{IEEE Trans.\ Inf.\ Theory}, vol.~66, no.~3, pp.~1674--1690, 2020], and a new quantity which we term the conditional smooth Rényi entropy. In particular, we examine asymptotic expansions of these entropies when the underlying source with its side-information is stationary and memoryless. Using these smooth Rényi entropies, we establish one-shot coding theorems of several information-theoretic problems: Campbell's source coding, guessing problems, and task encoding problem…
A conjectural expression of the asymptotic gap between the rate-distortion function of an arbitrary generalized Gaussian multiterminal source coding system and that of its centralized counterpart in the high-resolution regime is proposed. The validity of this expression is verified when the number of sources is no more than 3.
We consider the classic joint source-channel coding problem of transmitting a memoryless source over a memoryless channel. The focus of this work is on the long-standing open problem of finding the rate of convergence of the smallest attainable expected distortion to its asymptotic value, as a function of blocklength $n$. Our main result is that in general the convergence rate is not faster than $n^{-1/2}$. In particular, we show that for the problem of transmitting i.i.d uniform bits over a binary symmetric channels with Hamming distortion, the smallest attainable distortion (bit error rate) …
A major challenge of wireless multicast is to be able to support a large number of users while simultaneously maintaining low delay and low feedback overhead. In this paper, we develop a joint coding and feedback scheme named Moving Window Network Coding with Anonymous Feedback (MWNC-AF) that successfully addresses this challenge. In particular, we show that our scheme simultaneously achieves both a constant decoding delay and a constant feedback overhead, irrespective of the number of receivers $n$, without sacrificing either throughput or reliability. We explicitly characterize the asymptoti…
The paper presents exponentially-strong converses for source-coding, channel coding, and hypothesis testing problems. More specifically, it presents alternative proofs for the well-known exponentially-strong converse bounds for almost lossless source-coding with side-information and for channel coding over a discrete memoryless channel (DMC). These alternative proofs are solely based on a change of measure argument on the sets of conditionally or jointly typical sequences that result in a correct decision, and on the analysis of these measures in the asymptotic regime of infinite blocklengths.…
The second-order achievable asymptotics in typical random number generation problems such as resolvability, intrinsic randomness, fixed-length source coding are considered. In these problems, several researchers have derived the first-order and the second-order achievability rates for general sources using the information spectrum methods. Although these formulas are general, their computation are quite hard. Hence, an attempt to address explicit computation problems of achievable rates is meaningful. In particular, for i.i.d. sources, the second-order achievable rates have earlier been determ…
We investigate the MIMO broadcast channel in the high SNR regime when linear filtering is applied instead of dirty paper coding. Using a user-wise rate duality where the streams of every single user are not treated as self-interference as in the hitherto existing stream-wise rate dualities for linear filtering, we solve the weighted sum rate maximization problem of the broadcast channel in the dual multiple access channel. Thus, we can exactly quantify the asymptotic rate loss of linear filtering compared to dirty paper coding for any channel realization. Having converted the optimum covarianc…
Given a $q$-ary frequency hopping sequence set of length $n$ and size $M$ with Hamming correlation $H$, one can obtain a $q$-ary (nonlinear) cyclic code of length $n$ and size $nM$ with Hamming distance $n-H$. Thus, every upper bound on the size of a code from coding theory gives an upper bound on the size of a frequency hopping sequence set. Indeed, all upper bounds from coding theory have been converted to upper bounds on frequency hopping sequence sets (\cite{Ding09}). On the other hand, a lower bound from coding theory does not automatically produce a lower bound for frequency hopping sequ…
This paper investigates the asymptotic expansion for the maximum rate of fixed-length codes over a parallel Gaussian channel with feedback under the following setting: A peak power constraint is imposed on every transmitted codeword, and the average error probability of decoding the transmitted message is non-vanishing as the blocklength increases. It is well known that the presence of feedback does not increase the first-order asymptotics of the channel, i.e., capacity, in the asymptotic expansion, and the closed-form expression of the capacity can be obtained by the well-known water-filling …
This paper provides upper and lower bounds on the optimal guessing moments of a random variable taking values on a finite set when side information may be available. These moments quantify the number of guesses required for correctly identifying the unknown object and, similarly to Arikan's bounds, they are expressed in terms of the Arimoto-Rényi conditional entropy. Although Arikan's bounds are asymptotically tight, the improvement of the bounds in this paper is significant in the non-asymptotic regime. Relationships between moments of the optimal guessing function and the MAP error probabili…
Asymptotic energy-distortion performance of zero-delay communication scenarios under additive white Gaussian noise is investigated. Using high-resolution analysis for quantizer design, the higher-order term in the logarithm of the distortion (termed the {\em energy-distortion dispersion}) is optimized while keeping the leading term (i.e., {\em energy-distortion exponent}) at its optimal value. For uniform and Gaussian sources, significant gains are observed compared to naïvely performed quantization, i.e., aimed at optimizing the source coding performance instead of the end-to-end distortion i…
We consider the problem of source compression under three different scenarios in the one-shot (non- asymptotic) regime. To be specific, we prove one-shot achievability and converse bounds on the coding rates for distributed source coding, source coding with coded side information available at the decoder and source coding under maximum distortion criterion. The one-shot bounds obtained are in terms of smooth max Renyi entropy and smooth max Renyi divergence. Our results are powerful enough to yield the results that are known for these problems in the asymptotic regime both in the i.i.d. (indep…
The problem of designing new physical-layer network coding (PNC) schemes via lattice partitions is considered. Building on a recent work by Nazer and Gastpar, who demonstrated its asymptotic gain using information-theoretic tools, we take an algebraic approach to show its potential in non-asymptotic settings. We first relate Nazer-Gastpar's approach to the fundamental theorem of finitely generated modules over a principle ideal domain. Based on this connection, we generalize their code construction and simplify their encoding and decoding methods. This not only provides a transparent understan…
We provide two results concerning the optimality of the maximum mutual information (MMI) decoder. First, we prove that the error exponents of the typical random codes under the optimal maximum likelihood (ML) decoder and the MMI decoder are equal. As a corollary to this result, we also show that the error exponents of the expurgated codes under the ML and the MMI decoders are equal. These results strengthen the well known result due to Csiszár and Körner, according to which, these decoders achieve equal random coding error exponents, since the error exponents of the typical random code and the…
We evaluate the asymptotics of equivocations, their exponents as well as their second-order coding rates under various Rényi information measures. Specifically, we consider the effect of applying a hash function on a source and we quantify the level of non-uniformity and dependence of the compressed source from another correlated source when the number of copies of the sources is large. Unlike previous works that use Shannon information measures to quantify randomness, information or uniformity, we define our security measures in terms of a more general class of information measures--the Rényi…
In this paper, an integer programming approach is introduced to construct Unequal Error Protection (UEP) codes for multiuser broadcast channels. We show that the optimal codes can be constructed that satisfy the integer programming bound. Based on the bound, we compute asymptotic code rate and perform throughput analysis for the degraded broadcast channel.
In an unreliable single-hop broadcast network setting, we investigate the throughput and decoding-delay performance of random linear network coding as a function of the coding window size and the network size. Our model consists of a source transmitting packets of a single flow to a set of $n$ users over independent erasure channels. The source performs random linear network coding (RLNC) over $k$ (coding window size) packets and broadcasts them to the users. We note that the broadcast throughput of RLNC must vanish with increasing $n$, for any fixed $k.$ Hence, in contrast to other works in t…
This paper considers the Gaussian multiple-access channel (MAC) in the asymptotic regime where the number of users grows linearly with the code length. We propose efficient coding schemes based on random linear models with approximate message passing (AMP) decoding and derive the asymptotic error rate achieved for a given user density, user payload (in bits), and user energy. The tradeoff between energy-per-bit and achievable user density (for a fixed user payload and target error rate) is studied, and it is demonstrated that in the large system limit, a spatially coupled coding scheme with AM…
We study finite-length bounds for source coding with side information for Markov sources and channel coding for channels with conditional Markovian additive noise. For this purpose, we propose two criteria for finite-length bounds. One is the asymptotic optimality and the other is the efficient computability of the bound. Then, we derive finite-length upper and lower bounds for coding length in both settings so that their computational complexity is efficient. To discuss the first criterion, we derive the large deviation bounds, the moderate deviation bounds, and second order bounds for these …
We propose a source/channel duality in the exponential regime, where success/failure in source coding parallels error/correctness in channel coding, and a distortion constraint becomes a log-likelihood ratio (LLR) threshold. We establish this duality by first deriving exact exponents for lossy coding of a memoryless source P, at distortion D, for a general i.i.d. codebook distribution Q, for both encoding success (R < R(P,Q,D)) and failure (R > R(P,Q,D)). We then turn to maximum likelihood (ML) decoding over a memoryless channel P with an i.i.d. input Q, and show that if we substitute P=QP, Q=…
The stability of scheduled multiaccess communication with random coding and independent decoding of messages is investigated. The number of messages that may be scheduled for simultaneous transmission is limited to a given maximum value, and the channels from transmitters to receiver are quasi-static, flat, and have independent fades. Requests for message transmissions are assumed to arrive according to an i.i.d. arrival process. Then, we show the following: (1) in the limit of large message alphabet size, the stability region has an interference limited information-theoretic capacity interpre…
