The error exponent of Markov channels with feedback is studied in the variable-length block-coding setting. Burnashev's classic result is extended and a single letter characterization for the reliability function of finite-state Markov channels is presented, under the assumption that the channel state is causally observed both at the transmitter and at the receiver side. Tools from stochastic control theory are used in order to treat channels with intersymbol interference. In particular the convex analytical approach to Markov decision processes is adopted to handle problems with stopping time horizons arising from variable-length coding schemes.
Distributed arithmetic coding (DAC) has been shown to be effective for Slepian-Wolf coding, especially for short data blocks. In this letter, we propose to use the DAC to compress momery-correlated sources. More specifically, the correlation between sources is modeled as a hidden Markov process. Experimental results show that the performance is close to the theoretical Slepian-Wolf limit.
The optimal zero delay coding of a finite state Markov source is considered. The existence and structure of optimal codes are studied using a stochastic control formulation. Prior results in the literature established the optimality of deterministic Markov (Walrand-Varaiya type) coding policies for finite time horizon problem, and the optimality of both deterministic nonstationary and randomized stationary policies for the infinite time horizon problem. Our main result here shows that for any irreducible and aperiodic Markov source with a finite alphabet, \emph{deterministic and stationary} Ma…
We consider the problem of compression of two memoryless binary sources, the correlation between which is defined by a Hidden Markov Model (HMM). We propose a Decision Feedback (DF) based scheme which when used with low density parity check codes results in compression close to the Slepian Wolf limits.
Optimal zero-delay coding (quantization) of $\mathbb{R}^d$-valued linearly generated Markov sources is studied under quadratic distortion. The structure and existence of deterministic and stationary coding policies that are optimal for the infinite horizon average cost (distortion) problem are established. Prior results studying the optimality of zero-delay codes for Markov sources for infinite horizons either considered finite alphabet sources or, for the $\mathbb{R}^d$-valued case, only showed the existence of deterministic and non-stationary Markov coding policies or those which are randomi…
We consider the problem of compression of two memoryless binary sources, the correlation between which is defined by a Hidden Markov Model (HMM). We propose a Decision Feedback (DF) based scheme which when used with low density parity check codes results in compression close to the Slepian Wolf limits.
We derive novel upper and lower finite-length bounds of the error probability in joint source-channel coding when the source obeys an ergodic Markov process and the channel is a Markovian additive channel or a Markovian conditional additive channel. These bounds are tight in the large and moderate deviation regimes.
The optimal causal coding of a partially observed Markov process is studied, where the cost to be minimized is a bounded, non-negative, additive, measurable single-letter function of the source and the receiver output. A structural result is obtained extending Witsenhausen's and Walrand-Varaiya's structural results on optimal real-time coders to a partially observed setting. The decentralized (multi-terminal) setup is also considered. For the case where the source is an i.i.d. process, it is shown that the optimal decentralized causal coding of correlated observations problem admits a solution…
In this paper we explore tradeoffs, regarding coding performance, between the thickness and spacing of the cutset used in Reduced Cutset Coding (RCC) of a Markov random field image model. Considering MRF models on a square lattice of sites, we show that under a stationarity condition, increasing the thickness of the cutset reduces coding rate for the cutset, increasing the spacing between components of the cutset increases the coding rate of the non-cutset pixels, though the coding rate of the latter is always strictly less than that of the former. We show that the redundancy of RCC can be dec…
Motivated by the question of whether the recently introduced Reduced Cutset Coding (RCC) offers rate-complexity performance benefits over conventional context-based conditional coding for sources with two-dimensional Markov structure, this paper compares several row-centric coding strategies that vary in the amount of conditioning as well as whether a model or an empirical table is used in the encoding of blocks of rows. The conclusion is that, at least for sources exhibiting low-order correlations, 1-sided model-based conditional coding is superior to the method of RCC for a given constraint …
In this paper, we give a distributed joint source channel coding scheme for arbitrary correlated sources for arbitrary point in the Slepian-Wolf rate region, and arbitrary link capacities using LDPC codes. We consider the Slepian-Wolf setting of two sources and one destination, with one of the sources derived from the other source by some correlation model known at the decoder. Distributed encoding and separate decoding is used for the two sources. We also give a distributed source coding scheme when the source correlation has memory to achieve any point in the Slepian-Wolf rate achievable reg…
We study sequential coding of Markov sources under an error propagation constraint. An encoder sequentially compresses a sequence of vector-sources that are spatially i.i.d. but temporally correlated according to a first-order Markov process. The channel erases up to B packets in a single burst, but reveals all other packets to the destination. The destination is required to reproduce all the source-vectors instantaneously and in a lossless manner, except those sequences that occur in an error propagation window of length B + W following the start of the erasure burst. We define the rate-recov…
We study finite-length bounds for source coding with side information for Markov sources and channel coding for channels with conditional Markovian additive noise. For this purpose, we propose two criteria for finite-length bounds. One is the asymptotic optimality and the other is the efficient computability of the bound. Then, we derive finite-length upper and lower bounds for coding length in both settings so that their computational complexity is efficient. To discuss the first criterion, we derive the large deviation bounds, the moderate deviation bounds, and second order bounds for these …
One open problem in source coding is to characterize the limits of representing losslessly a non-identity discrete function of the data encoded independently by the encoders of several correlated sources with memory. This paper investigates this problem under Markovian conditions, namely either the sources or the functions considered are Markovian. We propose using linear mappings over finite rings as encoders. If the function considered admits certain polynomial structure, the linear encoders can make use of this structure to establish "implicit collaboration" and boost the performance. In fa…
Here we write in a unified fashion (using "R(P, Q, D)") the random coding exponents in channel coding and lossy source coding. We derive their explicit forms and show, that, for a given random codebook distribution Q, the channel decoding error exponent can be viewed as an encoding success exponent in lossy source coding, and the channel correct-decoding exponent can be viewed as an encoding failure exponent in lossy source coding. We then extend the channel exponents to arbitrary D, which corresponds for D > 0 to erasure decoding and for D < 0 to list decoding. For comparison, we also derive …
In this article, we study the problem of secret key generation in the multiterminal source model, where the terminals have access to correlated Gaussian sources. We assume that the sources form a Markov chain on a tree. We give a nested lattice-based key generation scheme whose computational complexity is polynomial in the number, N , of independent and identically distributed samples observed by each source. We also compute the achievable secret key rate and give a class of examples where our scheme is optimal in the fine quantization limit. However, we also give examples that show that our s…
In this work, we derive the random coding error exponent for the uplink phase of a two-way relay system where physical layer network coding (PNC) is employed. The error exponent is derived for the practical (yet sub-optimum) XOR channel decoding setting. We show that the random coding error exponent under optimum (i.e., maximum likelihood) PNC channel decoding can be achieved even under the sub-optimal XOR channel decoding. The derived achievability bounds provide us with valuable insight and can be used as a benchmark for the performance of practical channel-coded PNC systems employing low co…
We consider the situation in which a continuous-time vector Gauss-Markov process is observed through a vector Gaussian channel (sensor) and estimated by the Kalman-Bucy filter. Unlike in standard filtering problems where a sensor model is given a priori, we are concerned with the optimal sensor design by which (i) the mutual information between the source random process and the reproduction (estimation) process is minimized, and (ii) the minimum mean-square estimation error meets a given distortion constraint. We show that such a sensor design problem is tractable by semidefinite programming. …
In this paper, we study the upper and the lower bounds on the joint source-channel coding error exponent with decoder side-information. The results in the paper are non-trivial extensions of the Csiszar's classical paper [5]. Unlike the joint source-channel coding result in [5], it is not obvious whether the lower bound and the upper bound are equivalent even if the channel coding error exponent is known. For a class of channels, including the symmetric channels, we apply a game-theoretic result to establish the existence of a saddle point and hence prove that the lower and upper bounds are th…
A new universal coding/decoding scheme for random access with collision detection is given in the case of two senders. The result is used to give an achievable joint source-channel coding error exponent for multiple access channels in the case of independent sources. This exponent is improved in a modified model that admits error free 0 rate communication between the senders.
This work contains two main contributions concerning the asymmetric broadcast channel. The first is an analysis of the exact random coding error exponents for both users, and the second is the derivation of universal decoders for both users. These universal decoders are certain variants of the maximum mutual information (MMI) universal decoder, which achieve the corresponding random coding exponents of optimal decoding. In addition, we introduce some lower bounds, which involve optimization over very few parameters, unlike the original, exact exponents, which involve minimizations over auxilia…
We consider ensembles of channel codes that are partitioned into bins, and focus on analysis of exact random coding error exponents associated with optimum decoding of the index of the bin to which the transmitted codeword belongs. Two main conclusions arise from this analysis: (i) for independent random selection of codewords within a given type class, the random coding exponent of optimal bin index decoding is given by the ordinary random coding exponent function, computed at the rate of the entire code, independently of the exponential rate of the size of the bin. (ii) for this ensemble of …
We consider the one helper source coding problem posed and investigated by Ahlswede, Körner and Wyner. In this system, the error probability of decoding goes to one as the source block length $n$ goes to infinity. This implies that we have a strong converse theorem for the one helper source coding problem. In this paper we provide a much stronger version of this strong converse theorem for the one helper source coding problem. We prove that the error probability of decoding tends to one exponentially and derive an explicit lower bound of this exponent function.
Capacity formulas and random-coding exponents are derived for a generalized family of Gel'fand-Pinsker coding problems. These exponents yield asymptotic upper bounds on the achievable log probability of error. In our model, information is to be reliably transmitted through a noisy channel with finite input and output alphabets and random state sequence, and the channel is selected by a hypothetical adversary. Partial information about the state sequence is available to the encoder, adversary, and decoder. The design of the transmitter is subject to a cost constraint. Two families of channels a…
This paper is about deriving lower bounds on the error exponents for the two-user interference channel under the random coding regime for several ensembles. Specifically, we first analyze the standard random coding ensemble, where the codebooks are comprised of independently and identically distributed (i.i.d.) codewords. For this ensemble, we focus on optimum decoding, which is in contrast to other, suboptimal decoding rules that have been used in the literature (e.g., joint typicality decoding, treating interference as noise, etc.). The fact that the interfering signal is a codeword, rather …
We analyze the optimal trade-off between the error exponent and the excess-rate exponent for variable-rate Slepian-Wolf codes. In particular, we first derive upper (converse) bounds on the optimal error and excess-rate exponents, and then lower (achievable) bounds, via a simple class of variable-rate codes which assign the same rate to all source blocks of the same type class. Then, using the exponent bounds, we derive bounds on the optimal rate functions, namely, the minimal rate assigned to each type class, needed in order to achieve a given target error exponent. The resulting excess-rate e…
This paper studies expurgated random-coding bounds and exponents for channel coding with a given (possibly suboptimal) decoding rule. Variations of Gallager's analysis are presented, yielding several asymptotic and non-asymptotic bounds on the error probability for an arbitrary codeword distribution. A simple non-asymptotic bound is shown to attain an exponent of Csiszár and Körner under constant-composition coding. Using Lagrange duality, this exponent is expressed in several forms, one of which is shown to permit a direct derivation via cost-constrained coding which extends to infinite and c…
In 1973, Gallager proved that the random-coding bound is exponentially tight for the random code ensemble at all rates, even below expurgation. This result explained that the random-coding exponent does not achieve the expurgation exponent due to the properties of the random ensemble, irrespective of the utilized bounding technique. It has been conjectured that this same behavior holds true for a random ensemble of linear codes. This conjecture is proved in this paper. Additionally, it is shown that this property extends to Poltyrev's random-coding exponent for a random ensemble of lattices.
We investigate the computation of Csiszar's bounds for the joint source-channel coding (JSCC) error exponent, E_J, of a communication system consisting of a discrete memoryless source and a discrete memoryless channel. We provide equivalent expressions for these bounds and derive explicit formulas for the rates where the bounds are attained. These equivalent representations can be readily computed for arbitrary source-channel pairs via Arimoto's algorithm. When the channel's distribution satisfies a symmetry property, the bounds admit closed-form parametric expressions. We then use our results…
The paper presents exponentially-strong converses for source-coding, channel coding, and hypothesis testing problems. More specifically, it presents alternative proofs for the well-known exponentially-strong converse bounds for almost lossless source-coding with side-information and for channel coding over a discrete memoryless channel (DMC). These alternative proofs are solely based on a change of measure argument on the sets of conditionally or jointly typical sequences that result in a correct decision, and on the analysis of these measures in the asymptotic regime of infinite blocklengths.…
In this paper, we consider the problem of secure and reliable communication with uncertain channel state information (CSI) and present a new solution named active secure coding which combines the machine learning methods with the traditional physical layer secure coding scheme. First, we build a detectable wiretap channel model by combining the hidden Markov model with the compound wiretap channel model, in which the varying of channel block CSI is a Markov process and the detected information is a stochastic emission from the current CSI. Next, we present a CSI learning scheme to learn the CS…
This work presents a polar coding scheme for fading channels, focusing primarily on fading binary symmetric and additive exponential noise channels. For fading binary symmetric channels, a hierarchical coding scheme is presented, utilizing polar coding both over channel uses and over fading blocks. The receiver uses its channel state information (CSI) to distinguish states, thus constructing an overlay erasure channel over the underlying fading channels. By using this scheme, the capacity of a fading binary symmetric channel is achieved without CSI at the transmitter. Noting that a fading AWGN…
The error exponent of a discrete memoryless channel is expressed in two forms. One is Gallager's expression with a positive slope parameter and the other is Csiszar and Korner's information-theoretic representation expressed using the mutual information and the relative entropy. They differ in appearance, and existing methods to prove their agreement are not elementary, as they require an evaluation of the KKT conditions that the optimal distribution must satisfy. Similarly, there are two types of expressions for the strong converse exponent. They are Arimoto's expression with a negative slope…
A general method of source coding over expansion is proposed in this paper, which enables one to reduce the problem of compressing an analog (continuous-valued source) to a set of much simpler problems, compressing discrete sources. Specifically, the focus is on lossy compression of exponential and Laplacian sources, which is subsequently expanded using a finite alphabet prior to being quantized. Due to decomposability property of such sources, the resulting random variables post expansion are independent and discrete. Thus, each of the expanded levels corresponds to an independent discrete so…
We propose two coding schemes combining decode-forward (DF) and noisy network coding (NNC) with different flavors. The first is a combined DF-NNC scheme for the one-way relay channel which includes both DF and NNC as special cases by performing rate splitting, partial block Markov encoding and NNC. The second combines two different DF strategies and layered NNC for the two-way relay channel. One DF strategy performs coherent block Markov encoding at the source at the cost of power splitting at the relay, the other performs independent source and relay encoding but with full relay power, and la…
In the distributed coding of correlated sources, the problem of characterizing the joint probability distribution of a pair of random variables satisfying an n-letter Markov chain arises. The exact solution of this problem is intractable. In this paper, we seek a single-letter necessary condition for this n-letter Markov chain. To this end, we propose a new data processing inequality on a new measure of correlation by means of spectrum analysis. Based on this new data processing inequality, we provide a single-letter necessary condition for the required joint probability distribution. We apply…
We derive various error exponents in the bee identification problem under two different decoding rules. Under naïve decoding, which decodes each bee independently of the others, we analyze a general discrete memoryless channel and a relatively wide family of stochastic decoders. Upper and lower bounds to the random coding error exponent are derived and proved to be equal at relatively high coding rates. Then, we propose a lower bound on the error exponent of the typical random code, which improves upon the random coding exponent at low coding rates. We also derive a third bound, which is relat…
Let $X,Y$ be jointly Gaussian vectors, and consider random variables $U,V$ that satisfy the Markov constraint $U-X-Y-V$. We prove an extremal inequality relating the mutual informations between all ${4 \choose 2}$ pairs of random variables from the set $(U,X,Y,V)$. As a first application, we show that the rate region for the two-encoder quadratic Gaussian source coding problem follows as an immediate corollary of the the extremal inequality. In a second application, we establish the rate region for a vector-Gaussian source coding problem where Löwner-John ellipsoids are approximated based on r…
Strong typicality and the Markov lemma have been used in the proofs of several multiterminal source coding theorems. Since these two tools can be applied to finite alphabets only, the results proved by them are subject to the same limitation. Recently, a new notion of typicality, namely unified typicality, has been defined. It can be applied to both finite or countably infinite alphabets, and it retains the asymptotic equipartition property and the structural properties of strong typicality. In this paper, unified typicality is used to derive a version of the Markov lemma which works on both f…
This paper studies the random-coding exponent of joint source-channel coding for the multiple-access channel with correlated sources. For each user, by defining a threshold, the messages of each source are partitioned into two classes. The achievable exponent for correlated sources with two message-dependent input distributions for each user is determined and shown to be larger than that achieved using only one input distribution for each user. A system of equations is presented to determine the optimal thresholds maximizing the achievable exponent. The obtained exponent is compared with the o…
In this work, a new upper bound for average error probability of a two-user discrete memoryless (DM) multiple-access channel (MAC) is derived. This bound can be universally obtained for all discrete memoryless MACs with given input and output alphabets. This is the first bound of this type that explicitly uses the method of expurgation. It is shown that the exponent of this bound is greater than or equal to those of previously known bounds.
We analyze the exact exponential decay rate of the expected amount of information leaked to the wiretapper in Wyner's wiretap channel setting using wiretap channel codes constructed from both i.i.d. and constant-composition random codes. Our analysis for those sampled from i.i.d. random coding ensemble shows that the previously-known achievable secrecy exponent using this ensemble is indeed the exact exponent for an average code in the ensemble. Furthermore, our analysis on wiretap channel codes constructed from the ensemble of constant-composition random codes leads to an exponent which, in a…
An effective 'on-the-fly' mechanism for stochastic lossy coding of Markov sources using string matching techniques is proposed in this paper. Earlier work has shown that the rate-distortion bound can be asymptotically achieved by a 'natural type selection' (NTS) mechanism which iteratively encodes asymptotically long source strings (from an unknown source distribution P) and regenerates the codebook according to a maximum likelihood distribution framework, after observing a set of K codewords to 'd-match' (i.e., satisfy the distortion constraint for) a respective set of K source words. This re…
We derive various error exponents for communication channels with random states, which are available non-causally at the encoder only. For both the finite-alphabet Gel'fand-Pinsker channel and its Gaussian counterpart, the dirty-paper channel, we derive random coding exponents, error exponents of the typical random codes (TRCs), and error exponents of expurgated codes. For the two channel models, we analyze some sub-optimal bin-index decoders, which turn out to be asymptotically optimal, at least for the random coding error exponent. For the dirty-paper channel, we show explicitly via a numeri…
We study error exponents for source coding with side information. Both achievable exponents and converse bounds are obtained for the following two cases: lossless source coding with coded information (SCCSI) and lossy source coding with full side information (Wyner-Ziv). These results recover and extend several existing results on source-coding error exponents and are tight in some circumstances. Our bounds have a natural interpretation as a two-player game between nature and the code designer, with nature seeking to minimize the exponent and the code designer seeking to maximize it. In the Wy…
An upper bound on the error probability of specific lattices, based on their distance-spectrum, is constructed. The derivation is accomplished using a simple alternative to the Minkowski-Hlawka mean-value theorem of the geometry of numbers. In many ways, the new bound greatly resembles the Shulman-Feder bound for linear codes. Based on the new bound, an error-exponent is derived for specific lattice sequences (of increasing dimension) over the AWGN channel. Measuring the sequence's gap to capacity, using the new exponent, is demonstrated.
Second-order coding rate of channel coding is discussed for general sequence of channels. The optimum second-order transmission rate with a constant error constraint $ε$ is obtained by using the information spectrum method. We apply this result to the discrete memoryless case, the discrete memoryless case with a cost constraint, the additive Markovian case, and the Gaussian channel case with an energy constraint. We also clarify that the Gallager bound does not give the optimum evaluation in the second-order coding rate.
This paper considers the problem of channel coding with a given (possibly suboptimal) maximum-metric decoding rule. A cost-constrained random-coding ensemble with multiple auxiliary costs is introduced, and is shown to achieve error exponents and second-order coding rates matching those of constant-composition random coding, while being directly applicable to channels with infinite or continuous alphabets. The number of auxiliary costs required to match the error exponents and second-order rates of constant-composition coding is studied, and is shown to be at most two. For i.i.d. random coding…
An enhanced covering lemma for a Markov chain is proved in this paper, and then the distributed source coding problem of correlated general sources with one average distortion criterion under fixed-length coding is investigated. Based on the enhanced lemma, a sufficient and necessary condition for determining the achievability of rate-distortion triples is given.
In this paper, we consider the problem of variable-length source coding allowing errors. The exponential moment of the codeword length is analyzed in the non-asymptotic regime and in the asymptotic regime. Our results show that the smooth Renyi entropy characterizes the optimal exponential moment of the codeword length.
This paper characterizes the optimal type-II error exponent for a distributed hypothesis testing-against-independence problem when the \emph{expected} rate of the sensor-detector link is constrained. Unlike for the well-known Ahlswede-Csiszar result that holds under a \emph{maximum} rate constraint and where a strong converse holds, here the optimal exponent depends on the allowed type-I error exponent. Specifically, if the type-I error probability is limited by $ε$, then the optimal type-II error exponent under an \emph{expected} rate constraint $R$ coincides with the optimal type-II error ex…
