Lossless variable-length source coding with codeword cost is considered for general sources. The problem setting, where we impose on unequal costs on code symbols, is called the variable-length coding with codeword cost. In this problem, the infimum of average codeword cost have been determined for general sources. On the other hand, overflow probability, which is defined as the probability of codeword cost being above a threshold, have not been considered yet. In this paper, we determine the infimum of achievable threshold in the first-order sense and the second-order sense for general sources and compute it for some special sources such as i.i.d. sources and mixed sources. A relationship between the overflow probability of variable-length coding and the error probability of fixed-length coding is also revealed. Our analysis is based on the information-spectrum methods.
Lossless variable-length source coding with unequal cost function is considered for general sources. In this problem, the codeword cost instead of codeword length is important. The infimum of average codeword cost has already been determined for general sources. We consider the overflow probability of codeword cost and determine the infimum of achievable overflow threshold. Our analysis is on the basis of information-spectrum methods and hence valid through the general source.
The variable-length source coding problem allowing the error probability up to some constant is considered for general sources. In this problem the optimum mean codeword length of variable-length codes has already been determined. On the other hand, in this paper, we focus on the overflow (or excess codeword length) probability instead of the mean codeword length. The infimum of overflow thresholds under the constraint that both of the error probability and the overflow probability are smaller than or equal to some constant is called the optimum overflow threshold. In this paper, we first deri…
This paper investigates the problem of variable-length lossy source coding allowing a positive excess distortion probability and an overflow probability of codeword lengths. Novel one-shot achievability and converse bounds of the optimal rate are established by a new quantity based on the smooth max entropy (the smooth Rényi entropy of order zero). To derive the achievability bounds, we give an explicit code construction based on a distortion ball instead of using the random coding argument. The basic idea of the code construction is similar to the optimal code construction in the variable-len…
This paper presents lossless prefix codes optimized with respect to a pay-off criterion consisting of a convex combination of maximum codeword length and average codeword length. The optimal codeword lengths obtained are based on a new coding algorithm which transforms the initial source probability vector into a new probability vector according to a merging rule. The coding algorithm is equivalent to a partition of the source alphabet into disjoint sets on which a new transformed probability vector is defined as a function of the initial source probability vector and a scalar parameter. The p…
In this paper we study the problem of increasing the decoding success probability of random linear fountain code over GF(2) for small packet lengths used in delay-intolerant applications such as multimedia streaming. Such code over GF(2) are attractive as they have lower decoding complexity than codes over larger field size, but suffer from high transmission redundancy. In our proposed coding scheme we construct a codeword which is not a linear combination of any codewords previously transmitted to mitigate such transmission redundancy. We then note the observation that the probability of rece…
Let $P = \{p(i)\}$ be a measure of strictly positive probabilities on the set of nonnegative integers. Although the countable number of inputs prevents usage of the Huffman algorithm, there are nontrivial $P$ for which known methods find a source code that is optimal in the sense of minimizing expected codeword length. For some applications, however, a source code should instead minimize one of a family of nonlinear objective functions, $β$-exponential means, those of the form $\log_a \sum_i p(i) a^{n(i)}$, where $n(i)$ is the length of the $i$th codeword and $a$ is a positive constant. Applic…
Suitable estimators for a class of Large Deviation approximations of rare event probabilities based on sample realizations of random processes have been proposed in our earlier work. These estimators are expressed as non-linear multi-dimensional optimization problems of a special structure. In this paper, we develop an algorithm to solve these optimization problems very efficiently based on their characteristic structure. After discussing the nature of the objective function and constraint set and their peculiarities, we provide a formal proof that the developed algorithm is guaranteed to alwa…
This paper studies the effect of optimal power allocation on the performance of communication systems utilizing automatic repeat request (ARQ). Considering Type-I ARQ, the problem is cast as the minimization of the outage probability subject to an average power constraint. The analysis is based on some recent results on the achievable rates of finite-length codes and we investigate the effect of codewords length on the performance of ARQ-based systems. We show that the performance of ARQ protocols is (almost) insensitive to the length of the codewords, for codewords of length $\ge 50$ channel …
We study the tradeoff between the sum rate and the error probability in downlink of wireless networks. Using the recent results on the achievable rates of finite-length codewords, the problem is cast as a joint optimization of the network sum rate and the per-user error probability. Moreover, we develop an efficient algorithm based on the divide-and-conquer technique to simultaneously maximize the network sum rate and minimize the maximum users' error probability and to evaluate the effect of the codewords length on the system performance. The results show that, in delay-constrained scenarios,…
This paper presents prefix codes which minimize various criteria constructed as a convex combination of maximum codeword length and average codeword length or maximum redundancy and average redundancy, including a convex combination of the average of an exponential function of the codeword length and the average redundancy. This framework encompasses as a special case several criteria previously investigated in the literature, while relations to universal coding is discussed. The coding algorithm derived is parametric resulting in re-adjusting the initial source probabilities via a weighted pr…
Ultra-Reliable Low-Latency Communications have stringent delay constraints, and hence use codes with small block length (short codewords). In these cases, classical models that provide good approximations to systems with infinitely long codewords become imprecise. To remedy this, in this paper, an average coding rate expression is derived for a large scale network with short codewords using stochastic geometry and the theory of coding in the finite blocklength regime. The average coding rate and upper and lower bounds on the outage probability of the large-scale network are derived, and a tigh…
We consider a variant of the channel simulation problem with a single input and multiple outputs, where Alice observes a probability distribution $P$ from a set of prescribed probability distributions $\mathbb{\mathcal{P}}$, and sends a prefix-free codeword $W$ to Bob to allow him to generate $n$ i.i.d. random variables $X_{1},X_{2,}...,X_{n}$ which follow the distribution $P$. This can also be regarded as a lossy compression setting for probability distributions. This paper describes encoding schemes for three cases of $P$: $P$ is a distribution over positive integers, $P$ is a continuous dis…
In this paper, the linear programming (LP) decoder for binary linear codes, introduced by Feldman, et al. is extended to joint-decoding of binary-input finite-state channels. In particular, we provide a rigorous definition of LP joint-decoding pseudo-codewords (JD-PCWs) that enables evaluation of the pairwise error probability between codewords and JD-PCWs. This leads naturally to a provable upper bound on decoder failure probability. If the channel is a finite-state intersymbol interference channel, then the LP joint decoder also has the maximum-likelihood (ML) certificate property and all in…
This paper studies the outage probability and the throughput of amplify-and-forward relay networks with wireless information and energy transfer. We use some recent results on finite block-length codes to analyze the system performance in the cases with short codewords. Specifically, the time switching relaying and the power splitting relaying protocols are considered for energy and information transfer. We derive tight approximations for the outage probability/throughput. Then, we analyze the outage probability in asymptotically high signal-to-noise ratios. Finally, we use numerical results t…
The DNA storage channel is considered, in which a codeword is comprised of $M$ unordered DNA molecules. At reading time, $N$ molecules are sampled with replacement, and then each molecule is sequenced. A coded-index concatenated-coding scheme is considered, in which the $m$th molecule of the codeword is restricted to a subset of all possible molecules (an inner code), which is unique for each $m$. The decoder has low-complexity, and is based on first decoding each molecule separately (the inner code), and then decoding the sequence of molecules (an outer code). Only mild assumptions are made o…
A new channel coding approach was proposed in [1] for random multiple access communication over the discrete-time memoryless channel. The coding approach allows users to choose their communication rates independently without sharing the rate information among each other or with the receiver. The receiver will either decode the message or report a collision depending on whether reliable message recovery is possible. It was shown that, asymptotically as the codeword length goes to infinity, the set of communication rates supporting reliable message recovery can be characterized by an achievable …
A distribution matcher (DM) encodes a binary input data sequence into a sequence of symbols (codeword) with desired target probability distribution. The set of the output codewords constitutes a codebook (or code) of a DM. Constant-composition DM (CCDM) uses arithmetic coding to efficiently encode data into codewords from a constant-composition (CC) codebook. The CC constraint limits the size of the codebook, and hence the coding rate of the CCDM. The performance of CCDM degrades with decreasing output length. To improve the performance for short transmission blocks we present a class of multi…
This paper considers the achievability and converse bounds on the maximal channel coding rate at a given blocklength and error probability over AWGN channels. The problem stems from covert communication with Gaussian codewords. By re-visiting [18], we first present new and more general achievability bounds for random coding schemes under maximal or average probability of error requirements. Such general bounds are then applied to covert communication in AWGN channels where codewords are generated from Gaussian distribution while meeting the maximal power constraint. Further comparison is made …
In this paper we consider lossless source coding for a class of sources specified by the total variational distance ball centred at a fixed nominal probability distribution. The objective is to find a minimax average length source code, where the minimizers are the codeword lengths -- real numbers for arithmetic or Shannon codes -- while the maximizers are the source distributions from the total variational distance ball. Firstly, we examine the maximization of the average codeword length by converting it into an equivalent optimization problem, and we give the optimal codeword lenghts via a w…
Let $P = \{p(i)\}$ be a measure of strictly positive probabilities on the set of nonnegative integers. Although the countable number of inputs prevents usage of the Huffman algorithm, there are nontrivial $P$ for which known methods find a source code that is optimal in the sense of minimizing expected codeword length. For some applications, however, a source code should instead minimize one of a family of nonlinear objective functions, $β$-exponential means, those of the form $\log_a \sum_i p(i) a^{n(i)}$, where $n(i)$ is the length of the $i$th codeword and $a$ is a positive constant. Applic…
We consider the decoding of linear and array codes from errors when we are only allowed to download a part of the codeword. More specifically, suppose that we have encoded $k$ data symbols using an $(n,k)$ code with code length $n$ and dimension $k.$ During storage, some of the codeword coordinates might be corrupted by errors. We aim to recover the original data by reading the corrupted codeword with a limit on the transmitting bandwidth, namely, we can only download an $α$ proportion of the corrupted codeword. For a given $α,$ our objective is to design a code and a decoding scheme such that…
We consider a peak-power-limited single-antenna block-stationary Gaussian fading channel where neither the transmitter nor the receiver knows the channel state information, but both know the channel statistics. This model subsumes most previously studied Gaussian fading models. We first compute the asymptotic channel capacity in the high SNR regime and show that the behavior of channel capacity depends critically on the channel model. For the special case where the fading process is symbol-by-symbol stationary, we also reveal a fundamental interplay between the codeword length, communication r…
In this work, we present the first local-decoding algorithm for expander codes. This yields a new family of constant-rate codes that can recover from a constant fraction of errors in the codeword symbols, and where any symbol of the codeword can be recovered with high probability by reading $N^ε$ symbols from the corrupted codeword, where $N$ is the block-length of the code. Expander codes, introduced by Sipser and Spielman, are formed from an expander graph $G = (V,E)$ of degree $d$, and an inner code of block-length $d$ over an alphabet $Σ$. Each edge of the expander graph is associated with…
We present extensions to Raghavan and Baum's reliability-output Viterbi algorithm (ROVA) to accommodate tail-biting convolutional codes. These tail-biting reliability-output algorithms compute the exact word-error probability of the decoded codeword after first calculating the posterior probability of the decoded tail-biting codeword's starting state. One approach employs a state-estimation algorithm that selects the maximum a posteriori state based on the posterior distribution of the starting states. Another approach is an approximation to the exact tail-biting ROVA that estimates the word-e…
We introduce the concept of an \ff-maximal error-detecting block code, for some parameter \ff{} between 0 and 1, in order to formalize the situation where a block code is close to maximal with respect to being error-detecting. Our motivation for this is that constructing a maximal error-detecting code is a computationally hard problem. We present a randomized algorithm that takes as input two positive integers $N,\ell$, a probability value \ff, and a specification of the errors permitted in some application, and generates an error-detecting, or error-correcting, block code having up to $N$ cod…
We present nonasymptotic achievability and converse bounds on the maximum coding rate (for a fixed average error probability and a fixed average blocklength) of variable-length full-feedback (VLF) and variable-length stop-feedback (VLSF) codes operating over a binary erasure channel (BEC). For the VLF setup, the achievability bound relies on a scheme that maps each message onto a variable-length Huffman codeword and then repeats each bit of the codeword until it is received correctly. The converse bound is inspired by the meta-converse framework by Polyanskiy, Poor, and Verdú (2010) and relies…
The construction of asymmetric error correcting codes is a topic that was studied extensively, however, the existing approach for code construction assumes that every codeword should tolerate $t$ asymmetric errors. Our main observation is that in contrast to symmetric errors, asymmetric errors are content dependent. For example, in Z-channels, the all-1 codeword is prone to have more errors than the all-0 codeword. This motivates us to develop nonuniform codes whose codewords can tolerate different numbers of asymmetric errors depending on their Hamming weights. The idea in a nonuniform codes'…
Universal coding of integers~(UCI) is a class of variable-length code, such that the ratio of the expected codeword length to $\max\{1,H(P)\}$ is within a constant factor, where $H(P)$ is the Shannon entropy of the decreasing probability distribution $P$. However, if we consider the ratio of the expected codeword length to $H(P)$, the ratio tends to infinity by using UCI, when $H(P)$ tends to zero. To solve this issue, this paper introduces a class of codes, termed generalized universal coding of integers~(GUCI), such that the ratio of the expected codeword length to $H(P)$ is within a constan…
Undetected errors are important for linear codes, which are the only type of errors after hard decision and automatic-repeat-request (ARQ), but do not receive much attention on their correction. In concatenated channel coding, suboptimal source coding and joint source-channel coding, constrains among successive codewords may be utilized to improve decoding performance. In this paper, list decoding is used to correct the undetected errors. The benefit proportion of the correction is obviously improved especially on Hamming codes and Reed-Muller codes, which achieves about 40% in some cases. But…
In this paper, new two-user multiple access wiretap channel models are studied. First, the multiple access wiretap channel II with a discrete memoryless main channel, under three different wiretapping scenarios, is introduced. The wiretapper, as in the classical wiretap channel II model, chooses a fixed-length subset of the channel uses on which she obtains noise-free observations of one of the codewords, a superposition of the two codewords, or each of the two codewords. These thus extend the recently examined wiretap channel II with a noisy main channel to a multiple access setting with a va…
In this paper we study codes for correcting deletable errors in binary words, where each bit is either retained, substituted, erased or deleted and the total number of errors is much smaller compared to the length of the codeword. We construct codes capable of correcting errors in the received codeword in real-time with small delay and low probability of decoding error.
Huffman coding finds an optimal prefix code for a given probability mass function. Consider situations in which one wishes to find an optimal code with the restriction that all codewords have lengths that lie in a user-specified set of lengths (or, equivalently, no codewords have lengths that lie in a complementary set). This paper introduces a polynomial-time dynamic programming algorithm that finds optimal codes for this reserved-length prefix coding problem. This has applications to quickly encoding and decoding lossless codes. In addition, one modification of the approach solves any quasia…
This paper studies several properties of channel codes that approach the fundamental limits of a given (discrete or Gaussian) memoryless channel with a non-vanishing probability of error. The output distribution induced by an $ε$-capacity-achieving code is shown to be close in a strong sense to the capacity achieving output distribution. Relying on the concentration of measure (isoperimetry) property enjoyed by the latter, it is shown that regular (Lipschitz) functions of channel outputs can be precisely estimated and turn out to be essentially non-random and independent of the actual code. It…
In developing locally testable codes, information is added to the coded message by creating redundancy in the codewords. In this article, we propose an alternative method in which redundancy is introduced on the message that must be transmitted before its encoding and not in the codewords. This approach exploits the Set Shaping Theory whose goal is the study the bijection functions f(X)=Y that transform a set of strings into a set of equal size made up of strings of greater length. In this way, this type of function transforms the independent variable x into the dependent variable y whose emis…
Text compression schemes and compact data structures usually combine sophisticated probability models with basic coding methods whose average codeword length closely match the entropy of known distributions. In the frequent case where basic coding represents run-lengths of outcomes that have probability $p$, i.e. the geometric distribution $\Pr(i)=p^i(1-p)$, a \emph{Golomb code} is an optimal instantaneous code, which has the additional advantage that codewords can be computed using only an integer parameter calculated from $p$, without need for a large or sophisticated data structure. Golomb …
We consider upper bounds on the error probability in channel coding. We derive an improved maximum-likelihood union bound, which takes into account events where the likelihood of the correct codeword is tied with that of some competitors. We compare this bound to various previous results, both qualitatively and quantitatively. With respect to maximal error probability of linear codes, we observe that when the channel is additive, the derivation of bounds, as well as the assumptions on the admissible encoder and decoder, simplify considerably.
A locally testable code (LTC) is an error correcting code with a property tester. The tester tests if a word is codeword by reading constant random bits and rejects the word with probability proportional to the distance from the word to the closest codeword. An important open question until recently is whether there exist $c^3$-LTCs which are LTCs with constant rate, constant relative distance and constant locality. In this work, we construct a new LTC family using 1-sided lossless expanders and balanced products.
We introduce randomized Limited View (LV) adversary codes that provide protection against an adversary that uses their partial view of the communication to construct an adversarial error vector to be added to the channel. For a codeword of length N, the adversary selects a subset of ρ_rN of the codeword components to "see", and then "adds" an adversarial error vector of weight ρ_wN to the codeword. Performance of the code is measured by the probability of the decoder failure in recovering the sent message. An (N, q^{RN},δ)-limited view adversary code ensures that the success chance of the adve…
The Slepian-Wolf (SW) coding system is a source coding system with two encoders and a decoder, where these encoders independently encode source sequences from two correlated sources into codewords, and the decoder reconstructs both source sequences from the codewords. In this paper, we consider the situation in which the SW coding system is asynchronous, i.e., each encoder samples a source sequence with some unknown delay. We assume that delays are unknown but maximum and minimum values of possible delays are known to encoders and the decoder. We also assume that sources are discrete stationar…
A status updating system is considered in which a variable length code is used to transmit messages to a receiver over a noisy channel. The goal is to optimize the codewords lengths such that successfully-decoded messages are timely. That is, such that the age-of-information (AoI) at the receiver is minimized. A hybrid ARQ (HARQ) scheme is employed, in which variable-length incremental redundancy (IR) bits are added to the originally-transmitted codeword until decoding is successful. With each decoding attempt, a non-zero processing delay is incurred. The optimal codewords lengths are analytic…
We consider decoding of binary Tanner codes using message-passing iterative decoding and linear programming (LP) decoding in MBIOS channels. We present new certificates that are based on a combinatorial characterization for local-optimality of a codeword in irregular Tanner codes with respect to any MBIOS channel. This characterization is based on a conical combination of normalized weighted subtrees in the computation trees of the Tanner graph. These subtrees may have any finite height h (even equal or greater than half of the girth of the Tanner graph). In addition, the degrees of local-code…
Over a finite field $\F_q$ the $(n,d,q)$-Reed-Muller code is the code given by evaluations of $n$-variate polynomials of total degree at most $d$ on all points (of $\F_q^n$). The task of testing if a function $f:\F_q^n \to \F_q$ is close to a codeword of an $(n,d,q)$-Reed-Muller code has been of central interest in complexity theory and property testing. The query complexity of this task is the minimal number of queries that a tester can make (minimum over all testers of the maximum number of queries over all random choices) while accepting all Reed-Muller codewords and rejecting words that ar…
An encoder, subject to a rate constraint, wishes to describe a Gaussian source under squared error distortion. The decoder, besides receiving the encoder's description, also observes side information consisting of uncompressed source symbol subject to slow fading and noise. The decoder knows the fading realization but the encoder knows only its distribution. The rate-distortion function that simultaneously satisfies the distortion constraints for all fading states was derived by Heegard and Berger. A layered encoding strategy is considered in which each codeword layer targets a given fading st…
An index code for broadcast channel with receiver side information is locally decodable if each receiver can decode its demand by observing only a subset of the transmitted codeword symbols instead of the entire codeword. Local decodability in index coding is known to reduce receiver complexity, improve user privacy and decrease decoding error probability in wireless fading channels. Conventional index coding solutions assume that the receivers observe the entire codeword, and as a result, for these codes the number of codeword symbols queried by a user per decoded message symbol, which we ref…
The partition function pertaining to finite--temperature decoding of a (typical) randomly chosen code is known to have three types of behavior, corresponding to three phases in the plane of rate vs. temperature: the {\it ferromagnetic phase}, corresponding to correct decoding, the {\it paramagnetic phase}, of complete disorder, which is dominated by exponentially many incorrect codewords, and the {\it glassy phase} (or the condensed phase), where the system is frozen at minimum energy and dominated by subexponentially many incorrect codewords. We show that the statistical physics associated wi…
This paper considers the joint-decoding (JD) problem for finite-state channels (FSCs) and low-density parity-check (LDPC) codes. In the first part, the linear-programming (LP) decoder for binary linear codes is extended to JD of binary-input FSCs. In particular, we provide a rigorous definition of LP joint-decoding pseudo-codewords (JD-PCWs) that enables evaluation of the pairwise error probability between codewords and JD-PCWs in AWGN. This leads naturally to a provable upper bound on decoder failure probability. If the channel is a finite-state intersymbol interference channel, then the join…
For the additive white Gaussian noise channel with average codeword power constraint, sparse superposition codes are developed. These codes are based on the statistical high-dimensional regression framework. The paper [IEEE Trans. Inform. Theory 55 (2012), 2541 - 2557] investigated decoding using the optimal maximum-likelihood decoding scheme. Here a fast decoding algorithm, called adaptive successive decoder, is developed. For any rate R less than the capacity C communication is shown to be reliable with exponentially small error probability.
