Pattern Coding Meets Censoring: (almost) Adaptive Coding on Countable Alphabets

On redundancy of memoryless sources over countable alphabets

The minimum average number of bits need to describe a random variable is its entropy, assuming knowledge of the underlying statistics On the other hand, universal compression supposes that the distribution of the random variable, while unknown, belongs to a known set $\cal P$ of distributions. Such universal descriptions for the random variable are agnostic to the identity of the distribution in $\cal P$. But because they are not matched exactly to the underlying distribution of the random variable, the average number of bits they use is higher, and the excess over the entropy used is the "red…

Universal Coding on Infinite Alphabets: Exponentially Decreasing Envelopes

This paper deals with the problem of universal lossless coding on a countable infinite alphabet. It focuses on some classes of sources defined by an envelope condition on the marginal distribution, namely exponentially decreasing envelope classes with exponent $α$. The minimax redundancy of exponentially decreasing envelope classes is proved to be equivalent to $\frac{1}{4 α\log e} \log^2 n$. Then a coding strategy is proposed, with a Bayes redundancy equivalent to the maximin redundancy. At last, an adaptive algorithm is provided, whose redundancy is equivalent to the minimax redundancy

Prefix Codes for Power Laws with Countable Support

In prefix coding over an infinite alphabet, methods that consider specific distributions generally consider those that decline more quickly than a power law (e.g., Golomb coding). Particular power-law distributions, however, model many random variables encountered in practice. For such random variables, compression performance is judged via estimates of expected bits per input symbol. This correspondence introduces a family of prefix codes with an eye towards near-optimal coding of known distributions. Compression performance is precisely estimated for well-known probability distributions usin…

A note on the definition of sliding block codes and the Curtis-Hedlund-Lyndon Theorem

In this note we propose an alternative definition for sliding block codes between shift spaces. This definition coincides with the usual definition in the case that the shift space is defined on a finite alphabet, but it encompass a larger class of maps when the alphabet is infinite. In any case, the proposed definition keeps the idea that a sliding block code is a map with a local rule. Using this new definition we prove that the Curtis-Hedlund-Lyndon Theorem always holds for shift spaces over countable alphabets.

每个有限参考测度都存在通用密度

Universal Densities Exist for Every Finite Reference Measure

本文将通用编码推广为可数生成可测空间上的通用密度问题，证明有限参考测度下存在能一致估计微分熵率的通用密度，通过改进NPD熵率估计器构建实例，并导出通用预测器及无限参考测度下的一致估计充分条件。

Optimal Prefix Codes for Infinite Alphabets with Nonlinear Costs

Let $P = \{p(i)\}$ be a measure of strictly positive probabilities on the set of nonnegative integers. Although the countable number of inputs prevents usage of the Huffman algorithm, there are nontrivial $P$ for which known methods find a source code that is optimal in the sense of minimizing expected codeword length. For some applications, however, a source code should instead minimize one of a family of nonlinear objective functions, $β$-exponential means, those of the form $\log_a \sum_i p(i) a^{n(i)}$, where $n(i)$ is the length of the $i$th codeword and $a$ is a positive constant. Applic…

Infinite-Alphabet Prefix Codes Optimal for $β$-Exponential Penalties

Let $P = \{p(i)\}$ be a measure of strictly positive probabilities on the set of nonnegative integers. Although the countable number of inputs prevents usage of the Huffman algorithm, there are nontrivial $P$ for which known methods find a source code that is optimal in the sense of minimizing expected codeword length. For some applications, however, a source code should instead minimize one of a family of nonlinear objective functions, $β$-exponential means, those of the form $\log_a \sum_i p(i) a^{n(i)}$, where $n(i)$ is the length of the $i$th codeword and $a$ is a positive constant. Applic…

Benefiting from Disorder: Source Coding for Unordered Data

The order of letters is not always relevant in a communication task. This paper discusses the implications of order irrelevance on source coding, presenting results in several major branches of source coding theory: lossless coding, universal lossless coding, rate-distortion, high-rate quantization, and universal lossy coding. The main conclusions demonstrate that there is a significant rate savings when order is irrelevant. In particular, lossless coding of n letters from a finite alphabet requires Theta(log n) bits and universal lossless coding requires n + o(n) bits for many countable alpha…

Invariance: a Theoretical Approach for Coding Sets of Words Modulo Literal (Anti)Morphisms

Let $A$ be a finite or countable alphabet and let $θ$ be literal (anti)morphism onto $A^*$ (by definition, such a correspondence is determinated by a permutation of the alphabet). This paper deals with sets which are invariant under $θ$ ($θ$-invariant for short).We establish an extension of the famous defect theorem. Moreover, we prove that for the so-called thin $θ$-invariant codes, maximality and completeness are two equivalent notions. We prove that a similar property holds in the framework of some special families of $θ$-invariant codes such as prefix (bifix) codes, codes with a finite dec…

Universal compression of Gaussian sources with unknown parameters

For a collection of distributions over a countable support set, the worst case universal compression formulation by Shtarkov attempts to assign a universal distribution over the support set. The formulation aims to ensure that the universal distribution does not underestimate the probability of any element in the support set relative to distributions in the collection. When the alphabet is uncountable and we have a collection $\cal P$ of Lebesgue continuous measures instead, we ask if there is a corresponding universal probability density function (pdf) that does not underestimate the value of…

用于最小比特率LQG控制的在线变长信源编码

Online variable-length source coding for minimum bitrate LQG control

提出结合量化器设计与通用无损信源编码的自适应方法，实现近最小比特率无前缀反馈的LQG控制，证明量化器输出属于指数或幂律包络类，并开发实用零延迟定点算术算法，数值评估显示其在比特率与控制性能权衡上具竞争力。

整数通用编码的近最优构造

The Construction of Near-optimal Universal Coding of Integers

本文提出一种近最优通用整数编码方案，通过改进的概率不等式证明了其最小扩展因子为2.0386，将最优通用编码的最小扩展因子范围缩小至2到2.0386之间。