This paper presents prefix codes which minimize various criteria constructed as a convex combination of maximum codeword length and average codeword length or maximum redundancy and average redundancy, including a convex combination of the average of an exponential function of the codeword length and the average redundancy. This framework encompasses as a special case several criteria previously investigated in the literature, while relations to universal coding is discussed. The coding algorithm derived is parametric resulting in re-adjusting the initial source probabilities via a weighted probability vector according to a merging rule. The level of desirable merging has implication in applications where the maximum codeword length is bounded.
We address the recently suggested problem of causal lossless coding of a randomly arriving source samples. We construct variable-to-fixed coding schemes and show that they outperform the previously considered fixed-to-variable schemes when traffic is high both in terms of delay and Age of Information by appealing to tools from queueing theory. We supplement our theoretical bounds with numerical simulations.
Motivated from the fact that universal source coding on countably infinite alphabets is not feasible, this work introduces the notion of almost lossless source coding. Analog to the weak variable-length source coding problem studied by Han (IEEE TIT, 2000, 46, 1217-1226), almost lossless source coding aims at relaxing the lossless block-wise assumption to allow an average per-letter distortion that vanishes asymptotically as the block-length tends to infinity. In this setup, we show on one hand that Shannon entropy characterizes the minimum achievable rate (similarly to the case of finite alph…
This work studies point-to-point, multiple access, and random access lossless source coding in the finite-blocklength regime. In each scenario, a random coding technique is developed and used to analyze third-order coding performance. Asymptotic results include a third-order characterization of the Slepian-Wolf rate region with an improved converse that relies on a connection to composite hypothesis testing. For dependent sources, the result implies that the independent encoders used by Slepian-Wolf codes can achieve the same third-order-optimal performance as a single joint encoder. The conce…
Given a private source of information, $X^n$ and a public correlated source, $Y^n$, we study the problem of encoding the two-dimensional source $(X^n, Y^n)$ into an index $J$ such that a remote party, knowing $J$ and some external side information $Z^n$, can losslessly recover $Y^n$ while any eavesdropper knowing $J$ and possibly a correlated side information $E^n$ can retrieve very little information about $X^n$. We give general converse results for the amount of information about $X^n$ that might be leaked in such systems and and also achievability results that are optimal in some special ca…
Motivated by the question of whether the recently introduced Reduced Cutset Coding (RCC) offers rate-complexity performance benefits over conventional context-based conditional coding for sources with two-dimensional Markov structure, this paper compares several row-centric coding strategies that vary in the amount of conditioning as well as whether a model or an empirical table is used in the encoding of blocks of rows. The conclusion is that, at least for sources exhibiting low-order correlations, 1-sided model-based conditional coding is superior to the method of RCC for a given constraint …
We consider a real-time streaming source coding system in which an encoder observes a sequence of randomly arriving symbols from an i.i.d. source, and feeds binary codewords to a FIFO buffer that outputs one bit per time unit to a decoder. Each source symbol represents a status update by the source, and the timeliness of the system is quantified by the age of information (AoI), defined as the time difference between the present time and the generation time of the most up-to-date symbol at the output of the decoder. When the FIFO buffer is allowed to be empty, we propose an optimal prefix-free …
Tight bounds for several symmetric divergence measures are derived in terms of the total variation distance. It is shown that each of these bounds is attained by a pair of 2 or 3-element probability distributions. An application of these bounds for lossless source coding is provided, refining and improving a certain bound by Csiszár. Another application of these bounds has been recently introduced by Yardi. et al. for channel-code detection.
Depth coding in 3D-HEVC for the multiview video plus depth (MVD) architecture (i) deforms object shapes due to block-level edge-approximation; (ii) misses an opportunity for high compressibility at near-lossless quality by failing to exploit strong homogeneity (clustering tendency) in depth syntax, motion vector components, and residuals at frame-level; and (iii) restricts interactivity and limits responsiveness of independent use of depth information for "non-viewing" applications due to texture-depth coding dependency. This paper presents a standalone depth sequence coder, which operates in …
We examine lossless data compression from an average delay perspective. An encoder receives input symbols one per unit time from an i.i.d. source and submits binary codewords to a FIFO buffer that transmits bits at a fixed rate to a receiver/decoder. Each input symbol at the encoder is viewed as a status update by the source and the system performance is characterized by the status update age, defined as the number of time units (symbols) the decoder output lags behind the encoder input. An upper bound on the average status age is derived from the exponential bound on the probability of error …
Data compression is an efficient technique to save data storage and transmission costs. However, traditional data compression methods always ignore the impact of user preferences on the statistical distributions of symbols transmitted over the links. Notice that the development of big data technologies and popularization of smart devices enable analyses on user preferences based on data collected from personal handsets. This paper presents a user preference aware lossless data compression method, termed edge source coding, to compress data at the network edge. An optimization problem is formul…
This paper provides upper and lower bounds on the optimal guessing moments of a random variable taking values on a finite set when side information may be available. These moments quantify the number of guesses required for correctly identifying the unknown object and, similarly to Arikan's bounds, they are expressed in terms of the Arimoto-Rényi conditional entropy. Although Arikan's bounds are asymptotically tight, the improvement of the bounds in this paper is significant in the non-asymptotic regime. Relationships between moments of the optimal guessing function and the MAP error probabili…
University College London（伦敦大学学院）；University of Edinburgh（爱丁堡大学）
We generalize the 'bits back with ANS' method to time-series models with a latent Markov structure. This family of models includes hidden Markov models (HMMs), linear Gaussian state space models (LGSSMs) and many more. We provide experimental evidence that our method is effective for small scale models, and discuss its applicability to larger scale settings such as video compression.
The bits-back argument suggests that latent variable models can be turned into lossless compression schemes. Translating the bits-back argument into efficient and practical lossless compression schemes for general latent variable models, however, is still an open problem. Bits-Back with Asymmetric Numeral Systems (BB-ANS), recently proposed by Townsend et al. (2019), makes bits-back coding practically feasible for latent variable models with one latent layer, but it is inefficient for hierarchical latent variable models. In this paper we propose Bit-Swap, a new compression scheme that generali…
University of Toronto（多伦多大学）；Vector Institute（向量研究所）；Facebook AI Research（脸书人工智能研究院）；University College London（伦敦大学学院）；University of Oxford（牛津大学）
Latent variable models have been successfully applied in lossless compression with the bits-back coding algorithm. However, bits-back suffers from an increase in the bitrate equal to the KL divergence between the approximate posterior and the true posterior. In this paper, we show how to remove this gap asymptotically by deriving bits-back coding algorithms from tighter variational bounds. The key idea is to exploit extended space representations of Monte Carlo estimators of the marginal likelihood. Naively applied, our schemes would require more initial bits than the standard bits-back coder,…
Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）；Tsinghua University（清华大学）
Soft compression is a lossless image compression method, which is committed to eliminating coding redundancy and spatial redundancy at the same time by adopting locations and shapes of codebook to encode an image from the perspective of information theory and statistical distribution. In this paper, we propose a new concept, compressible indicator function with regard to image, which gives a threshold about the average number of bits required to represent a location and can be used for revealing the performance of soft compression. We investigate and analyze soft compression for binary image, …
Adapting Diffusion Language Models for Lossless Pixel-Level Image Transmission
College of Information Science and Electronic Engineering, Zhejiang University（浙江大学信息科学与电子工程学院）；Shenzhen CyberAray Network Technology Co., Ltd（深圳CyberAray网络技术有限公司）；School of Mechanical Engineering and Electronic Information, China University of Geosciences（中国地质大学（武汉）机械与电子信息学院）；Zhejiang Lab（浙江实验室）
Deep latent variable models have seen recent success in many data domains. Lossless compression is an application of these models which, despite having the potential to be highly useful, has yet to be implemented in a practical manner. We present `Bits Back with ANS' (BB-ANS), a scheme to perform lossless compression with latent variable models at a near optimal rate. We demonstrate this scheme by using it to compress the MNIST dataset with a variational auto-encoder model (VAE), achieving compression rates superior to standard methods with only a simple VAE. Given that the scheme is highly am…
This paper asks a basic question: how much training is required to beat a universal source coder? Traditionally, there have been two types of source coders: fixed, optimum coders such as Huffman coders; and universal source coders, such as Lempel-Ziv The paper considers a third type of source coders: learned coders. These are coders that are trained on data of a particular type, and then used to encode new data of that type. This is a type of coder that has recently become very popular for (lossy) image and video coding. The paper consider two criteria for performance of learned coders: the av…
We apply so-called tree straight-line programs to the problem of lossless compression of binary trees. We derive upper bound on the maximal pointwise redundancy (or worst-case redundancy) that improve previous bounds obtained by Zhang, Yang, and Kieffer using directed acyclic graphs. Using this, we obtain universal codes for new classes of structered tree sources.
Consider a sequence $X^n$ of length $n$ emitted by a Discrete Memoryless Source (DMS) with unknown distribution $p_X$. The objective is to construct a lossless source code that maps $X^n$ to a sequence $\widehat{Y}^m$ of length $m$ that is indistinguishable, in terms of Kullback-Leibler divergence, from a sequence emitted by another DMS with known distribution $p_Y$. The main result is the existence of a coding scheme that performs this task with an optimal ratio $m/n$ equal to $H(X)/H(Y)$, the ratio of the Shannon entropies of the two distributions, as $n$ goes to infinity. The coding scheme …
A technique of lossless compression via substring enumeration (CSE) attains compression ratios as well as popular lossless compressors for one-dimensional (1D) sources. The CSE utilizes a probabilistic model built from the circular string of an input source for encoding the source.The CSE is applicable to two-dimensional (2D) sources such as images by dealing with a line of pixels of 2D source as a symbol of an extended alphabet. At the initial step of the CSE encoding process, we need to output the number of occurrences of all symbols of the extended alphabet, so that the time complexity incr…
This paper studies an almost-lossless source-channel coding scheme in which source messages are assigned to different classes and encoded with a channel code that depends on the class index. The code performance is analyzed by means of random-coding error exponents and validated by simulation of a low-complexity implementation using existing source and channel codes. While each class code can be seen as a concatenation of a source code and a channel code, the overall performance improves on that of separate source-channel coding and approaches that of joint source-channel coding when the numbe…
The problem of variable length and fixed-distortion universal source coding (or D-semifaithful source coding) for stationary and memoryless sources on countably infinite alphabets ($\infty$-alphabets) is addressed in this paper. The main results of this work offer a set of sufficient conditions (from weaker to stronger) to obtain weak minimax universality, strong minimax universality, and corresponding achievable rates of convergences for the worse-case redundancy for the family of stationary memoryless sources whose densities are dominated by an envelope function (or the envelope family) on $…
Polar coding is a recently proposed coding technique that can provably achieve the channel capacity. The polar code structure, which is based on the original 2x2 generator matrix, polarises the channels, i.e., a portion of the channel capacities approach 1, while the remaining channel capacities approach 0. Due to the specific size of this original generator matrix, polar codes can only have code lengths equal to the powers of 2, resulting in inefficiency for codes of practical lengths. In this paper, the performance of finite-length polar codes over the binary erasure channel is analysed. A n…
In information theory, lossless compression of general data is based on an explicit assumption of a stochastic generative model on target data. However, in lossless image compression, the researchers have mainly focused on the coding procedure that outputs the coded sequence from the input image, and the assumption of the stochastic generative model is implicit. In these studies, there is a difficulty in confirming the information-theoretical optimality of the coding procedure to the stochastic generative model. Hence, in this paper, we propose a novel stochastic generative model of images by …
In this work we consider discrete-time multiple-input multiple-output (MIMO) linear-quadratic-Gaussian (LQG) control where the feedback consists of variable length binary codewords. To simplify the decoder architecture, we enforce a strict prefix constraint on the codewords. We develop a data compression architecture that provably achieves a near minimum time-average expected bitrate for a fixed constraint on the LQG performance. The architecture conforms to the strict prefix constraint and does not require time-varying lossless source coding, in contrast to the prior art.
