Soft compression is a lossless image compression method, which is committed to eliminating coding redundancy and spatial redundancy at the same time by adopting locations and shapes of codebook to encode an image from the perspective of information theory and statistical distribution. In this paper, we propose a new concept, compressible indicator function with regard to image, which gives a threshold about the average number of bits required to represent a location and can be used for revealing the performance of soft compression. We investigate and analyze soft compression for binary image, gray image and multi-component image by using specific algorithms and compressible indicator value. It is expected that the bandwidth and storage space needed when transmitting and storing the same kind of images can be greatly reduced by applying soft compression.
We address the recently suggested problem of causal lossless coding of a randomly arriving source samples. We construct variable-to-fixed coding schemes and show that they outperform the previously considered fixed-to-variable schemes when traffic is high both in terms of delay and Age of Information by appealing to tools from queueing theory. We supplement our theoretical bounds with numerical simulations.
An alternative approach to two-part 'critical compression' is presented. Whereas previous results were based on summing a lossless code at reduced precision with a lossy-compressed error or noise term, the present approach uses a similar lossless code at reduced precision to establish absolute bounds which constrain an arbitrary lossy data compression algorithm applied to the original data.
There is a special type of text which the order of the rows makes no difference (e.g., a word list). To compress these special texts, the traditional lossless compression method is not the ideal choice. A new method that can achieve better compression results for this type of texts is proposed. The texts are pre-processed by a method named SSE and are then compressed through the traditional lossless compression method. Comparison shows that an improved compression result is achieved.
A general lossless joint source-channel coding scheme based on linear codes is proposed and then analyzed in this paper. It is shown that a linear code with good joint spectrum can be used to establish limit-approaching joint source-channel coding schemes for arbitrary sources and channels, where the joint spectrum of the code is a generalization of the input-output weight distribution.
In this work, we establish an asymptotic limit of almost-lossless compression of a random, finite alphabet tensor which admits a low-rank canonical polyadic decomposition.
A new binary (bit-level) lossless compression catalyst method based on a modular arithmetic, called Binary Allocation via Modular Arithmetic (BAMA), has been introduced in this paper. In other words, BAMA is for storage and transmission of binary sequences, digital signal, images and video, also streaming and all kinds of digital transmission. As we said, our method does not compress, but facilitates the action of the real compressor, in our case, any lossless compression algorithm (Run Length Encoding, Lempel-Ziv-Welch, Huffman, Arithmetic, etc), that is, it acts as a compression catalyst. Fi…
This paper deals with a coding problem called complementary delivery, where messages from two correlated sources are jointly encoded and each decoder reproduces one of two messages using the other message as the side information. Both lossless and lossy universal complementary delivery coding schemes are investigated. In the lossless case, it is demonstrated that a universal complementary delivery code can be constructed by only combining two Slepian-Wolf codes. Especially, it is shown that a universal lossless complementary delivery code, for which error probability is exponentially tight, ca…
We develop a simple and elegant method for lossless compression using latent variable models, which we call 'bits back with asymmetric numeral systems' (BB-ANS). The method involves interleaving encode and decode steps, and achieves an optimal rate when compressing batches of data. We demonstrate it firstly on the MNIST test set, showing that state-of-the-art lossless compression is possible using a small variational autoencoder (VAE) model. We then make use of a novel empirical insight, that fully convolutional generative models, trained on small images, are able to generalize to images of ar…
The bits-back argument suggests that latent variable models can be turned into lossless compression schemes. Translating the bits-back argument into efficient and practical lossless compression schemes for general latent variable models, however, is still an open problem. Bits-Back with Asymmetric Numeral Systems (BB-ANS), recently proposed by Townsend et al. (2019), makes bits-back coding practically feasible for latent variable models with one latent layer, but it is inefficient for hierarchical latent variable models. In this paper we propose Bit-Swap, a new compression scheme that generali…
University College London（伦敦大学学院）；University of Edinburgh（爱丁堡大学）
We generalize the 'bits back with ANS' method to time-series models with a latent Markov structure. This family of models includes hidden Markov models (HMMs), linear Gaussian state space models (LGSSMs) and many more. We provide experimental evidence that our method is effective for small scale models, and discuss its applicability to larger scale settings such as video compression.
This article describes a technique of using a trigonometric function and combinatorial calculations to code or transform any finite sequence of binary numbers (0s and 1s) of any length to a unique set of three Real numbers. In reverse, these three Real numbers can be used independently to reconstruct the original Binary sequence precisely. The main principles of this technique are then applied in a proposal for a highly efficient model for Lossless Data Compression.
Tight bounds for several symmetric divergence measures are derived in terms of the total variation distance. It is shown that each of these bounds is attained by a pair of 2 or 3-element probability distributions. An application of these bounds for lossless source coding is provided, refining and improving a certain bound by Csiszár. Another application of these bounds has been recently introduced by Yardi. et al. for channel-code detection.
We propose an information-theoretic framework for phase retrieval. Specifically, we consider the problem of recovering an unknown n-dimensional vector x up to an overall sign factor from m=Rn phaseless measurements with compression rate R and derive a general achievability bound for R. Surprisingly, it turns out that this bound on the compression rate is the same as the one for almost lossless analog compression obtained by Wu and Verdú (2010): Phaseless linear measurements are as good as linear measurements with full phase information in the sense that ignoring the sign of m measurements only…
Generative model based image lossless compression algorithms have seen a great success in improving compression ratio. However, the throughput for most of them is less than 1 MB/s even with the most advanced AI accelerated chips, preventing them from most real-world applications, which often require 100 MB/s. In this paper, we propose PILC, an end-to-end image lossless compression framework that achieves 200 MB/s for both compression and decompression with a single NVIDIA Tesla V100 GPU, 10 times faster than the most efficient one before. To obtain this result, we first develop an AI codec tha…
Sparse representation has been widely used in data compression, signal and image denoising, dimensionality reduction and computer vision. While overcomplete dictionaries are required for sparse representation of multidimensional data, orthogonal bases represent one-dimensional data well. In this paper, we propose a data-driven sparse representation using orthonormal bases under the lossless compression constraint. We show that imposing such constraint under the Minimum Description Length (MDL) principle leads to a unique and optimal sparse representation for one-dimensional data, which results…
Motivated from the fact that universal source coding on countably infinite alphabets is not feasible, this work introduces the notion of almost lossless source coding. Analog to the weak variable-length source coding problem studied by Han (IEEE TIT, 2000, 46, 1217-1226), almost lossless source coding aims at relaxing the lossless block-wise assumption to allow an average per-letter distortion that vanishes asymptotically as the block-length tends to infinity. In this setup, we show on one hand that Shannon entropy characterizes the minimum achievable rate (similarly to the case of finite alph…
Deep latent variable models have seen recent success in many data domains. Lossless compression is an application of these models which, despite having the potential to be highly useful, has yet to be implemented in a practical manner. We present `Bits Back with ANS' (BB-ANS), a scheme to perform lossless compression with latent variable models at a near optimal rate. We demonstrate this scheme by using it to compress the MNIST dataset with a variational auto-encoder model (VAE), achieving compression rates superior to standard methods with only a simple VAE. Given that the scheme is highly am…
We consider the problem of lossless compression of binary trees, with the aim of reducing the number of code bits needed to store or transmit such trees. A lossless grammar-based code is presented which encodes each binary tree into a binary codeword in two steps. In the first step, the tree is transformed into a context-free grammar from which the tree can be reconstructed. In the second step, the context-free grammar is encoded into a binary codeword. The decoder of the grammar-based code decodes the original tree from its codeword by reversing the two encoding steps. It is shown that the re…
The order of letters is not always relevant in a communication task. This paper discusses the implications of order irrelevance on source coding, presenting results in several major branches of source coding theory: lossless coding, universal lossless coding, rate-distortion, high-rate quantization, and universal lossy coding. The main conclusions demonstrate that there is a significant rate savings when order is irrelevant. In particular, lossless coding of n letters from a finite alphabet requires Theta(log n) bits and universal lossless coding requires n + o(n) bits for many countable alpha…
This paper presents prefix codes which minimize various criteria constructed as a convex combination of maximum codeword length and average codeword length or maximum redundancy and average redundancy, including a convex combination of the average of an exponential function of the codeword length and the average redundancy. This framework encompasses as a special case several criteria previously investigated in the literature, while relations to universal coding is discussed. The coding algorithm derived is parametric resulting in re-adjusting the initial source probabilities via a weighted pr…
This work studies point-to-point, multiple access, and random access lossless source coding in the finite-blocklength regime. In each scenario, a random coding technique is developed and used to analyze third-order coding performance. Asymptotic results include a third-order characterization of the Slepian-Wolf rate region with an improved converse that relies on a connection to composite hypothesis testing. For dependent sources, the result implies that the independent encoders used by Slepian-Wolf codes can achieve the same third-order-optimal performance as a single joint encoder. The conce…
Institute of Computing Technology（计算技术研究所）；Chinese Academy of Science（中国科学院）；Peng Cheng Laboratory（鹏城实验室）；University of Waterloo（滑铁卢大学）；Dalian University of Technology（大连理工大学）；Central China Institute of Artificial Intelligence（中南中国人工智能研究所）；Ningbo Institute of Artificial Intelligence Industry（宁波人工智能产业研究院）
The problem of lossless fixed-rate streaming coding of discrete memoryless sources with side information at the decoder is studied. A random time-varying tree-code is used to sequentially bin strings and a Stack Algorithm with a variable bias uses the side information to give a delay-universal coding system for lossless source coding with side information. The scheme is shown to give exponentially decaying probability of error with delay, with exponent equal to Gallager's random coding exponent for sources with side information. The mean of the random variable of computation for the stack deco…
Given a private source of information, $X^n$ and a public correlated source, $Y^n$, we study the problem of encoding the two-dimensional source $(X^n, Y^n)$ into an index $J$ such that a remote party, knowing $J$ and some external side information $Z^n$, can losslessly recover $Y^n$ while any eavesdropper knowing $J$ and possibly a correlated side information $E^n$ can retrieve very little information about $X^n$. We give general converse results for the amount of information about $X^n$ that might be leaked in such systems and and also achievability results that are optimal in some special ca…
Motivated by the question of whether the recently introduced Reduced Cutset Coding (RCC) offers rate-complexity performance benefits over conventional context-based conditional coding for sources with two-dimensional Markov structure, this paper compares several row-centric coding strategies that vary in the amount of conditioning as well as whether a model or an empirical table is used in the encoding of blocks of rows. The conclusion is that, at least for sources exhibiting low-order correlations, 1-sided model-based conditional coding is superior to the method of RCC for a given constraint …
University of Toronto（多伦多大学）；Vector Institute（向量研究所）；Facebook AI Research（脸书人工智能研究院）；University College London（伦敦大学学院）；University of Oxford（牛津大学）
Latent variable models have been successfully applied in lossless compression with the bits-back coding algorithm. However, bits-back suffers from an increase in the bitrate equal to the KL divergence between the approximate posterior and the true posterior. In this paper, we show how to remove this gap asymptotically by deriving bits-back coding algorithms from tighter variational bounds. The key idea is to exploit extended space representations of Monte Carlo estimators of the marginal likelihood. Naively applied, our schemes would require more initial bits than the standard bits-back coder,…
This paper provides an extensive study of the behavior of the best achievable rate (and other related fundamental limits) in variable-length lossless compression. In the non-asymptotic regime, the fundamental limits of fixed-to-variable lossless compression with and without prefix constraints are shown to be tightly coupled. Several precise, quantitative bounds are derived, connecting the distribution of the optimal codelengths to the source information spectrum, and an exact analysis of the best achievable rate for arbitrary sources is given. Fine asymptotic results are proved for arbitrary (…
Lossless compression implementations typically contain two programs, an encoder and a decoder, which are required to be inverse to one another. We observe that a significant class of compression methods, based on asymmetric numeral systems (ANS), have shared structure between the encoder and decoder -- the decoder program is the 'reverse' of the encoder program -- allowing both to be simultaneously specified by a single, reversible function. To exploit this, we have implemented a small reversible language, embedded in Agda, which we call 'Flipper' (available at https://github.com/j-towns/flipp…
In this study, we propose a novel scheme for systematic improvement of lossless image compression coders from the point of view of the universal codes in information theory. In the proposed scheme, we describe a generative model class of images as a stochastic model. Using the Bayes codes, we are able to construct a lossless image compression coder which is optimal under the Bayes criterion for a model class described appropriately. Since the compression coder is optimal for the assumed model class, we are able to focus on the expansion of the model class. To validate the efficiency of the pro…
The penalty incurred by imposing a finite delay constraint in lossless source coding of a memoryless source is investigated. It is well known that for the so-called block-to-variable and variable-to-variable codes, the redundancy decays at best polynomially with the delay, where in this case the delay is identified with the source block length or maximal source phrase length, respectively. In stark contrast, it is shown that for sequential codes (e.g., a delay-limited arithmetic code) the redundancy can be made to decay exponentially with the delay constraint. The corresponding redundancy-dela…
Secure distributed data compression in the presence of an eavesdropper is explored. Two correlated sources that need to be reliably transmitted to a legitimate receiver are available at separate encoders. Noise-free, limited rate links from the encoders to the legitimate receiver, one of which can also be perfectly observed by the eavesdropper, are considered. The eavesdropper also has its own correlated observation. Inner and outer bounds on the achievable compression-equivocation rate region are given. Several different scenarios involving the side information at the transmitters as well as …
It was estimated that the world produced $59 ZB$ ($5.9 \times 10^{13} GB$) of data in 2020, resulting in the enormous costs of both data storage and transmission. Fortunately, recent advances in deep generative models have spearheaded a new class of so-called "neural compression" algorithms, which significantly outperform traditional codecs in terms of compression ratio. Unfortunately, the application of neural compression garners little commercial interest due to its limited bandwidth; therefore, developing highly efficient frameworks is of critical practical importance. In this paper, we dis…
The modern practice of Radio Astronomy is characterized by extremes of data volume and rates, principally because of the direct relationship between the signal to noise ratio that can be achieved and the need to Nyquist sample the RF bandwidth necessary by way of support. The transport of these data flows is costly. By examining the statistical nature of typical data flows and applying well known techniques from the field of Information Theory the following work shows that lossless compression of typical radio astronomy data flows is in theory possible. The key parameter in determining the deg…
This work proposes lossless and near-lossless compression algorithms for multi-channel biomedical signals. The algorithms are sequential and efficient, which makes them suitable for low-latency and low-power signal transmission applications. We make use of information theory and signal processing tools (such as universal coding, universal prediction, and fast online implementations of multivariate recursive least squares), combined with simple methods to exploit spatial as well as temporal redundancies typically present in biomedical signals. The algorithms are tested with publicly available e…
Various graphs such as web or social networks may contain up to trillions of edges. Compressing such datasets can accelerate graph processing by reducing the amount of I/O accesses and the pressure on the memory subsystem. Yet, selecting a proper compression method is challenging as there exist a plethora of techniques, algorithms, domains, and approaches in compressing graphs. To facilitate this, we present a survey and taxonomy on lossless graph compression that is the first, to the best of our knowledge, to exhaustively analyze this domain. Moreover, our survey does not only categorize exis…
This paper presents conditional versions of Lempel-Ziv (LZ) algorithm for settings where compressor and decompressor have access to the same side information. We propose a fixed-length-parsing LZ algorithm with side information, motivated by the Willems algorithm, and prove the optimality for any stationary processes. In addition, we suggest strategies to improve the algorithm which lower the data compression rate. A modification of a variable-length-parsing LZ algorithm with side information is proposed and proved to be asymptotically optimal for any stationary and ergodic processes.
Distributed source coding is traditionally viewed in the block coding context -- all the source symbols are known in advance at the encoders. This paper instead considers a streaming setting in which iid source symbol pairs are revealed to the separate encoders in real time and need to be reconstructed at the decoder with some tolerable end-to-end delay using finite rate noiseless channels. A sequential random binning argument is used to derive a lower bound on the error exponent with delay and show that both ML decoding and universal decoding achieve the same positive error exponents inside t…
Huffman compression is a statistical, lossless, data compression algorithm that compresses data by assigning variable length codes to symbols, with the more frequently appearing symbols given shorter codes than the less. This work is a modification of the Huffman algorithm which permits uncompressed data to be decomposed into indepen- dently compressible and decompressible blocks, allowing for concurrent compression and decompression on multiple processors. We create implementations of this modified algorithm on a current NVIDIA GPU using the CUDA API as well as on a current Intel chip and the…
As a probabilistic modeling technique, the flow-based model has demonstrated remarkable potential in the field of lossless compression \cite{idf,idf++,lbb,ivpf,iflow},. Compared with other deep generative models (eg. Autoregressive, VAEs) \cite{bitswap,hilloc,pixelcnn++,pixelsnail} that explicitly model the data distribution probabilities, flow-based models perform better due to their excellent probability density estimation and satisfactory inference speed. In flow-based models, multi-scale architecture provides a shortcut from the shallow layer to the output layer, which significantly reduce…
In this paper, the context dependence multilevel pattern matching(in short CDMPM) grammar transform is proposed; based on this grammar transform, the universal lossless data compression algorithm, CDMPM code is then developed. Moreover we get a upper bound of this algorithms' worst case redundancy among all individual sequences of length n from a finite alphabet.
We consider a real-time streaming source coding system in which an encoder observes a sequence of randomly arriving symbols from an i.i.d. source, and feeds binary codewords to a FIFO buffer that outputs one bit per time unit to a decoder. Each source symbol represents a status update by the source, and the timeliness of the system is quantified by the age of information (AoI), defined as the time difference between the present time and the generation time of the most up-to-date symbol at the output of the decoder. When the FIFO buffer is allowed to be empty, we propose an optimal prefix-free …
UCLA（加州大学洛杉矶分校）；University of California, Irvine（加州大学欧文分校）
Despite extensive progress on image generation, common deep generative model architectures are not easily applied to lossless compression. For example, VAEs suffer from a compression cost overhead due to their latent variables. This overhead can only be partially eliminated with elaborate schemes such as bits-back coding, often resulting in poor single-sample compression rates. To overcome such problems, we establish a new class of tractable lossless compression models that permit efficient encoding and decoding: Probabilistic Circuits (PCs). These are a class of neural networks involving $|p|…
Wu and Verdú developed a theory of almost lossless analog compression, where one imposes various regularity conditions on the compressor and the decompressor with the input signal being modelled by a (typically infinite-entropy) stationary stochastic process. In this work we consider all stationary stochastic processes with trajectories in a prescribed set $\mathcal{S} \subset [0,1]^\mathbb{Z}$ of (bi)infinite sequences and find uniform lower and upper bounds for certain compression rates in terms of metric mean dimension and mean box dimension. An essential tool is the recent Lindenstrauss-Ts…
A general lossless joint source-channel coding (JSCC) scheme based on linear codes and random interleavers for multiple-access channels (MACs) is presented and then analyzed in this paper. By the information-spectrum approach and the code-spectrum approach, it is shown that a linear code with a good joint spectrum can be used to establish limit-approaching lossless JSCC schemes for correlated general sources and general MACs, where the joint spectrum is a generalization of the input-output weight distribution. Some properties of linear codes with good joint spectra are investigated. A formula …
Depth coding in 3D-HEVC for the multiview video plus depth (MVD) architecture (i) deforms object shapes due to block-level edge-approximation; (ii) misses an opportunity for high compressibility at near-lossless quality by failing to exploit strong homogeneity (clustering tendency) in depth syntax, motion vector components, and residuals at frame-level; and (iii) restricts interactivity and limits responsiveness of independent use of depth information for "non-viewing" applications due to texture-depth coding dependency. This paper presents a standalone depth sequence coder, which operates in …
In this paper we consider lossless source coding for a class of sources specified by the total variational distance ball centred at a fixed nominal probability distribution. The objective is to find a minimax average length source code, where the minimizers are the codeword lengths -- real numbers for arithmetic or Shannon codes -- while the maximizers are the source distributions from the total variational distance ball. Firstly, we examine the maximization of the average codeword length by converting it into an equivalent optimization problem, and we give the optimal codeword lenghts via a w…
Many information sources are not just sequences of distinguishable symbols but rather have invariances governed by alternative counting paradigms such as permutations, combinations, and partitions. We consider an entire classification of these invariances called the twelvefold way in enumerative combinatorics and develop a method to characterize lossless compression limits. Explicit computations for all twelve settings are carried out for i.i.d. uniform and Bernoulli distributions. Comparisons among settings provide quantitative insight.
A binary string of length $2^k$ induces the Boolean function of $k$ variables whose Shannon expansion is the given binary string. This Boolean function then is representable via a unique reduced ordered binary decision diagram (ROBDD). The given binary string is fully recoverable from this ROBDD. We exhibit a lossless data compression algorithm in which a binary string of length a power of two is compressed via compression of the ROBDD associated to it as described above. We show that when binary strings of length $n$ a power of two are compressed via this algorithm, the maximal pointwise redu…
We examine lossless data compression from an average delay perspective. An encoder receives input symbols one per unit time from an i.i.d. source and submits binary codewords to a FIFO buffer that transmits bits at a fixed rate to a receiver/decoder. Each input symbol at the encoder is viewed as a status update by the source and the system performance is characterized by the status update age, defined as the number of time units (symbols) the decoder output lags behind the encoder input. An upper bound on the average status age is derived from the exponential bound on the probability of error …
