Efficient optimal prefix coding has long been accomplished via the Huffman algorithm. However, there is still room for improvement and exploration regarding variants of the Huffman problem. Length-limited Huffman coding, useful for many practical applications, is one such variant, in which codes are restricted to the set of codes in which none of the $n$ codewords is longer than a given length, $l_{\max}$. Binary length-limited coding can be done in $O(n l_{\max})$ time and O(n) space via the widely used Package-Merge algorithm. In this paper the Package-Merge approach is generalized without increasing complexity in order to introduce a minimum codeword length, $l_{\min}$, to allow for objective functions other than the minimization of expected codeword length, and to be applicable to both binary and nonbinary codes; nonbinary codes were previously addressed using a slower dynamic programming approach. These extensions have various applications -- including faster decompression -- and can be used to solve the problem of finding an optimal code with limited fringe, that is, finding the best code among codes with a maximum difference between the longest and shortest codewords. The previously proposed method for solving this problem was nonpolynomial time, whereas solving this using the novel algorithm requires only $O(n (l_{\max}- l_{\min})^2)$ time and O(n) space.
In this paper we implement an n-ary Huffman Encoding and Decoding application using different degrees of tree structures. Our goal is to compare the performance of the algorithm in terms of compression ratio, decompression speed and weighted path length when using higher degree trees, compared to the 2-ary Huffman Code. The Huffman tree degrees that we compare are 2-ary, 3-ary, 4-ary, 5-ary, 6-ary, 7-ary, 8-ary and 16-mal. We also present the impact that branch prediction has on the performance of the n-ary Huffman Decoding.
Algorithms for deriving Huffman codes and the recently developed algorithm for compiling PIFO trees to trees of fixed shape (Mohan et al. 2022) are similar, but work with different underlying algebraic operations. In this paper, we exploit the monadic structure of prefix codes to create a generalized Huffman algorithm that has these two applications as special cases.
This paper proposes a novel model of the two-level scalar quantizer with extended Huffman coding. It is designed for the average bit rate to approach the source entropy as close as possible provided that the signal to quantization noise ratio (SQNR) value does not decrease more than 1 dB from the optimal SQNR value. Assuming the asymmetry of representation levels for the symmetric Laplacian probability density function, the unequal probabilities of representation levels are obtained, i.e. the proper basis for further implementation of lossless compression techniques is provided. In this paper,…
Huffman coding is well known to be useful in certain decision problems involving minimizing the average number of (freely chosen) queries to determine an unknown random variable. However, in problems where the queries are more constrained, the original Huffman coding no longer works. In this paper, we proposed a general model to describe such problems and two code schemes: one is Huffman-based, and the other called GBSC (Greedy Binary Separation Coding). We proved the optimality of GBSC by induction on a binary decision tree, telling us that GBSC is at least as good as Shannon coding. We then …
In this paper, we study the problem of designing prefix-free encoding schemes having minimum average code length that can be decoded efficiently under a decode cost model that captures memory hierarchy induced cost functions. We also study a special case of this problem that is closely related to the length limited Huffman coding (LLHC) problem; we call this the {\em soft-length limited Huffman coding} problem. In this version, there is a penalty associated with each of the $n$ characters of the alphabet whose encodings exceed a specified bound $D$($\leq n$), where the penalty increases linear…
The ``state-of-the-art'' in Length Limited Huffman Coding algorithms is the $Θ(ND)$-time, $Θ(N)$-space one of Hirschberg and Larmore, where $D\le N$ is the length restriction on the code. This is a very clever, very problem specific, technique. In this note we show that there is a simple Dynamic-Programming (DP) method that solves the problem with the same time and space bounds. The fact that there was an $Θ(ND)$ time DP algorithm was previously known; it is a straightforward DP with the Monge property (which permits an order of magnitude speedup). It was not interesting, though, because it al…
Data compression has become a necessity not only the in the field of communication but also in various scientific experiments. The data that is being received is more and the processing time required has also become more. A significant change in the algorithms will help to optimize the processing speed. With the invention of Technologies like IoT and in technologies like Machine Learning there is a need to compress data. For example training an Artificial Neural Network requires a lot of data that should be processed and trained in small interval of time for which compression will be very help…
Huffman Compression, also known as Huffman Coding, is one of many compression techniques in use today. The two important features of Huffman coding are instantaneousness that is the codes can be interpreted as soon as they are received and variable length that is a most frequent symbol has length smaller than a less frequent symbol. The traditional Huffman coding has two procedures: constructing a tree in O(n^2) and then traversing it in O(n). Quantum computing is a promising approach of computation that is based on equations from Quantum Mechanics. Instantaneousness and variable length featur…
Compression is a technique to reduce the quantity of data without excessively reducing the quality of the multimedia data. The transition and storing of compressed multimedia data is much faster and more efficient than original uncompressed multimedia data. There are various techniques and standards for multimedia data compression, especially for image compression such as the JPEG and JPEG2000 standards. These standards consist of different functions such as color space conversion and entropy coding. Arithmetic and Huffman coding are normally used in the entropy coding phase. In this paper we …
Huffman coding is a widely used method for lossless data compression because it optimally stores data based on how often the characters occur in Huffman trees. An $n$-ary Huffman tree is a connected, cycle-lacking graph where each vertex can have either $n$ "children" vertices connecting to it, or 0 children. Vertices with 0 children are called \textit{leaves}. We let $h_n(q)$ represent the total number of $n$-ary Huffman trees with $q$ leaves. In this paper, we use a recursive method to generate upper and lower bounds on $h_n(q)$ and get $h_2(q) \approx (0.1418532)(1.7941471)^q+(0.0612410)(1.…
In this paper we study binary trees with choosable edge lengths, in particular rooted binary trees with the property that the two edges leading from every non-leaf to its two children are assigned integral lengths $l_1$ and $l_2$ with $l_1+l_2 =k$ for a constant $k\in\mathbb{N}$. The depth of a leaf is the total length of the edges of the unique root-leaf-path. We present a generalization of the Huffman Coding that can decide in polynomial time if for given values $d_1,...,d_n\in\mathbb{N}_{\geq 0}$ there exists a rooted binary tree with choosable edge lengths with $n$ leaves having depths at …
In this work, a coding technique called cost constrained Geometric Huffman coding (ccGhc) is developed. ccGhc minimizes the Kullback-Leibler distance between a dyadic probability mass function (pmf) and a target pmf subject to an affine inequality constraint. An analytical proof is given that when ccGhc is applied to blocks of symbols, the optimum is asymptotically achieved when the blocklength goes to infinity. The derivation of ccGhc is motivated by the problem of encoding a text to a sequence of slats subject to architectural design criteria. For the considered architectural problem, for a …
Can one choose a good Huffman code on the fly, without knowing the underlying distribution? Online Slot Allocation (OSA) models this and similar problems: There are n slots, each with a known cost. There are n items. Requests for items are drawn i.i.d. from a fixed but hidden probability distribution p. After each request, if the item, i, was not previously requested, then the algorithm (knowing the slot costs and the requests so far, but not p) must place the item in some vacant slot j(i). The goal is to minimize the sum, over the items, of the probability of the item times the cost of its as…
Multiple input multiple output (MIMO) precoding is an efficient scheme that may significantly enhance the communication link. However, this enhancement comes with a cost. Many precoding schemes require channel knowledge at the transmitter that is obtained through feedback from the receiver. Focusing on the natural common fusion of orthogonal frequency division multiplexing (OFDM) and MIMO, we exploit the channel correlation in the frequency and spatial domain to reduce the required feedback rate in a frequency division duplex (FDD) system. The proposed feedback method is based on Huffman codin…
Efficient optimal prefix coding has long been accomplished via the Huffman algorithm. However, there is still room for improvement and exploration regarding variants of the Huffman problem. Length-limited Huffman coding, useful for many practical applications, is one such variant, for which codes are restricted to the set of codes in which none of the $n$ codewords is longer than a given length, $l_{\max}$. Binary length-limited coding can be done in $O(n l_{\max})$ time and O(n) space via the widely used Package-Merge algorithm and with even smaller asymptotic complexity using a lesser-known …
The modern data compression is mainly based on two approaches to entropy coding: Huffman (HC) and arithmetic/range coding (AC). The former is much faster, but approximates probabilities with powers of 2, usually leading to relatively low compression rates. The latter uses nearly exact probabilities - easily approaching theoretical compression rate limit (Shannon entropy), but at cost of much larger computational cost. Asymmetric numeral systems (ANS) is a new approach to accurate entropy coding, which allows to end this trade-off between speed and rate: the recent implementation [1] provides a…
Graphs have been extensively used to represent data from various domains. In the era of Big Data, information is being generated at a fast pace, and analyzing the same is a challenge. Various methods have been proposed to speed up the analysis of the data and also mining it for information. All of this often involves using a massive array of compute nodes, and transmitting the data over the network. Of course, with the huge quantity of data, this poses a major issue to the task of gathering intelligence from data. Therefore, in order to address such issues with Big Data, using data compression…
The design of the channel part of a digital communication system (e.g., error correction, modulation) is heavily based on the assumption that the data to be transmitted forms a fair bit stream. However, simple source encoders such as short Huffman codes generate bit streams that poorly match this assumption. As a result, the channel input distribution does not match the original design criteria. In this work, a simple method called half Huffman coding (halfHc) is developed. halfHc transforms a Huffman code into a source code whose output is more similar to a fair bit stream. This is achieved b…
Given a probability distribution over a set of n words to be transmitted, the Huffman Coding problem is to find a minimal-cost prefix free code for transmitting those words. The basic Huffman coding problem can be solved in O(n log n) time but variations are more difficult. One of the standard techniques for solving these variations utilizes a top-down dynamic programming approach. In this paper we show that this approach is amenable to dynamic programming speedup techniques, permitting a speedup of an order of magnitude for many algorithms in the literature for such variations as mixed radix,…
Canonical Huffman code is an optimal prefix-free compression code whose codewords enumerated in the lexicographical order form a list of binary words in non-decreasing lengths. Gagie et al. (2015) gave a representation of this coding capable to encode or decode a symbol in constant worst case time. It uses $σ\lg \ell_{\text{max}} + o(σ) + O(\ell_{\text{max}}^2)$ bits of space, where $σ$ and $\ell_{\text{max}}$ are the alphabet size and maximum codeword length, respectively. We refine their representation to reduce the space complexity to $σ\lg \ell_{\text{max}} (1 + o(1))$ bits while preservin…
One of the biggest criticisms of the Set Shaping Theory is the lack of a practical application. This is due to the difficulty of its application. In fact, to apply this technique from an experimental point of view we must use a table that defines the correspondences between two sets. However, this approach is not usable in practice, because the table has A^N elements, with A number of symbols and N length of the message to be encoded. Consequently, these tables can be implemented in a program only when A and N have a low value. Unfortunately, in these cases, there are no compression algorithms…
This paper presents new lower and upper bounds for the compression rate of binary prefix codes optimized over memoryless sources according to various nonlinear codeword length objectives. Like the most well-known redundancy bounds for minimum average redundancy coding - Huffman coding - these are in terms of a form of entropy and/or the probability of an input symbol, often the most probable one. The bounds here, some of which are tight, improve on known bounds of the form L in [H,H+1), where H is some form of entropy in bits (or, in the case of redundancy objectives, 0) and L is the length ob…
A strategy for computing upper code-length limits of AC Huffman codes for an 8x8 block in JPEG Baseline coding is developed. The method is based on a geometric interpretation of the DCT, and the calculated limits are as close as 14% to the maximum code-lengths. The proposed strategy can be adapted to other transform coding methods, e.g., MPEG 2 and 4 video compressions, to calculate close upper code length limits for the respective processing blocks.
Zero-error single-channel source coding has been studied extensively over the past decades. Its natural multi-channel generalization is however not well investigated. While the special case with multiple symmetric-alphabet channels was studied a decade ago, codes in such setting have no advantage over single-channel codes in data compression, making them worthless in most applications. With essentially no development since the last decade, in this paper, we break the stalemate by showing that it is possible to beat single-channel source codes in terms of compression assuming asymmetric-alphabe…
Huffman coding finds a prefix code that minimizes mean codeword length for a given probability distribution over a finite number of items. Campbell generalized the Huffman problem to a family of problems in which the goal is to minimize not mean codeword length but rather a generalized mean known as a quasiarithmetic or quasilinear mean. Such generalized means have a number of diverse applications, including applications in queueing. Several quasiarithmetic-mean problems have novel simple redundancy bounds in terms of a generalized entropy. A related property involves the existence of optimal …
A framework with two scalar parameters is introduced for various problems of finding a prefix code minimizing a coding penalty function. The framework encompasses problems previously proposed by Huffman, Campbell, Nath, and Drmota and Szpankowski, shedding light on the relationships among these problems. In particular, Nath's range of problems can be seen as bridging the minimum average redundancy problem of Huffman with the minimum maximum pointwise redundancy problem of Drmota and Szpankowski. Using this framework, two linear-time Huffman-like algorithms are devised for the minimum maximum p…
Many data compressors regularly encode probability distributions for entropy coding - requiring minimal description length type of optimizations. Canonical prefix/Huffman coding usually just writes lengths of bit sequences, this way approximating probabilities with powers-of-2. Operating on more accurate probabilities usually allows for better compression ratios, and is possible e.g. using arithmetic coding and Asymmetric Numeral Systems family. Especially the multiplication-free tabled variant of the latter (tANS) builds automaton often replacing Huffman coding due to better compression at si…
This paper studies coding schemes for the $q$-ary symmetric channel based on binary low-density parity-check (LDPC) codes that work for any alphabet size $q=2^m$, $m\in\mathbb{N}$, thus complementing some recently proposed packet-based schemes requiring large $q$. First, theoretical optimality of a simple layered scheme is shown, then a practical coding scheme based on a simple modification of standard binary LDPC decoding is proposed. The decoder is derived from first principles and using a factor-graph representation of a front-end that maps $q$-ary symbols to groups of $m$ bits connected to…
Consider the set of source distributions within a fixed maximum relative entropy with respect to a given nominal distribution. Lossless source coding over this relative entropy ball can be approached in more than one way. A problem previously considered is finding a minimax average length source code. The minimizing players are the codeword lengths --- real numbers for arithmetic codes, integers for prefix codes --- while the maximizing players are the uncertain source distributions. Another traditional minimizing objective is the first one considered here, maximum (average) redundancy. This p…
This paper describes a new set of block source codes well suited for data compression. These codes are defined by sets of productions rules of the form a.l->b, where a in A represents a value from the source alphabet A and l, b are -small- sequences of bits. These codes naturally encompass other Variable Length Codes (VLCs) such as Huffman codes. It is shown that these codes may have a similar or even a shorter mean description length than Huffman codes for the same encoding and decoding complexity. A first code design method allowing to preserve the lexicographic order in the bit domain is de…
It has been shown that an extension of the basic binary polar transformation also polarizes over finite fields. With it the direct encoding of q-ary sources and channels is a process that can be implemented with simple and efficient algorithms. However, direct polar decoding of q-ary sources and channels is more involved. In this paper we obtain a recursive equation for the likelihood ratio expressed as a LR vector. With it successive cancellation (SC) decoding is applied in a straightforward way. The complexity is quadratic in the order of the field, but the use of the LR vector introduces fa…
We present a new algorithm for dynamic prefix-free coding, based on Shannon coding. We give a simple analysis and prove a better upper bound on the length of the encoding produced than the corresponding bound for dynamic Huffman coding. We show how our algorithm can be modified for efficient length-restricted coding, alphabetic coding and coding with unequal letter costs.
We present a constraint-coding scheme to correct asymmetric magnitude-$1$ errors in multi-level non-volatile memories. For large numbers of such errors, the scheme is shown to deliver better correction capability compared to known alternatives, while admitting low-complexity of decoding. Our results include an algebraic formulation of the constraint, necessary and sufficient conditions for correctability, a maximum-likelihood decoder running in complexity linear in the alphabet size, and upper bounds on the probability of failing to correct $t$ errors. Besides the superior rate-correction trad…
Huffman coding finds an optimal prefix code for a given probability mass function. Consider situations in which one wishes to find an optimal code with the restriction that all codewords have lengths that lie in a user-specified set of lengths (or, equivalently, no codewords have lengths that lie in a complementary set). This paper introduces a polynomial-time dynamic programming algorithm that finds optimal codes for this reserved-length prefix coding problem. This has applications to quickly encoding and decoding lossless codes. In addition, one modification of the approach solves any quasia…
Huffman coding is often presented as the optimal solution to Twenty Questions. However, a caveat is that Twenty Questions games always end with a reply of "Yes," whereas Huffman codewords need not obey this constraint. We bring resolution to this issue, and prove that the average number of questions still lies between H(X) and H(X)+1.
In this paper, we study amplitude shaping schemes for the probabilistic amplitude shaping (PAS) framework as well as algorithms for constant-composition distribution matching (CCDM). Huffman-coded sphere shaping (HCSS) is discussed in detail, which internally uses Huffman coding to determine the composition to be used and relies on conventional CCDM algorithms for mapping and demapping. Numerical simulations show that HCSS closes the performance gap between distribution matching schemes and sphere shaping techniques such as enumerative sphere shaping (ESS). HCSS is based on an architecture tha…
This report presents the results of applying different compression algorithms to the network protocol of an online game. The algorithm implementations compared are zlib, liblzma and my own implementation based on LZ77 and a variation of adaptive Huffman coding. The comparison data was collected from the game TomeNET. The results show that adaptive coding is especially useful for compressing large amounts of very small packets.
We investigate two source coding problems with secrecy constraints. In the first problem we consider real--time fully secure transmission of a memoryless source. We show that although classical variable--rate coding is not an option since the lengths of the codewords leak information on the source, the key rate can be as low as the average Huffman codeword length of the source. In the second problem we consider causal source coding with a fidelity criterion and side information at the decoder and the eavesdropper. We show that when the eavesdropper has degraded side information, it is optimal …
We propose almost instantaneous fixed-to-variable-length (AIFV) codes such that two (resp. $K-1$) code trees are used if code symbols are binary (resp. $K$-ary for $K \geq 3$), and source symbols are assigned to incomplete internal nodes in addition to leaves. Although the AIFV codes are not instantaneous codes, they are devised such that the decoding delay is at most two bits (resp. one code symbol) in the case of binary (resp. $K$-ary) code alphabet. The AIFV code can attain better average compression rate than the Huffman code at the expenses of a little decoding delay and a little large me…
In this paper, we proposed a construction of a UD $k$-ary $T$-user coding scheme for MAAC. We first give a construction of $k$-ary $T^{f+g}$-user UD code from a $k$-ary $T^{f}$-user UD code and a $k^{\pm}$-ary $T^{g}$-user difference set with its two component sets $\mathcal{D}^{+}$ and $\mathcal{D}^{-}$ {\em a priori}. Based on the $k^{\pm}$-ary $T^{g}$-user difference set constructed from a $(2k-1)$-ary UD code, we recursively construct a UD $k$-ary $T$-user codes with code length of $2^m$ from initial multi-user codes of $k$-ary, $2(k-1)+1$-ary, \dots, $(2^m(k-1)+1)$-ary. Introducing multi-…
We extend earlier work on the design of convolutional code-specific CRC codes to $Q$-ary alphabets, with an eye toward $Q$-ary orthogonal signaling. Starting with distance-spectrum optimal, zero-terminated, $Q$-ary convolutional codes, we design $Q$-ary CRC codes so that the CRC/convolutional concatenation is distance-spectrum optimal. The $Q$-ary code symbols are mapped to a $Q$-ary orthogonal signal set and sent over an AWGN channel with noncoherent reception. We focus on $Q = 4$, rate-1/2 convolutional codes in our designs. The random coding union bound and normal approximation are used in …
Linear codes with a few weights are very important in coding theory and have attracted a lot of attention. In this paper, we present a construction of $q$-ary linear codes from trace and norm functions over finite fields. The weight distributions of the linear codes are determined in some cases based on Gauss sums. It is interesting that our construction can produce optimal or almost optimal codes. Furthermore, we show that our codes can be used to construct secret sharing schemes with interesting access structures and strongly regular graphs with new parameters.
The weight distribution and weight hierarchy of linear codes are two important research topics in coding theory. In this paper, by choosing proper defining sets from inhomogeneous quadratic functions over $\mathbb{F}_{q}^{2},$ we construct a family of $3$-weight $p$-ary linear codes and determine their weight distributions and weight hierarchies. Most of the codes can be used in secret sharing schemes.
Lattices have been used in several problems in coding theory and cryptography. In this paper we approach $q$-ary lattices obtained via Constructions D, $\D'$ and $\overline{D}$. It is shown connections between Constructions D and $\D'$. Bounds for the minimum $l_1$-distance of lattices $Λ_{D}$, $Λ_{D'}$ and $Λ_{\overline{D}}$ and, under certain conditions, a generator matrix for $Λ_{D'}$ are presented. In addition, when the chain of codes used is closed under the zero-one addition, we derive explicit expressions for the minimum $l_1$-distances of the lattices $Λ_{D}$ and $Λ_{\overline{D}}$ att…
A novel lossless source coding paradigm applies to problems of unreliable lossless channels with low bit rates, in which a vital message needs to be transmitted prior to termination of communications. This paradigm can be applied to Alfred Renyi's secondhand account of an ancient siege in which a spy was sent to scout the enemy but was captured. After escaping, the spy returned to his base in no condition to speak and unable to write. His commander asked him questions that he could answer by nodding or shaking his head, and the fortress was defended with this information. Renyi told this story…
A new coding scheme for image transmission over noisy channel is proposed. Similar to standard image compression, the scheme includes a linear transform followed by successive refinement scalar quantization. Unlike conventional schemes, in the proposed system the quantized transform coefficients are linearly mapped into channel symbols using systematic linear encoders. This fixed-to-fixed length "linear index coding" approach avoids the use of an explicit entropy coding stage (e.g., arithmetic or Huffman coding), which is typically fragile to channel post-decoding residual errors. We use linea…
Existing literature on source coding with side information (SCSI) mostly uses the state-of-the-art channel codes namely LDPC codes, turbo codes, and their variants and assume classical unique decoding. In this paper, we present an algebraic approach to SCSI based on the list decoding of the underlying channel codes. We show that the theoretical limit of SCSI can be achieved in the proposed list decoding based framework when the correlation between the source and side information is $q$-ary symmetric. We argue that, as opposed to channel coding, the correct sequence from the list produced by th…
This study focuses on efficient schemes for enumerative coding of $σ$--ary sequences by mainly borrowing ideas from Öktem & Astola's \cite{Oktem99} hierarchical enumerative coding and Schalkwijk's \cite{Schalkwijk72} asymptotically optimal combinatorial code on binary sequences. By observing that the number of distinct $σ$--dimensional vectors having an inner sum of $n$, where the values in each dimension are in range $[0...n]$ is $K(σ,n) = \sum_{i=0}^{σ-1} {{n-1} \choose {σ-1-i}} {σ \choose {i}}$, we propose representing $C$ vector via enumeration, and present necessary algorithms to perform …
Given a $q$-ary frequency hopping sequence set of length $n$ and size $M$ with Hamming correlation $H$, one can obtain a $q$-ary (nonlinear) cyclic code of length $n$ and size $nM$ with Hamming distance $n-H$. Thus, every upper bound on the size of a code from coding theory gives an upper bound on the size of a frequency hopping sequence set. Indeed, all upper bounds from coding theory have been converted to upper bounds on frequency hopping sequence sets (\cite{Ding09}). On the other hand, a lower bound from coding theory does not automatically produce a lower bound for frequency hopping sequ…
Text compression schemes and compact data structures usually combine sophisticated probability models with basic coding methods whose average codeword length closely match the entropy of known distributions. In the frequent case where basic coding represents run-lengths of outcomes that have probability $p$, i.e. the geometric distribution $\Pr(i)=p^i(1-p)$, a \emph{Golomb code} is an optimal instantaneous code, which has the additional advantage that codewords can be computed using only an integer parameter calculated from $p$, without need for a large or sophisticated data structure. Golomb …
A code over GF$(q^m)$ can be imaged or expanded into a code over GF$(q)$ using a basis for the extension field over the base field. The properties of such an image depend on the original code and the basis chosen for imaging. Problems relating the properties of a code and its image with respect to a basis have been of great interest in the field of coding theory. In this work, a generalized version of the problem of self-orthogonality of the $q$-ary image of a $q^m$-ary code has been considered. Given an inner product (more generally, a biadditive form), necessary and sufficient conditions hav…
Linear codes with a few weights are an important class of codes in coding theory and have attracted a lot of attention. In this paper, we present several constructions of $q$-ary linear codes with two or three weights from vectorial dual-bent functions, where $q$ is a power of an odd prime $p$. The weight distributions of the constructed $q$-ary linear codes are completely determined. We illustrate that some known constructions in the literature can be obtained by our constructions. In some special cases, our constructed linear codes can meet the Griesmer bound. Furthermore, based on the const…
