Hash Property and Fixed-rate Universal Coding Theorems
Jun Muramatsu, Shigeki Miyake
备注：
submitted to IEEE Transactions on Information Theory, Mar. 2008. 15 pages
发表信息：
IEEE Transactions on Information Theory, vol. 56, no. 6, pp. 2688-2698, June 2010. Corrections: IEEE Transactions on Information Theory, vol. 58, no. 5, pp. 3305-3307, May 2012
The aim of this paper is to prove the achievability of fixed-rate universal coding problems by using our previously introduced notion of hash property. These problems are the fixed-rate lossless universal source coding problem and the fixed-rate universal channel coding problem. Since an ensemble of sparse matrices satisfies the hash property requirement, it is proved that we can construct universal codes by using sparse matrices.
The aim of this paper is to prove the achievability of several coding problems by using sparse matrices (the maximum column weight grows logarithmically in the block length) and maximal-likelihood (ML) coding. These problems are the Slepian-Wolf problem, the Gel'fand-Pinsker problem, the Wyner-Ziv problem, and the One-helps-one problem (source coding with partial side information at the decoder). To this end, the notion of a hash property for an ensemble of functions is introduced and it is proved that an ensemble of $q$-ary sparse matrices satisfies the hash property. Based on this property, …
We consider wiretap channels with uncertainty on the eavesdropper channel under (i) noisy blockwise type II, (ii) compound, or (iii) arbitrarily varying models. We present explicit wiretap codes that can handle these models in a unified manner and only rely on three primitives, namely source coding with side information, universal hashing, and distribution approximation. Our explicit wiretap codes achieve the best known single-letter achievable rates, previously obtained non-constructively, for the models considered. Our results are obtained for strong secrecy, do not require a pre-shared secr…
The aim of this paper is to prove theorems for the Slepian-Wolf source coding and the broadcast channel coding (independent messages and no common message) based on the the notion of a stronger version of the hash property for an ensemble of functions. Since an ensemble of sparse matrices has a strong hash property, codes using sparse matrices can realize the achievable rate region. Furthermore, extensions to the multiple source coding and multiple output broadcast channel coding are investigated.
The information theoretic approach to security entails harnessing the correlated randomness available in nature to establish security. It uses tools from information theory and coding and yields provable security, even against an adversary with unbounded computational power. However, the feasibility of this approach in practice depends on the development of efficiently implementable schemes. In this article, we review a special class of practical schemes for information theoretic security that are based on 2-universal hash families. Specific cases of secret key agreement and wiretap coding are…
We furnish a procedure based on universal hash families (UHFs) that can convert an error correcting coding scheme (ECC) of rate $R$ into a semantically secure wiretap coding scheme of rate $R - ξ$ where $ξ$ is a parameter derived from the eavesdropper's point-to-point channel. This conversion is shown to be polynomial time efficient with block length and is applicable to any channel. When an ECC is chosen, our procedure induces a wiretap coding scheme that is concrete and efficient as long as the ECC is also such. To prove this induced wiretap coding scheme is semantically secure, we have cons…
Quantum stabilizer codes (QSCs) suffer from a low quantum coding rate, since they have to recover the quantum bits (qubits) in the face of both bit-flip and phase-flip errors. In this treatise, we conceive a low-complexity concatenated quantum turbo code (QTC) design exhibiting a high quantum coding rate. The high quantum coding rate is achieved by combining the quantum-domain version of short-block codes (SBCs) also known as single parity check (SPC) codes as the outer codes and quantum unity-rate codes (QURCs) as the inner codes. Despite its design simplicity, the proposed QTC yields a near-…
Multi-hop Deep Joint Source-Channel Coding with Deep Hash Distillation for Semantically Aligned Image Recovery
Department of Electrical and Electronic Engineering, Imperial College London, UK（电气与电子工程系，帝国理工学院伦敦分校，英国）；Lehrstuhl für Nachrichtentechnik, Technische Universität Dortmund, Germany（信息论系，德意志理工大学多特蒙德分校，德国）
The aim of this paper is to introduce the construction of codes for a general discrete stationary memoryless multiple access channel based on the the notion of the hash property. Since an ensemble of sparse matrices has a hash property, we can use sparse matrices for code construction. Our approach has a potential advantage compared to the conventional random coding because it is expected that we can use some approximation algorithms by using the sparse structure of codes.
We explicitly construct random hash functions for privacy amplification (extractors) that require smaller random seed lengths than the previous literature, and still allow efficient implementations with complexity $O(n\log n)$ for input length $n$. The key idea is the concept of dual universal$_2$ hash function introduced recently. We also use a new method for constructing extractors by concatenating $δ$-almost dual universal$_2$ hash functions with other extractors. Besides minimizing seed lengths, we also introduce methods that allow one to use non-uniform random seeds for extractors. These …
In this paper, we introduce the concept of dual universality of hash functions and present its applications to quantum cryptography. We begin by establishing the one-to-one correspondence between a linear function family {\cal F} and a code family {\cal C}, and thereby defining \varepsilon-almost dual universal_2 hash functions, as a generalization of the conventional universal_2 hash functions. Then we show that this generalized (and thus broader) class of hash functions is in fact sufficient for the security of quantum cryptography. This result can be explained in two different formalisms. F…
We show that every almost universal hash function also has the storage enforcement property. Almost universal hash functions have found numerous applications and we show that this new storage enforcement property allows the application of almost universal hash functions in a wide range of remote verification tasks: (i) Proof of Secure Erasure (where we want to remotely erase and securely update the code of a compromised machine with memory-bounded adversary), (ii) Proof of Ownership (where a storage server wants to check if a client has the data it claims to have before giving access to dedupl…
This paper proposes and proves security of a QKD protocol which uses two-universal hashing instead of random sampling to estimate the number of bit flip and phase flip errors. This protocol dramatically outperforms previous QKD protocols for small block sizes. More generally, for the two-universal hashing QKD protocol, the difference between asymptotic and finite key rate decreases with the number $n$ of qubits as $cn^{-1}$, where $c$ depends on the security parameter. For comparison, the same difference decreases no faster than $c'n^{-1/3}$ for an optimized protocol that uses random sampling …
This paper deals with a universal coding problem for a certain kind of multiterminal source coding system that we call the complementary delivery coding system. In this system, messages from two correlated sources are jointly encoded, and each decoder has access to one of the two messages to enable it to reproduce the other message. Both fixed-to-fixed length and fixed-to-variable length lossless coding schemes are considered. Explicit constructions of universal codes and bounds of the error probabilities are clarified via type-theoretical and graph-theoretical analyses. [[Keywords]] multiterm…
This work contains two main contributions concerning the asymmetric broadcast channel. The first is an analysis of the exact random coding error exponents for both users, and the second is the derivation of universal decoders for both users. These universal decoders are certain variants of the maximum mutual information (MMI) universal decoder, which achieve the corresponding random coding exponents of optimal decoding. In addition, we introduce some lower bounds, which involve optimization over very few parameters, unlike the original, exact exponents, which involve minimizations over auxilia…
This paper deals with a coding problem called complementary delivery, where messages from two correlated sources are jointly encoded and each decoder reproduces one of two messages using the other message as the side information. Both lossless and lossy universal complementary delivery coding schemes are investigated. In the lossless case, it is demonstrated that a universal complementary delivery code can be constructed by only combining two Slepian-Wolf codes. Especially, it is shown that a universal lossless complementary delivery code, for which error probability is exponentially tight, ca…
A multiterminal lossy coding problem, which includes various problems such as the Wyner-Ziv problem and the complementary delivery problem as special cases, is considered. It is shown that any point in the achievable rate-distortion region can be attained even if the source statistics are not known.
We apply so-called tree straight-line programs to the problem of lossless compression of binary trees. We derive upper bound on the maximal pointwise redundancy (or worst-case redundancy) that improve previous bounds obtained by Zhang, Yang, and Kieffer using directed acyclic graphs. Using this, we obtain universal codes for new classes of structered tree sources.
This paper deals with a universal coding problem for a certain kind of multiterminal source coding network called a generalized complementary delivery network. In this network, messages from multiple correlated sources are jointly encoded, and each decoder has access to some of the messages to enable it to reproduce the other messages. Both fixed-to-fixed length and fixed-to-variable length lossless coding schemes are considered. Explicit constructions of universal codes and the bounds of the error probabilities are clarified by using methods of types and graph-theoretical analysis.
We consider a setting of Slepian--Wolf coding, where the random bin of the source vector undergoes channel coding, and then decoded at the receiver, based on additional side information, correlated to the source. For a given distribution of the randomly selected channel codewords, we propose a universal decoder that depends on the statistics of neither the correlated sources nor the channel, assuming first that they are both memoryless. Exact analysis of the random-binning/random-coding error exponent of this universal decoder shows that it is the same as the one achieved by the optimal maximu…
Universal fixed-to-variable lossless source coding for memoryless sources is studied in the finite blocklength and higher-order asymptotics regimes. Optimal third-order coding rates are derived for general fixed-to-variable codes and for prefix codes. It is shown that the non-prefix Type Size code, in which codeword lengths are chosen in ascending order of type class size, achieves the optimal third-order rate and outperforms classical Two-Stage codes. Converse results are proved making use of a result on the distribution of the empirical entropy and Laplace's approximation. Finally, the fixed…
This paper deals with the problem of universal lossless coding on a countable infinite alphabet. It focuses on some classes of sources defined by an envelope condition on the marginal distribution, namely exponentially decreasing envelope classes with exponent $α$. The minimax redundancy of exponentially decreasing envelope classes is proved to be equivalent to $\frac{1}{4 α\log e} \log^2 n$. Then a coding strategy is proposed, with a Bayes redundancy equivalent to the maximin redundancy. At last, an adaptive algorithm is provided, whose redundancy is equivalent to the minimax redundancy
Separating hash families are useful combinatorial structures which are generalizations of many well-studied objects in combinatorics, cryptography and coding theory. In this paper, using tools from graph theory and additive number theory, we solve several open problems and conjectures concerning bounds and constructions for separating hash families. Firstly, we discover that the cardinality of a separating hash family satisfies a Johnson-type inequality. As a result, we obtain a new upper bound, which is superior to all previous ones. Secondly, we present a construction for an infinite class o…
The order of letters is not always relevant in a communication task. This paper discusses the implications of order irrelevance on source coding, presenting results in several major branches of source coding theory: lossless coding, universal lossless coding, rate-distortion, high-rate quantization, and universal lossy coding. The main conclusions demonstrate that there is a significant rate savings when order is irrelevant. In particular, lossless coding of n letters from a finite alphabet requires Theta(log n) bits and universal lossless coding requires n + o(n) bits for many countable alpha…
We define an algorithm that parses multidimensional arrays sequentially into mainly unrepeated but nested multidimensional sub-arrays of increasing size, and show that the resulting sub-block pointer encoder compresses almost every realization of any finite-alphabet ergodic process on $\mathbb{Z}_{\geq0}^d$ to the entropy, in the limit.
We derive universal classical-quantum superposition coding and universal classical-quantum multiple access channel code by using generalized packing lemmas for the type method. Using our classical-quantum universal superposition code, we establish the capacity region of a classical-quantum compound broadcast channel with degraded message sets. Our universal classical-quantum multiple access channel codes have two types of codes. One is a code with joint decoding and the other is a code with separate decoding. The former universally achieves corner points of the capacity region and the latter u…
Universal variable-to-fixed (V-F) length coding of $d$-dimensional exponential family of distributions is considered. We propose an achievable scheme consisting of a dictionary, used to parse the source output stream, making use of the previously-introduced notion of quantized types. The quantized type class of a sequence is based on partitioning the space of minimal sufficient statistics into cuboids. Our proposed dictionary consists of sequences in the boundaries of transition from low to high quantized type class size. We derive the asymptotics of the $ε$-coding rate of our coding scheme fo…
In this work, a new upper bound for average error probability of a two-user discrete memoryless (DM) multiple-access channel (MAC) is derived. This bound can be universally obtained for all discrete memoryless MACs with given input and output alphabets. This is the first bound of this type that explicitly uses the method of expurgation. It is shown that the exponent of this bound is greater than or equal to those of previously known bounds.
We present a novel lossless universal source coding algorithm that uses parallel computational units to increase the throughput. The length-$N$ input sequence is partitioned into $B$ blocks. Processing each block independently of the other blocks can accelerate the computation by a factor of $B$, but degrades the compression quality. Instead, our approach is to first estimate the minimum description length (MDL) source underlying the entire input, and then encode each of the $B$ blocks in parallel based on the MDL source. With this two-pass approach, the compression loss incurred by using more…
In this paper, we first propose an universal polar coding scheme for parallel Gaussian channels with non-binary inputs. It is assumed that the encoder knows only the sum capacity of M parallel channels instead of the capacity of any single channel. By decomposing each parallel channel into T = [log2r] sub channels, we therefore obtain MT binary sub-channels. A super polar coding scheme that across all sub-channels is then proposed. This scheme can achieve the sum capacity when the block length is sufficiently large. We have also discussed the applications of parallel polar coding design for bo…
We study the problem of universal decoding for unknown discrete memoryless channels in the presence of erasure/list option at the decoder, in the random coding regime. Specifically, we harness a universal version of Forney's classical erasure/list decoder developed in earlier studies, which is based on the competitive minimax methodology, and guarantees universal achievability of a certain fraction of the optimum random coding error exponents. In this paper, we derive an exact single-letter expression for the maximum achievable fraction. Examples are given in which the maximal achievable fract…
Motivated from the fact that universal source coding on countably infinite alphabets is not feasible, this work introduces the notion of almost lossless source coding. Analog to the weak variable-length source coding problem studied by Han (IEEE TIT, 2000, 46, 1217-1226), almost lossless source coding aims at relaxing the lossless block-wise assumption to allow an average per-letter distortion that vanishes asymptotically as the block-length tends to infinity. In this setup, we show on one hand that Shannon entropy characterizes the minimum achievable rate (similarly to the case of finite alph…
This paper investigates universal polar coding schemes. In particular, a notion of ordering (called convolutional path) is introduced between probability distributions to determine when a polar compression (or communication) scheme designed for one distribution can also succeed for another one. The original polar decoding algorithm is also generalized to an algorithm allowing to learn information about the source distribution using the idea of checkers. These tools are used to construct a universal compression algorithm for binary sources, operating at the lowest achievable rate (entropy), wit…
Silva et al. proposed a universal secure network coding scheme based on MRD codes, which can be applied to any underlying network code. This paper considers a stronger eavesdropping model where the eavesdroppers possess the ability to re-select the tapping links during the transmission. We give a proof for the impossibility of attaining universal security against such adversaries using Silva et al.'s code for all choices of code parameters, even with restricted number of tapped links. We also consider the cases with restricted tapping duration and derive some conditions for this code to be sec…
Universal coding of integers~(UCI) is a class of variable-length code, such that the ratio of the expected codeword length to $\max\{1,H(P)\}$ is within a constant factor, where $H(P)$ is the Shannon entropy of the decreasing probability distribution $P$. However, if we consider the ratio of the expected codeword length to $H(P)$, the ratio tends to infinity by using UCI, when $H(P)$ tends to zero. To solve this issue, this paper introduces a class of codes, termed generalized universal coding of integers~(GUCI), such that the ratio of the expected codeword length to $H(P)$ is within a constan…
We study the problem of reducing the communication overhead from a noisy wire-tap channel or storage system where data is encoded as a matrix, when more columns (or their linear combinations) are available. We present its applications to reducing communication overheads in universal secure linear network coding and secure distributed storage with crisscross errors and erasures and in the presence of a wire-tapper. Our main contribution is a method to transform coding schemes based on linear rank-metric codes, with certain properties, to schemes with lower communication overheads. By applying t…
This paper considers structure learning from data with $n$ samples of $p$ variables, assuming that the structure is a forest, using the Chow-Liu algorithm. Specifically, for incomplete data, we construct two model selection algorithms that complete in $O(p^2)$ steps: one obtains a forest with the maximum posterior probability given the data, and the other obtains a forest that converges to the true one as $n$ increases. We show that the two forests are generally different when some values are missing. Additionally, we present estimations for benchmark data sets to demonstrate that both algorit…
Beckman Institute（贝克曼研究所）；University of Illinois（伊利诺伊大学）
The problem of joint universal source coding and modeling, treated in the context of lossless codes by Rissanen, was recently generalized to fixed-rate lossy coding of finitely parametrized continuous-alphabet i.i.d. sources. We extend these results to variable-rate lossy block coding of stationary ergodic sources and show that, for bounded metric distortion measures, any finitely parametrized family of stationary sources satisfying suitable mixing, smoothness and Vapnik-Chervonenkis learnability conditions admits universal schemes for joint lossy source coding and identification. We also give…
A solution to the problem of asymptotically optimum perfect universal steganography of finite memoryless sources with a passive warden is provided, which is then extended to contemplate a distortion constraint. The solution rests on the fact that Slepian's Variant I permutation coding implements first-order perfect universal steganography of finite host signals with optimum embedding rate. The duality between perfect universal steganography with asymptotically optimum embedding rate and lossless universal source coding with asymptotically optimum compression rate is evinced in practice by show…
Many applications require data processing to be performed on individual pieces of data which are of finite sizes, e.g., files in cloud storage units and packets in data networks. However, traditional universal compression solutions would not perform well over the finite-length sequences. Recently, we proposed a framework called memory-assisted universal compression that holds a significant promise for reducing the amount of redundant data from the finite-length sequences. The proposed compression scheme is based on the observation that it is possible to learn source statistics (by memorizing p…
We consider nonparametric or universal sequential hypothesis testing problem when the distribution under the null hypothesis is fully known but the alternate hypothesis corresponds to some other unknown distribution. These algorithms are primarily motivated from spectrum sensing in Cognitive Radios and intruder detection in wireless sensor networks. We use easily implementable universal lossless source codes to propose simple algorithms for such a setup. The algorithms are first proposed for discrete alphabet. Their performance and asymptotic properties are studied theoretically. Later these a…
The problem of variable length and fixed-distortion universal source coding (or D-semifaithful source coding) for stationary and memoryless sources on countably infinite alphabets ($\infty$-alphabets) is addressed in this paper. The main results of this work offer a set of sufficient conditions (from weaker to stronger) to obtain weak minimax universality, strong minimax universality, and corresponding achievable rates of convergences for the worse-case redundancy for the family of stationary memoryless sources whose densities are dominated by an envelope function (or the envelope family) on $…
We consider a setup in which Alice selects a pdf $f$ from a set of prescribed pdfs $\mathscr{P}$ and sends a prefix-free codeword $W$ to Bob in order to allow him to generate a single instance of the random variable $X\sim f$. We describe a universal coding scheme for this setup and establish an upper bound on the expected codeword length when the pdf $f$ is bounded, orthogonally concave (which includes quasiconcave pdf), and has a finite first absolute moment. A dyadic decomposition scheme is used to express the pdf as a mixture of uniform pdfs over hypercubes. Alice randomly selects a hyperc…
In entropy coding, universal coding of integers~(UCI) is a binary universal prefix code, such that the ratio of the expected codeword length to $\max\{1, H(P)\}$ is less than or equal to a constant expansion factor $K_{\mathcal{C}}$ for any probability distribution $P$, where $H(P)$ is the Shannon entropy of $P$. $K_{\mathcal{C}}^{*}$ is the infimum of the set of expansion factors. The optimal UCI is defined as a class of UCI possessing the smallest $K_{\mathcal{C}}^{*}$. Based on prior research, the range of $K_{\mathcal{C}}^{*}$ for the optimal UCI is $2\leq K_{\mathcal{C}}^{*}\leq 2.75$. Cu…
We investigate the Wyner-Ziv coding in which the statistics of the principal source is known but the statistics of the channel generating the side-information is unknown except that it is in a certain class. The class consists of channels such that the distortion between the principal source and the side-information is smaller than a threshold, but channels may be neither stationary nor ergodic. In this situation, we define a new rate-distortion function as the minimum rate such that there exists a Wyner-Ziv code that is universal for every channel in the class. Then, we show an upper bound an…
The problem of the universal compression of a sequence from a library of several small to moderate length sequences from similar context arises in many practical scenarios, such as the compression of the storage data and the Internet traffic. In such scenarios, it is often required to compress and decompress every sequence individually. However, the universal compression of the individual sequences suffers from significant redundancy overhead. In this paper, we aim at answering whether or not having a memory unit in the middle can result in a fundamental gain in the universal compression. We p…
The problem of joint universal source coding and modeling, addressed by Rissanen in the context of lossless codes, is generalized to fixed-rate lossy coding of continuous-alphabet memoryless sources. We show that, for bounded distortion measures, any compactly parametrized family of i.i.d. real vector sources with absolutely continuous marginals (satisfying appropriate smoothness and Vapnik--Chervonenkis learnability conditions) admits a joint scheme for universal lossy block coding and parameter estimation, and give nonasymptotic estimates of convergence rates for distortion redundancies and …
We consider a novel variant of $d$-semifaithful lossy coding in which the distortion measure is revealed only to the encoder and only at run-time, as well as an extension of it in which the distortion constraint $d$ is also revealed at run-time. Two forms of rate redundancy are used to analyze the performance, and achievability results of both a pointwise and minimax nature are demonstrated. The first coding scheme uses ideas from VC dimension and growth functions, the second uses appropriate quantization of the space of distortion measures, and the third relies on a random coding argument.
In this paper, we propose {\em distributed network compression via memory}. We consider two spatially separated sources with correlated unknown source parameters. We wish to study the universal compression of a sequence of length $n$ from one of the sources provided that the decoder has access to (i.e., memorized) a sequence of length $m$ from the other source. In this setup, the correlation does not arise from symbol-by-symbol dependency of two outputs from the two sources (as in Slepian-Wolf setup). Instead, the two sequences are correlated because they are originated from the two sources wi…
Universality Sacrifices Reliability in Classical-Quantum Channel Coding
The University of Tokyo（东京大学）；RIKEN Center for Quantum Computing (RQC)（理化学研究所量子计算中心）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；International Quantum Academy（国际量子学院）；Nagoya University（名古屋大学）；National Taiwan University（台湾大学）；National Center for Theoretical Sciences, Taiwan（台湾理论科学中心）；Hon Hai (Foxconn) Quantum Computing Center（鸿海（富士康）量子计算中心）
The Lempel-Ziv universal coding scheme is asymptotically optimal for the class of all stationary ergodic sources. A problem of robustness of this property under small violations of ergodicity is studied. A notion of deficiency of algorithmic randomness is used as a measure of disagreement between data sequence and probability measure. We prove that universal compressing schemes from a large class are non-robust in the following sense: if the randomness deficiency grows arbitrarily slowly on initial fragments of an infinite sequence then the property of asymptotic optimality of any universal co…
Sparse data models, where data is assumed to be well represented as a linear combination of a few elements from a dictionary, have gained considerable attention in recent years, and their use has led to state-of-the-art results in many signal and image processing tasks. It is now well understood that the choice of the sparsity regularization term is critical in the success of such models. Based on a codelength minimization interpretation of sparse coding, and using tools from universal coding theory, we propose a framework for designing sparsity regularization terms which have theoretical and …
Suppose that we have a method which estimates the conditional probabilities of some unknown stochastic source and we use it to guess which of the outcomes will happen. We want to make a correct guess as often as it is possible. What estimators are good for this? In this work, we consider estimators given by a familiar notion of universal coding for stationary ergodic measures, while working in the framework of algorithmic randomness, i.e, we are particularly interested in prediction of Martin-Löf random points. We outline the general theory and exhibit some counterexamples. Completing a result…
