Hash Property and Coding Theorems for Sparse Matrices and Maximum-Likelihood Coding
Jun Muramatsu, Shigeki Miyake
备注：
This manuscript has been submitted to IEEE Transactions on Information Theory and a part of this manuscript has been submitted to IEEE International Symposium on Information Theory (ISIT2008,ISIT2009). 55 pages v2: major changes
发表信息：
IEEE Transactions on Information Theory, vol 56, no. 5, pp.2143-2167, May 2010; Corrections: IEEE Transactions on Information Theory, vol. 56, no.9, p. 4762, Sep. 2010. Corrections: vol.56, no.9, p.4762, 2010
The aim of this paper is to prove the achievability of several coding problems by using sparse matrices (the maximum column weight grows logarithmically in the block length) and maximal-likelihood (ML) coding. These problems are the Slepian-Wolf problem, the Gel'fand-Pinsker problem, the Wyner-Ziv problem, and the One-helps-one problem (source coding with partial side information at the decoder). To this end, the notion of a hash property for an ensemble of functions is introduced and it is proved that an ensemble of $q$-ary sparse matrices satisfies the hash property. Based on this property, it is proved that the rate of codes using sparse matrices and maximal-likelihood (ML) coding can achieve the optimal rate.
The aim of this paper is to prove the achievability of fixed-rate universal coding problems by using our previously introduced notion of hash property. These problems are the fixed-rate lossless universal source coding problem and the fixed-rate universal channel coding problem. Since an ensemble of sparse matrices satisfies the hash property requirement, it is proved that we can construct universal codes by using sparse matrices.
The aim of this paper is to prove theorems for the Slepian-Wolf source coding and the broadcast channel coding (independent messages and no common message) based on the the notion of a stronger version of the hash property for an ensemble of functions. Since an ensemble of sparse matrices has a strong hash property, codes using sparse matrices can realize the achievable rate region. Furthermore, extensions to the multiple source coding and multiple output broadcast channel coding are investigated.
This purpose of this letter is to handle a gap that was found in the proof of Theorem 2 in the paper "The generalized stochastic likelihood decoder: random coding and expurgated bounds."
The likelihood encoder with a random codebook is demonstrated as an effective tool for source coding. Coupled with a soft covering lemma (associated with channel resolvability), likelihood encoders yield simple achievability proofs for known results, such as rate-distortion theory. They also produce a tractable analysis for secure rate-distortion theory and strong coordination.
Multi-hop Deep Joint Source-Channel Coding with Deep Hash Distillation for Semantically Aligned Image Recovery
Department of Electrical and Electronic Engineering, Imperial College London, UK（电气与电子工程系，帝国理工学院伦敦分校，英国）；Lehrstuhl für Nachrichtentechnik, Technische Universität Dortmund, Germany（信息论系，德意志理工大学多特蒙德分校，德国）
The likelihood decoder is a stochastic decoder that selects the decoded message at random, using the posterior distribution of the true underlying message given the channel output. In this work, we study a generalized version of this decoder where the posterior is proportional to a general function that depends only on the joint empirical distribution of the output vector and the codeword. This framework allows both mismatched versions and universal (MMI) versions of the likelihood decoder, as well as the corresponding ordinary deterministic decoders, among many others. We provide a direct ana…
The aim of this paper is to introduce the construction of codes for a general discrete stationary memoryless multiple access channel based on the the notion of the hash property. Since an ensemble of sparse matrices has a hash property, we can use sparse matrices for code construction. Our approach has a potential advantage compared to the conventional random coding because it is expected that we can use some approximation algorithms by using the sparse structure of codes.
In this work, a likelihood encoder is studied in the context of lossy source compression. The analysis of the likelihood encoder is based on a soft-covering lemma. It is demonstrated that the use of a likelihood encoder together with the soft-covering lemma gives alternative achievability proofs for classical source coding problems. The case of the rate-distortion function with side information at the decoder (i.e. the Wyner-Ziv problem) is carefully examined and an application of the likelihood encoder to the multi-terminal source coding inner bound (i.e. the Berger-Tung region) is outlined.
Separating hash families are useful combinatorial structures which are generalizations of many well-studied objects in combinatorics, cryptography and coding theory. In this paper, using tools from graph theory and additive number theory, we solve several open problems and conjectures concerning bounds and constructions for separating hash families. Firstly, we discover that the cardinality of a separating hash family satisfies a Johnson-type inequality. As a result, we obtain a new upper bound, which is superior to all previous ones. Secondly, we present a construction for an infinite class o…
We consider stability of scheduled multiaccess message communication with random coding and joint maximum-likehood decoding of messages. The framework we consider here models both the random message arrivals and the subsequent reliable communication by suitably combining techniques from queueing theory and information theory. The number of messages that may be scheduled for simultaneous transmission is limited to a given maximum value, and the channels from transmitters to receiver are quasi-static, flat, and have independent fades. Requests for message transmissions are assumed to arrive acco…
A likelihood encoder is studied in the context of lossy source compression. The analysis of the likelihood encoder is based on the soft-covering lemma. It is demonstrated that the use of a likelihood encoder together with the soft-covering lemma yields simple achievability proofs for classical source coding problems. The cases of the point-to-point rate-distortion function, the rate-distortion function with side information at the decoder (i.e. the Wyner-Ziv problem), and the multi-terminal source coding inner bound (i.e. the Berger-Tung problem) are examined in this paper. Furthermore, a non-…
An efficient decoder for the generalized first-order Reed-Muller code RM_q(1,m) is essential for the decoding of various block-coding schemes for orthogonal frequency-division multiplexing with reduced peak-to-mean power ratio. We present an efficient and simple maximum-likelihood decoding algorithm for RM_q(1,m). It is shown that this algorithm has lower complexity than other previously known maximum-likelihood decoders for RM_q(1,m).
INSTITUT NATIONAL DE RECHERCHE EN INFORMATIQUE ET EN AUTOMATIQUE（法国国家信息与自动化研究所）
This paper proposes a binarization scheme for vectors of high dimension based on the recent concept of anti-sparse coding, and shows its excellent performance for approximate nearest neighbor search. Unlike other binarization schemes, this framework allows, up to a scaling factor, the explicit reconstruction from the binary representation of the original vector. The paper also shows that random projections which are used in Locality Sensitive Hashing algorithms, are significantly outperformed by regular frames for both synthetic and real data if the number of bits exceeds the vector dimensiona…
Likelihood-based generative models are the backbones of lossless compression due to the guaranteed existence of codes with lengths close to negative log likelihood. However, there is no guaranteed existence of computationally efficient codes that achieve these lengths, and coding algorithms must be hand-tailored to specific types of generative models to ensure computational efficiency. Such coding algorithms are known for autoregressive models and variational autoencoders, but not for general types of flow models. To fill in this gap, we introduce local bits-back coding, a new compression tech…
In this work we develop the maximum likelihood detection (MLD) algorithm for noncoherent amplitude shift keying (NCASK) systems in additive white Gaussian noise (AWGN) channels. The developed algorithm was used to investigate the performance of the NCASK system with convolutional coding and soft-decision Viterbi decoding. Tight and simple upper bounds have been derived to describe the system performance; simulation results have shown that the derived upper bounds are within 0.1 dB of the simulated points.
Index coded PSK modulation over an AWGN broadcast channel, for a given index coding problem (ICP) is studied. For a chosen index code and an arbitrary mapping (of broadcast vectors to PSK signal points), we have derived a decision rule for the maximum likelihood (ML) decoder. The message error performance of a receiver at high SNR is characterized by a parameter called PSK Index Coding Gain (PSK-ICG). The PSK-ICG of a receiver is determined by a metric called minimum inter-set distance. For a given ICP with an order of priority among the receivers, and a chosen $2^N$-PSK constellation we propo…
The aim of this paper is to prove coding theorems for the wiretap channel coding problem and secret key agreement problem based on the the notion of a hash property for an ensemble of functions. These theorems imply that codes using sparse matrices can achieve the optimal rate. Furthermore, fixed-rate universal coding theorems for a wiretap channel and a secret key agreement are also proved.
An adaptive delay-tolerant distributed space-time coding (DSTC) scheme that exploits feedback is proposed for two-hop cooperative MIMO networks. Maximum likelihood (ML) receivers and adjustable code matrices are considered subject to a power constraint with a decode-and-forward (DF) cooperation strategy. In the proposed delay-tolerant DSTC scheme, an adjustable code matrix is employed to transform the space-time coded matrix at the relay nodes. Least-squares (LS) algorithms are developed with reduced computational complexity to adjust the parameters of the codes. Simulation results show that t…
This work contains two main contributions concerning the expurgation of hierarchical ensembles for the asymmetric broadcast channel. The first is an analysis of the optimal maximum likelihood (ML) decoders for the weak and strong user. Two different methods of code expurgation will be used, that will provide two competing error exponents. The second is the derivation of expurgated exponents under the generalized stochastic likelihood decoder (GLD). We prove that the GLD exponents are at least as tight as the maximum between the random coding error exponents derived in an earlier work by Averbu…
An erasure channel with a fixed alphabet size $q$, where $q \gg 1$, is studied. It is proved that over any erasure channel (with or without memory), Maximum Distance Separable (MDS) codes achieve the minimum probability of error (assuming maximum likelihood decoding). Assuming a memoryless erasure channel, the error exponent of MDS codes are compared with that of random codes and linear random codes. It is shown that the envelopes of all these exponents are identical for rates above the critical rate. Noting the optimality of MDS codes, it is concluded that both random codes and linear random …
Motivated by a wide-spread use of convex optimization techniques, convexity properties of bit error rate of the maximum likelihood detector operating in the AWGN channel are studied for arbitrary constellations and bit mappings, which also includes coding under maximum-likelihood decoding. Under this generic setting, the pairwise probability of error and bit error rate are shown to be convex functions of the SNR and noise power in the high SNR/low noise regime with explicitly-determined boundary. Any code, including capacity-achieving ones, whose decision regions include the hardened noise sph…
In this paper, network error control coding is studied for robust and efficient multicast in a directed acyclic network with imperfect links. The block network error control coding framework, BNEC, is presented and the capability of the scheme to correct a mixture of symbol errors and packet erasures and to detect symbol errors is studied. The idea of syndrome-based decoding and error detection is introduced for BNEC, which removes the effect of input data and hence decreases the complexity. Next, an efficient three-stage syndrome-based BNEC decoding scheme for network error correction is prop…
The 3D MIMO code is a robust and efficient space-time coding scheme for the distributed MIMO broadcasting. However, it suffers from the high computational complexity if the optimal maximum-likelihood (ML) decoding is used. In this paper we first investigate the unique properties of the 3D MIMO code and consequently propose a simplified decoding algorithm without sacrificing the ML optimality. Analysis shows that the decoding complexity is reduced from O(M^8) to O(M^{4.5}) in quasi-static channels when M-ary square QAM constellation is used. Moreover, we propose an efficient implementation of t…
The golden code is a full-rate full-diversity space-time code for two transmit antennas that has a maximal coding gain. Because each codeword conveys four information symbols from an M-ary quadrature-amplitude modulation alphabet, the complexity of an exhaustive search decoder is proportional to M^2. In this paper we present a new fast algorithm for maximum-likelihood decoding of the golden code that has a worst-case complexity of only O(2M^2.5). We also present an efficient implementation of the fast decoder that exhibits a low average complexity. Finally, in contrast to the overlaid Alamouti…
Due to dependence between codeword elements, index modulation (IM) and related modulation techniques struggle to provide simple solutions for practical problems such as Gray coding between information bits and constellation points; and low-complexity log-likelihood ratio (LLR) calculations for channel-encoded information bits. In this paper, we show that a modulation technique based on a simple maximum distance separable (MDS) code, in other words, MDS modulation, can provide simple yet effective solutions to these problems, rendering the MDS techniques more beneficial in the presence of codin…
We derive various error exponents in the bee identification problem under two different decoding rules. Under naïve decoding, which decodes each bee independently of the others, we analyze a general discrete memoryless channel and a relatively wide family of stochastic decoders. Upper and lower bounds to the random coding error exponent are derived and proved to be equal at relatively high coding rates. Then, we propose a lower bound on the error exponent of the typical random code, which improves upon the random coding exponent at low coding rates. We also derive a third bound, which is relat…
Maximum-likelihood decoding is one of the central algorithmic problems in coding theory. It has been known for over 25 years that maximum-likelihood decoding of general linear codes is NP-hard. Nevertheless, it was so far unknown whether maximum- likelihood decoding remains hard for any specific family of codes with nontrivial algebraic structure. In this paper, we prove that maximum-likelihood decoding is NP-hard for the family of Reed-Solomon codes. We moreover show that maximum-likelihood decoding of Reed-Solomon codes remains hard even with unlimited preprocessing, thereby strengthening a …
We consider upper bounds on the error probability in channel coding. We derive an improved maximum-likelihood union bound, which takes into account events where the likelihood of the correct codeword is tied with that of some competitors. We compare this bound to various previous results, both qualitatively and quantitatively. With respect to maximal error probability of linear codes, we observe that when the channel is additive, the derivation of bounds, as well as the assumptions on the admissible encoder and decoder, simplify considerably.
The normalized maximum likelihood (NML) is one of the most important distribution in coding theory and statistics. NML is the unique solution (if exists) to the pointwise minimax regret problem. However, NML is not defined even for simple family of distributions such as the normal distributions. Since there does not exist any meaningful minimax-regret distribution for such case, it is pointed out that NML with luckiness (LNML) can be employed as an alternative to NML. In this paper, we develop the closed form of LNMLs for multivariate normal distributions.
In this paper, we propose CodedSketch, as a distributed straggler-resistant scheme to compute an approximation of the multiplication of two massive matrices. The objective is to reduce the recovery threshold, defined as the total number of worker nodes that we need to wait for to be able to recover the final result. To exploit the fact that only an approximated result is required, in reducing the recovery threshold, some sorts of pre-compression are required. However, compression inherently involves some randomness that would lose the structure of the matrices. On the other hand, considering t…
Yale University（耶鲁大学）；University Helsinki（赫尔辛基大学）；Nara Institute of Science and Technology（奈良先端科学技术大学院大学）
The normalized maximized likelihood (NML) provides the minimax regret solution in universal data compression, gambling, and prediction, and it plays an essential role in the minimum description length (MDL) method of statistical modeling and estimation. Here we show that the normalized maximum likelihood has a Bayes-like representation as a mixture of the component models, even in finite samples, though the weights of linear combination may be both positive and negative. This representation addresses in part the relationship between MDL and Bayes modeling. This representation has the advantage…
This paper considers the average complexity of maximum likelihood (ML) decoding of convolutional codes. ML decoding can be modeled as finding the most probable path taken through a Markov graph. Integrated with the Viterbi algorithm (VA), complexity reduction methods such as the sphere decoder often use the sum log likelihood (SLL) of a Markov path as a bound to disprove the optimality of other Markov path sets and to consequently avoid exhaustive path search. In this paper, it is shown that SLL-based optimality tests are inefficient if one fixes the coding memory and takes the codeword length…
A wireless communication system using multiple antennas promises reliable transmission under Rayleigh flat fading assumptions. Design criteria and practical schemes have been presented for both coherent and non-coherent communication channels. In this paper we generalize one dimensional phase shift keying (PSK) signals and introduce space time constellations from generalized phase shift keying (GPSK) signals based on the complex and real orthogonal designs. The resulting space time constellations reallocate the energy for each transmitting antenna and feature good diversity products, consequen…
