Distributed Source Coding Using Constrained-Random-Number Generators
Jun Muramatsu
备注：
25 pages, this is the extended version of the paper submitted to ISIT2024, appendices are the revision of arXiv:2206.00792, (v2) add Refs. [10][31], (v3) fix typos
针对 Jana 和 Blahut 提出的通用分布式无损/有损信源编码问题，本文利用熵函数刻画其多字符率失真区域，并基于受限随机数生成器构造编码证明可实现性。
摘要：
本文研究了由 Jana 和 Blahut 提出的通用分布式无损/有损信源编码。其多字符率失真区域是 Yang 和 Qin 推导区域的替代方案，通过任意一般相关信源的熵函数来刻画。通过构造基于受限随机数生成器的码，证明了该区域的可实现性。
英文摘要：
This paper investigates the general distributed lossless/lossy source coding formulated by Jana and Blahut. Their multi-letter rate-distortion region, an alternative to the region derived by Yang and Qin, is characterized by entropy functions for arbitrary general correlated sources. Achievability is shown by constructing a code based on constrained-random number generators.
Stochastic encoders for channel coding and lossy source coding are introduced with a rate close to the fundamental limits, where the only restriction is that the channel input alphabet and the reproduction alphabet of the lossy source code are finite. Random numbers, which satisfy a condition specified by a function and its value, are used to construct stochastic encoders. The proof of the theorems is based on the hash property of an ensemble of functions, where the results are extended to general channels/sources and alternative formulas are introduced for channel capacity and the rate-distor…
We show that the broadcast capacity of an infinite-depth tree-structured network of error-free half-duplex-constrained relays can be achieved using constrained coding at the source and symbol forwarding at the relays.
Necessary conditions for asymptotically optimal sliding-block or stationary codes for source coding and rate-constrained simulation of memoryless sources are presented and used to motivate a design technique for trellis-encoded source coding and rate-constrained simulation. The code structure has intuitive similarities to classic random coding arguments as well as to ``fake process'' methods and alphabet-constrained methods. Experimental evidence shows that the approach provides comparable or superior performance in comparison with previously published methods on common examples, sometimes by …
A general multi-terminal source code and a general multi-terminal channel code are presented. Constrained-random-number generators with sparse matrices, which are building blocks for the code construction, are used in the construction of both encoders and decoders. Achievable regions for source coding and channel coding are derived in terms of entropy functions, where the capacity region for channel coding provides an alternative to the region of [Somekh-Baruch and Verdú, ISIT2006].
Bi-modal (respectively, multi-modal) constrained coding refers to an encoding model whereby a user input block can be mapped to two (respectively, multiple) codewords. In current storage applications, such as optical disks, multi-modal coding allows to achieve DC control, in addition to satisfying the runlength limited (RLL) constraint specified by the recording channel. In this work, a study is initiated on bi-modal fixed-length constrained encoders. Necessary and sufficient conditions are presented for the existence of such encoders for a given constraint. It is also shown that under somewha…
The zero-error channel capacity is the maximum asymptotic rate that can be reached with error probability exactly zero, instead of a vanishing error probability. The nature of this problem, essentially combinatorial rather than probabilistic, has led to various researches both in Information Theory and Combinatorics. However, the zero-error capacity is still an open problem, for example the capacity of the noisy-typewriter channel with 7 letters is unknown. In this article, we propose a new approach to construct optimal zero-error codes, based on the concatenation of words of variable-length, …
A joint communication and channel state estimation problem is investigated, in which reliable information transmission over a noisy channel, and high-fidelity estimation of the channel state, are simultaneously sought. The tradeoff between the achievable information rate and the estimation distortion is quantified by formulating the problem as a constrained channel coding problem, and the resulting capacity-distortion function characterizes the fundamental limit of the joint communication and channel estimation problem. The analytical results are illustrated through case studies, and further i…
Huffman coding is well known to be useful in certain decision problems involving minimizing the average number of (freely chosen) queries to determine an unknown random variable. However, in problems where the queries are more constrained, the original Huffman coding no longer works. In this paper, we proposed a general model to describe such problems and two code schemes: one is Huffman-based, and the other called GBSC (Greedy Binary Separation Coding). We proved the optimality of GBSC by induction on a binary decision tree, telling us that GBSC is at least as good as Shannon coding. We then …
In this work, a coding technique called cost constrained Geometric Huffman coding (ccGhc) is developed. ccGhc minimizes the Kullback-Leibler distance between a dyadic probability mass function (pmf) and a target pmf subject to an affine inequality constraint. An analytical proof is given that when ccGhc is applied to blocks of symbols, the optimum is asymptotically achieved when the blocklength goes to infinity. The derivation of ccGhc is motivated by the problem of encoding a text to a sequence of slats subject to architectural design criteria. For the considered architectural problem, for a …
This paper considers the memoryless input-constrained binary erasure channel (BEC). The channel input constraint is the $(d,\infty)$-runlength limited (RLL) constraint, which mandates that any pair of successive $1$s in the input sequence be separated by at least $d$ $0$s. We consider a scenario where there is causal, noiseless feedback from the decoder. We demonstrate a simple, labelling-based, zero-error feedback coding scheme, which we prove to be feedback capacity-achieving, and, as a by-product, obtain an explicit characterization of the feedback capacity. Our proof is based on showing th…
The constrained linear representability problem (CLRP) for polymatroids determines whether there exists a polymatroid that is linear over a specified field while satisfying a collection of constraints on the rank function. Using a computer to test whether a certain rate vector is achievable with vector linear network codes for a multi-source network coding instance and whether there exists a multi-linear secret sharing scheme achieving a specified information ratio for a given secret sharing instance are shown to be special cases of CLRP. Methods for solving CLRP built from group theoretic tec…
We present a constraint-coding scheme to correct asymmetric magnitude-$1$ errors in multi-level non-volatile memories. For large numbers of such errors, the scheme is shown to deliver better correction capability compared to known alternatives, while admitting low-complexity of decoding. Our results include an algebraic formulation of the constraint, necessary and sufficient conditions for correctability, a maximum-likelihood decoder running in complexity linear in the alphabet size, and upper bounds on the probability of failing to correct $t$ errors. Besides the superior rate-correction trad…
Privacy-constrained source coding problems have become increasingly important recently, and the utility-privacy trade-off has been investigated for various systems. As pioneering work, Yamamoto (1983) found theoretical limits of the coding rate, privacy and utility in two cases; (i) both public and private information is encoded and (ii) only public information is encoded. However, the theoretical limit has not been characterized in a more general case; (iii) encoded messages consist of public information and a part of private information. Then in this paper, we characterize the trade-off rela…
Support constrained generator matrix for a linear code has been an active topic in recent years. The necessary and sufficient condition for the existence of MDS codes over small fields with support constrained generator matrices were conjectured by Dau, Song, Yuen in 2014. This GM-MDS conjecture was proved independently by Lovett and Yildiz-Hassibi in 2018. In this paper we propose the necessary and sufficient conditions for support constrained generator matrices of general linear codes based on the generalized Hamming weights. It is proved that the direct generalization of the GM-MDS conjectu…
In this paper, we propose a new distributed coding structure with a soft input soft output (SISO) relay encoder for error-prone parallel relay channels. We refer to it as the distributed soft coding (DISC). In the proposed scheme, each relay first uses the received noisy signals to calculate the soft bit estimate (SBE) of the source symbols. A simple SISO encoder is developed to encode the SBEs of source symbols based on a constituent code generator matrix. The SISO encoder outputs at different relays are then forwarded to the destination and form a distributed codeword. The performance of the…
We consider the problem of revealing/sharing data in an efficient and secure way via a compact representation. The representation should ensure reliable reconstruction of the desired features/attributes while still preserve privacy of the secret parts of the data. The problem is formulated as a remote lossy source coding with a privacy constraint where the remote source consists of public and secret parts. Inner and outer bounds for the optimal tradeoff region of compression rate, distortion, and privacy leakage rate are given and shown to coincide for some special cases. When specializing the…
A scenario involving a source, a channel, and a destination, where the destination is interested in {\em both} reliably reconstructing the message transmitted by the source and estimating with a fidelity criterion the state of the channel, is considered. The source knows the channel statistics, but is oblivious to the actual channel state realization. Herein it is established that a distortion constraint for channel state estimation can be reduced to an additional cost constraint on the source input distribution, in the limit of large coding block length. A newly defined capacity-distortion fu…
We present a general theory of entanglement-assisted quantum convolutional coding. The codes have a convolutional or memory structure, they assume that the sender and receiver share noiseless entanglement prior to quantum communication, and they are not restricted to possess the Calderbank-Shor-Steane structure as in previous work. We provide two significant advances for quantum convolutional coding theory. We first show how to "expand" a given set of quantum convolutional generators. This expansion step acts as a preprocessor for a polynomial symplectic Gram-Schmidt orthogonalization procedur…
The source-coding problem with side information at the decoder is studied subject to a constraint that the encoder---to whom the side information is unavailable---be able to compute the decoder's reconstruction sequence to within some distortion. For discrete memoryless sources and finite single-letter distortion measures, an expression is given for the minimal description rate as a function of the joint law of the source and side information and of the allowed distortions at the encoder and at the decoder. The minimal description rate is also computed for a memoryless Gaussian source with squ…
Previous work by the authors on parity-preserving fixed-length constrained encoders is extended to the variable-length case. Parity-preserving variable-length encoders are formally defined, and, to this end, Kraft conditions are developed for the parity-preserving variable-length setting. Then, a necessary and sufficient condition is presented for the existence of deterministic parity-preserving variable-length encoders for a given constraint. Examples are provided that show that there are coding ratios where parity-preserving variable-length encoders exist, while fixed-length encoders do not.
We investigate the Wyner-Ziv coding in which the statistics of the principal source is known but the statistics of the channel generating the side-information is unknown except that it is in a certain class. The class consists of channels such that the distortion between the principal source and the side-information is smaller than a threshold, but channels may be neither stationary nor ergodic. In this situation, we define a new rate-distortion function as the minimum rate such that there exists a Wyner-Ziv code that is universal for every channel in the class. Then, we show an upper bound an…
The pivotal storage density win achieved by solid-state devices over magnetic devices in 2015 is a result of multiple innovations in physics, architecture, and signal processing. One of the most important innovations in that regard is enabling the storage of more than one bit per cell in the Flash device, i.e., having more than two charge levels per cell. Constrained coding is used in Flash devices to increase reliability via mitigating inter-cell interference that stems from charge propagation among cells. Recently, capacity-achieving constrained codes were introduced to serve that purpose in…
This paper studies fixed-rate randomized vector quantization under the constraint that the quantizer's output has a given fixed probability distribution. A general representation of randomized quantizers that includes the common models in the literature is introduced via appropriate mixtures of joint probability measures on the product of the source and reproduction alphabets. Using this representation and results from optimal transport theory, the existence of an optimal (minimum distortion) randomized quantizer having a given output distribution is shown under various conditions. For sources…
DNA Data storage has recently attracted much attention due to its durable preservation and extremely high information density (bits per gram) properties. In this work, we propose a hybrid coding strategy comprising of generalized constrained codes to tackle homopolymer (run-length) limit and a protograph based low-density parity-check (LDPC) code to correct asymmetric nucleotide level (i.e., A/T/C/G) substitution errors that may occur in the process of DNA sequencing. Two sequencing techniques namely, Nanopore sequencer and Illumina sequencer with their equivalent channel models and capacities…
In many information-theoretic channel coding problems, adding an input cost constraint to the operational setup amounts to restricting the optimization domain in the capacity formula. This paper shows that, in contrast to common belief, such a simple modification does not hold for the cost-constrained (CC) wiretap channel (WTC). The secrecy-capacity of the discrete memoryless (DM) WTC without cost constraints is described by a single auxiliary random variable. For the CC DM-WTC, however, we show that two auxiliaries are necessary to achieve capacity. Specifically, we first derive the secrecy-c…
The input-constrained binary erasure channel (BEC) with strictly causal feedback is studied. The channel input sequence must satisfy the $(0,k)$-runlength limited (RLL) constraint, i.e., no more than $k$ consecutive `$0$'s are allowed. The feedback capacity of this channel is derived for all $k\geq 1$, and is given by $$C^\mathrm{fb}_{(0,k)}(\varepsilon) = \max\frac{\overline{\varepsilon}H_2(δ_0)+\sum_{i=1}^{k-1}\left(\overline{\varepsilon}^{i+1}H_2(δ_i)\prod_{m=0}^{i-1}δ_m\right)}{1+\sum_{i=0}^{k-1}\left(\overline{\varepsilon}^{i+1} \prod_{m=0}^{i}δ_m\right)},$$ where $\varepsilon$ is the era…
The demand of two-dimensional source coding and constrained coding has been getting higher these days, but compared to the one-dimensional case, many problems have remained open as the analysis is cumbersome. A main reason for that would be because there are no graph representations discovered so far. In this paper, we focus on a two-dimensional finite type constrained system, a set of two-dimensional blocks characterized by a finite number of two-dimensional constraints, and propose its graph representation. We then show how to generate an element of the two-dimensional finite type constraine…
Ahlswede and Dueck showed possibility to identify with high probability one out of $M$ messages by transmitting $1/C\log\log M$ bits only, where $C$ is the channel capacity. It is known that this identification can be based on error-correcting codes. We propose an identification procedure based on random codes that achieves channel capacity. Then we show that this procedure can be simplified using pseudo-random generators.
We give constructions of some special cases of $[n,k]$ Reed-Solomon codes over finite fields of size at least $n$ and $n+1$ whose generator matrices have constrained support. Furthermore, we consider a generalisation of the GM-MDS conjecture proposed by Lovett in 2018. We show that Lovett's conjecture is false in general and we specify when the conjecture is true.
We consider a wireless broadcast station that transmits packets to multiple users. The packet requests for each user may overlap, and some users may already have certain packets. This presents a problem of broadcasting in the presence of side information, and is a generalization of the well known (and unsolved) index coding problem of information theory. Rather than achieving the full capacity region, we develop a code-constrained capacity region, which restricts attention to a pre-specified set of coding actions. We develop a dynamic max-weight algorithm that allows for random packet arrivals…
This paper studies a Shannon-theoretic version of the generalized distribution preserving quantization problem where a stationary and memoryless source is encoded subject to a distortion constraint and the additional requirement that the reproduction also be stationary and memoryless with a given distribution. The encoder and decoder are stochastic and assumed to have access to independent common randomness. Recent work has characterized the minimum achievable coding rate at a given distortion level when unlimited common randomness is available. Here we consider the general case where the avai…
A network coding-based scheme is proposed to improve the energy efficiency of distributed storage systems in WSNs (wireless sensor networks), which mainly focuses on two problems: firstly, consideration is given to effective distributed storage technology in WSNs; secondly, we address how to repair the data in failed storage nodes with less resource. For the first problem, we propose a method to obtain a sparse generator matrix to construct network codes, and this sparse generator matrix is proven to be the sparsest. Benefiting from the sparse generator matrix, the energy consumption required …
This paper investigates the achievability of the interference channel coding. It is clarified that the rate-splitting technique is unnecessary to achieve Han-Kobayashi and Jian-Xin-Garg inner regions. Codes are constructed by using sparse matrices (with logarithmic column degree) and the constrained-random-number generators. By extending the problem, we can establish a possible extension of known inner regions.
Minimum energy (ME) source coding is an effective technique for efficient communication with energy-constrained devices, such as sensor network nodes. In this paper, the principles of generalized ME source coding is developed that is broadly applicable. Two scenarios - fixed and variable length codewords - are analyzed. The application of this technique to RFID systems where ME source coding is particularly advantageous due to the asymmetric nature of data communications is demonstrated, a first to the best of our knowledge.
Gabidulin codes over fields of characteristic zero were recently constructed by Augot et al., whenever the Galois group of the underlying field extension is cyclic. In parallel, the interest in sparse generator matrices of Reed-Solomon and Gabidulin codes has increased lately, due to applications in distributed computations. In particular, a certain condition pertaining the intersection of zero entries at different rows, was shown to be necessary and sufficient for the existence of the sparsest possible generator matrix of Gabidulin codes over finite fields. In this paper we complete the pictu…
In this work, we address the question of the largest rate of linear subcodes of Reed-Muller (RM) codes, all of whose codewords respect a runlength-limited (RLL) constraint. Our interest is in the $(d,\infty)$-RLL constraint, which mandates that every pair of successive $1$s be separated by at least $d$ $0$s. Consider any sequence $\{{\mathcal{C}_m}\}_{m\geq 1}$ of RM codes with increasing blocklength, whose rates approach $R$, in the limit as the blocklength goes to infinity. We show that for any linear $(d,\infty)$-RLL subcode, $\hat{\mathcal{C}}_m$, of the code $\mathcal{C}_m$, it holds that…
A coding scheme based on irregular low-density parity-check (LDPC) codes is proposed to send secret messages from a source over the Gaussian wiretap channel to a destination in the presence of a wiretapper, with the restriction that the source can send only binary phase-shift keyed (BPSK) symbols. The secrecy performance of the proposed coding scheme is measured by the secret message rate through the wiretap channel as well as the equivocation rate about the message at the wiretapper. A code search procedure is suggested to obtain irregular LDPC codes that achieve good secrecy performance in s…
This paper focuses on the encoding and decoding of Construction D' coding lattices that can be used with shaping lattices for power-constrained channels. Two encoding methods and a decoding algorithm for Construction D' lattices are given. A design of quasi-cyclic low-density parity-check (QC-LDPC) codes to form Construction D' lattices is presented. This allows construction of nested lattice codes which are good for coding, good for shaping, and have low complexity encoding and decoding. Numerical results of using $E_8$, $BW_{16}$ and Leech lattices for shaping a Construction D' lattice indic…
In this paper we consider the use of variable length non prefix-free codes for coding constrained sequences of symbols. We suppose to have a Markov source where some state transitions are impossible, i.e. the stochastic matrix associated with the Markov chain has some null entries. We show that classic Kraft inequality is not a necessary condition, in general, for unique decodability under the above hypothesis and we propose a relaxed necessary inequality condition. This allows, in some cases, the use of non prefix-free codes that can give very good performance, both in terms of compression an…
We study a new encoding scheme for lossy source compression based on spatially coupled low-density generator-matrix codes. We develop a belief-propagation guided-decimation algorithm, and show that this algorithm allows to approach the optimal distortion of spatially coupled ensembles. Moreover, using the survey propagation formalism, we also observe that the optimal distortions of the spatially coupled and individual code ensembles are the same. Since regular low-density generator-matrix codes are known to achieve the Shannon rate-distortion bound under optimal encoding as the degrees grow, o…
Designs and methods for nested lattice codes using Construction D' lattices for coding and convolutional code lattices for shaping are described. Two encoding methods and a decoding algorithm for Construction D' coding lattices that can be used with shaping lattices for power-constrained channels are given. We construct nested lattice codes with good coding properties, a high shaping gain, and low-complexity encoding and decoding. Convolutional code generator polynomials for Construction A lattices with the greatest shaping gain are given, as a result of an extensive search. It is shown that r…
Gabidulin codes are the first general construction of linear codes that are maximum rank distant (MRD). They have found applications in linear network coding, for example, when the transmitter and receiver are oblivious to the inner workings and topology of the network (the so-called incoherent regime). The reason is that Gabidulin codes can be used to map information to linear subspaces, which in the absence of errors cannot be altered by linear operations, and in the presence of errors can be corrected if the subspace is perturbed by a small rank. Furthermore, in distributed coding and distr…
Constrained lossy source coding and channel coding with side information problems which extend the classic Wyner-Ziv and Gel'fand-Pinsker problems are considered. Inspired by applications in sensor networking and control, we first consider lossy source coding with two-sided partial side information where the quality/availability of the side information can be influenced by a cost-constrained action sequence. A decoder reconstructs a source sequence subject to the distortion constraint, and at the same time, an encoder is additionally required to be able to estimate the decoder's reconstruction…
We consider a number of constrained coding techniques that can be used to mitigate a nonlinear effect in the optical fiber channel that causes the formation of spurious pulses, called ``ghost pulses.'' Specifically, if $b_1 b_2 ... b_{n}$ is a sequence of bits sent across an optical channel, such that $b_k=b_l=b_m=1$ for some $k,l,m$ (not necessarily all distinct) but $b_{k+l-m} = 0$, then the ghost-pulse effect causes $b_{k+l-m}$ to change to 1, thereby creating an error. We design and analyze several coding schemes using binary and ternary sequences constrained so as to avoid patterns that g…
In a distributed space-time coding scheme, based on the relay channel model, the relay nodes co-operate to linearly process the transmitted signal from the source and forward them to the destination such that the signal at the destination appears as a space time block code. Recently, a code design criteria for achieving full diversity in a partially-coherent environment have been proposed along with codes based on differential encoding and decoding techniques. For such a set up, in this paper, a non-differential encoding technique and construction of distributed space time block codes from uni…
