Capacity-Achieving Codes with Bounded Graphical Complexity on Noisy Channels
Chun-Hao Hsu, Achilleas Anastasopoulos
备注：
17 pages, 2 figures. This paper is to be presented in the 43rd Annual Allerton Conference on Communication, Control and Computing, Monticello, IL, USA, Sept. 28-30, 2005
We introduce a new family of concatenated codes with an outer low-density parity-check (LDPC) code and an inner low-density generator matrix (LDGM) code, and prove that these codes can achieve capacity under any memoryless binary-input output-symmetric (MBIOS) channel using maximum-likelihood (ML) decoding with bounded graphical complexity, i.e., the number of edges per information bit in their graphical representation is bounded. In particular, we also show that these codes can achieve capacity on the binary erasure channel (BEC) under belief propagation (BP) decoding with bounded decoding complexity per information bit per iteration for all erasure probabilities in (0, 1). By deriving and analyzing the average weight distribution (AWD) and the corresponding asymptotic growth rate of these codes with a rate-1 inner LDGM code, we also show that these codes achieve the Gilbert-Varshamov bound with asymptotically high probability. This result can be attributed to the presence of the inner rate-1 LDGM code, which is demonstrated to help eliminate high weight codewords in the LDPC code while maintaining a vanishingly small amount of low weight codewords.
The capacity regions of multiple-input multiple-output Gaussian Z-interference channels are established for the very strong interference and aligned strong interference cases. The sum-rate capacity of such channels is established under noisy interference. These results generalize known results for scalar Gaussian Z-interference channels.
The adaptive zero-error capacity of discrete memoryless channels (DMC) with noiseless feedback has been shown to be positive whenever there exists at least one channel output "disprover", i.e. a channel output that cannot be reached from at least one of the inputs. Furthermore, whenever there exists a disprover, the adaptive zero-error capacity attains the Shannon (small-error) capacity. Here, we study the zero-error capacity of a DMC when the channel feedback is noisy rather than perfect. We show that the adaptive zero-error capacity with noisy feedback is lower bounded by the forward channel…
The capacity region of a broadcast channel consisting of k-receivers that lie in a less noisy sequence is an open problem, when k >= 3. We solve this problem for the case k=3. We prove that superposition coding is optimal for a class of broadcast channels with a sequence of less noisy receivers. T
This paper presents an analysis of the concept of capacity for noisy computations, i.e. algorithms implemented by unreliable computing devices (e.g. noisy Turing Machines). The capacity of a noisy computation is defined and justified by companion coding theorems. Under some constraints on the encoding process, capacity is the upper bound of input rates allowing reliable computation, i.e. decodability of noisy outputs into expected outputs. A model of noisy computation of a perfect function f thanks to an unreliable device F is given together with a model of reliable computation based on input …
We propose two coding schemes for discrete memoryless broadcast channels (DMBCs) with rate-limited feedback from only one receiver. For any positive feedback rate and for the class of strictly less-noisy DMBCs, our schemes strictly improve over the no-feedback capacity region.
Fundamental limits of the cognitive interference channel (CIC) with two pairs of transmitter-receiver has been under exploration for several years. In this paper, we study the discrete memoryless cognitive interference channel (DM-CIC) in which the cognitive transmitter non-causally knows the message of the primary transmitter. The capacity of this channel is not known in general; it is only known in some special cases. Inspired by the concept of less noisy broadcast channel (BC), in this work we introduce the notion of less noisy cognitive interference channel. Unlike BC, due to the inherent …
In this paper, we consider some long-standing problems in communication systems with access to noisy feedback. We introduce a new notion, the residual directed information, to capture the effective information flow (i.e. mutual information between the message and the channel outputs) in the forward channel. In light of this new concept, we investigate discrete memoryless channels (DMC) with noisy feedback and prove that the noisy feedback capacity is not achievable by using any typical closed-loop encoder (non-trivially taking feedback information to produce channel inputs). We then show that …
We consider the capacity of memoryless finite-state multiple access channel (FS-MAC) with causal asymmetric noisy state information available at both transmitters and complete state information available at the receiver. Single letter inner and outer bounds are provided for the capacity of such channels when the state process is independent and identically distributed. The outer bound is attained by observing that the proposed inner bound is tight for the sum-rate capacity.
New sufficient conditions for a vector Gaussian interference channel to achieve the sum-rate capacity by treating interference as noise are derived, which generalize the existing results. More concise conditions for multiple-input-single-output, and single-input-multiple-output scenarios are obtained.
This paper presents an analysis of the concept of capacity for noisy com- putations, i.e. functions implemented by unreliable or random devices. An information theoretic model of noisy computation of a perfect function f (measurable function between sequence spaces) thanks to an unreliable device (random channel) F is given: a noisy computation is a product fxF of channels. A model of reliable computation based on input encoding and output decoding is also proposed. These models extend those of noisy communication channel and of reliable communication through a noisy channel. The capacity of a…
We consider an additive Gaussian channel with additive Gaussian noise feedback. We derive an upper bound on the n-block capacity (defined by Cover [1]). It is shown that this upper bound can be obtained by solving a convex optimization problem. With stationarity assumptions on Gaussian noise processes, we characterize the limit of the n-block upper bound and prove that this limit is the upper bound of the noisy feedback (shannon) capacity.
Given side information that an Ising tree-structured graphical model is homogeneous and has no external field, we derive the exact asymptotics of learning its structure from independently drawn samples. Our results, which leverage the use of probabilistic tools from the theory of strong large deviations, refine the large deviation (error exponents) results of Tan, Anandkumar, Tong, and Willsky [IEEE Trans. on Inform. Th., 57(3):1714--1735, 2011] and strictly improve those of Bresler and Karzand [Ann. Statist., 2020]. In addition, we extend our results to the scenario in which the samples are o…
Motivated by DNA-based storage, we study the noisy shuffling channel, which can be seen as the concatenation of a standard noisy channel (such as the BSC) and a shuffling channel, which breaks the data block into small pieces and shuffles them. This channel models a DNA storage system, by capturing two of its key aspects: (1) the data is written onto many short DNA molecules that are stored in an unordered way and (2) the molecules are corrupted by noise at synthesis, sequencing, and during storage. For the BSC-shuffling channel we characterize the capacity exactly (for a large set of paramete…
Capacity-Achieving Codes for Noisy Insertion Channels
School of Information Science and Technology, Southwest Jiaotong University（西南交通大学信息科学与技术学院）；School of Mathematics, Southwest Jiaotong University（西南交通大学数学学院）
A new outer bound on the capacity region of Gaussian interference channels is developed. The bound combines and improves existing genie-aided methods and is shown to give the sum-rate capacity for noisy interference as defined in this paper. Specifically, it is shown that if the channel coefficients and power constraints satisfy a simple condition then single-user detection at each receiver is sum-rate optimal, i.e., treating the interference as noise incurs no loss in performance. This is the first concrete (finite signal-to-noise ratio) capacity result for the Gaussian interference channel w…
It is well known that, in general, feedback may enlarge the capacity region of Gaussian broadcast channels. This has been demonstrated even when the feedback is noisy (or partial-but-perfect) and only from one of the receivers. The only case known where feedback has been shown not to enlarge the capacity region is when the channel is physically degraded (El Gamal 1978, 1981). In this paper, we show that for a class of two-user Gaussian broadcast channels (not necessarily physically degraded), passively feeding back the stronger user's signal over a link corrupted by Gaussian noise does not enl…
Binary-input memoryless channels with a runlength constrained input are considered. Upper bounds to the capacity of such noisy runlength constrained channels are derived using the dual capacity method with Markov test distributions satisfying the Karush-Kuhn-Tucker (KKT) conditions for the capacity-achieving output distribution. Simplified algebraic characterizations of the bounds are presented for the binary erasure channel (BEC) and the binary symmetric channel (BSC). These upper bounds are very close to achievable rates, and improve upon previously known feedback-based bounds for a large ra…
The sum-rate capacity of the parallel Gaussian interference channel is shown to be achieved by independent transmission across sub-channels and treating interference as noise in each sub-channel if the channel coefficients and power constraints satisfy a certain condition. The condition requires the interference to be weak, a situation commonly encountered in, e.g., digital subscriber line transmission. The optimal power allocation is characterized by using the concavity of sum-rate capacity as a function of the power constraints.
An upper bound on the capacity of a cascade of nonlinear and noisy channels is presented. The cascade mimics the split-step Fourier method for computing waveform propagation governed by the stochastic generalized nonlinear Schroedinger equation. It is shown that the spectral efficiency of the cascade is at most log(1+SNR), where SNR is the receiver signal-to-noise ratio. The results may be applied to optical fiber channels. However, the definition of bandwidth is subtle and leaves open interpretations of the bound. Some of these interpretations are discussed.
In this paper, we first consider a channel that is contaminated by two independent Gaussian noises $S ~ N(0,Q)$ and $Z_0 ~ N(0,N_0)$. The capacity of this channel is computed when independent noisy versions of $S$ are known to the transmitter and/or receiver. It is shown that the channel capacity is greater then the capacity when $S$ is completely unknown, but is less then the capacity when $S$ is perfectly known at the transmitter or receiver. For example, if there is one noisy version of $S$ known at the transmitter only, the capacity is $0.5log(1+P/(Q(N_1/(Q+N_1))+N_0))$, where $P$ is the i…
We review how Shannon's classical notion of capacity is not enough to characterize a noisy communication channel if the channel is intended to be used as part of a feedback loop to stabilize an unstable scalar linear system. While classical capacity is not enough, another sense of capacity (parametrized by reliability) called ``anytime capacity'' is shown to be necessary for the stabilization of an unstable process. The required rate is given by the log of the unstable system gain and the required reliability comes from the sense of stability desired. A consequence of this necessity result is …
We consider the problem of reliable communication over multiple-access channels (MAC) where the channel is driven by an independent and identically distributed state process and the encoders and the decoder are provided with various degrees of asymmetric noisy channel state information (CSI). For the case where the encoders observe causal, asymmetric noisy CSI and the decoder observes complete CSI, we provide inner and outer bounds to the capacity region, which are tight for the sum-rate capacity. We then observe that, under a Markov assumption, similar capacity results also hold in the case w…
The rate regions of many variations of the standard and wire-tap channels have been thoroughly explored. Secrecy capacity characterizes the loss of rate required to ensure that the adversary gains no information about the transmissions. Authentication does not have a standard metric, despite being an important counterpart to secrecy. While some results have taken an information-theoretic approach to the problem of authentication coding, the full rate region and accompanying trade-offs have yet to be characterized. In this paper, we provide an inner bound of achievable rates with an average aut…
It is well known that the capacity region of an average transmit power constrained Gaussian Broadcast Channel (GBC) with independent noise realizations at the receivers is enlarged by the presence of causal noiseless feedback. Capacity region enlargement is also known to be possible by using only passive noisy feedback, when the GBC has identical noise variances at the receivers. The last fact remains true even when the feedback noise variance is very high, and available only from one of the receivers. While such capacity enlargements are feasible for several other feedback models in the Gauss…
Noisy channels are a valuable resource from a cryptographic point of view. They can be used for exchanging secret-keys as well as realizing other cryptographic primitives such as commitment and oblivious transfer. To be really useful, noisy channels have to be consider in the scenario where a cheating party has some degree of control over the channel characteristics. Damgård et al. (EUROCRYPT 1999) proposed a more realistic model where such level of control is permitted to an adversary, the so called unfair noisy channels, and proved that they can be used to obtain commitment and oblivious tra…
In this paper, an achievability region and a converse region for the two-user Gaussian interference channel with noisy channel-output feedback (G-IC-NOF) are presented. The achievability region is obtained using a random coding argument and three well-known techniques: rate splitting, superposition coding and backward decoding. The converse region is obtained using some of the existing perfect-output feedback outer-bounds as well as a set of new outer-bounds that are obtained by using genie-aided models of the original G-IC-NOF. Finally, it is shown that the achievability region and the conver…
In this technical report, the capacity region of the two-user linear deterministic (LD) interference channel with noisy output feedback (IC-NOF) is fully characterized. This result allows the identification of several asymmetric scenarios in which implementing channel-output feedback in only one of the transmitter-receiver pairs is as beneficial as implementing it in both links, in terms of achievable individual rate and sum-rate improvements w.r.t. the case without feedback. In other scenarios, the use of channel-output feedback in any of the transmitter-receiver pairs benefits only one of th…
The fact that the results for 2-receiver broadcast channels (BCs) are not generalized to the 3-receiver ones is of information theoretical importance. In this paper we study two classes of discrete memoryless BCs with non-causal side information (SI), i.e. multilevel BC (MBC) and 3-receiver less noisy BC. First, we obtain an achievable rate region and a capacity outer bound for the MBC. Second, we prove a special capacity region for the 3-receiver less noisy BC. Third, the obtained special capacity region for the 3-receiver less noisy BC is extended to continuous alphabet fading Gaussian versi…
In this paper, a channel that is contaminated by two independent Gaussian noises $S ~ N(0,Q)$ and $Z_0 ~ N(0,N_0)$ is considered. The capacity of this channel is computed when independent noisy versions of $S$ are known to the transmitter and/or receiver. It is shown that the channel capacity is greater then the capacity when $S$ is completely unknown, but is less then the capacity when $S$ is perfectly known at the transmitter or receiver. For example, if there is one noisy version of $S$ known at the transmitter only, the capacity is $0.5\log(1+\frac{P}{Q(N_1/(Q+N_1))+N_0})$, where $P$ is th…
We consider finite-state channels (FSCs) where the channel state is stochastically dependent on the previous channel output. We refer to these as Noisy Output is the STate (NOST) channels. We derive the feedback capacity of NOST channels in two scenarios: with and without causal state information (CSI) available at the encoder. If CSI is unavailable, the feedback capacity is $C_{\text{FB}}= \max_{P(x|y')} I(X;Y|Y')$, while if it is available at the encoder, the feedback capacity is $C_{\text{FB-CSI}}= \max_{P(u|y'),x(u,s')} I(U;Y|Y')$, where $U$ is an auxiliary RV with finite cardinality. In b…
In this research report, an achievability region and a converse region for the two-user Gaussian interference channel with noisy channel-output feedback (G-IC-NOF) are presented. The achievability region is obtained using a random coding argument and three well-known techniques: rate splitting, superposition coding and backward decoding. The converse region is obtained using some of the existing perfect-output feedback outer-bounds as well as a set of new outer-bounds that are obtained by using genie-aided models of the original G-IC-NOF. Finally, it is shown that the achievability region and …
In this fourth part of our multi-part papers, classes of interference networks with a sequence of less-noisy receivers are identified for which a successive decoding scheme achieve the sum-rate capacity. First, the two-receiver networks are analyzed: it is demonstrated that the unified outer bounds derived in Part III of our multi-part papers are sum-rate optimal for network scenarios which satisfy certain less-noisy conditions. Then, the multi-receiver networks are considered. These networks are far less understood. One of the main difficulties in the analysis of such scenarios is how to esta…
In this paper, the capacity region of the linear deterministic interference channel with noisy channel-output feedback (LD-IC-NF) is fully characterized. The proof of achievability is based on random coding arguments and rate splitting; block-Markov superposition coding; and backward decoding. The proof of converse reuses some of the existing outer bounds and includes new ones obtained using genie-aided models. Following the insight gained from the analysis of the LD-IC-NF, an achievability region and a converse region for the two-user Gaussian interference channel with noisy channel-output fe…
We explore the capacity and generalized degrees of freedom of the two-user Gaussian X channel, i.e. a generalization of the 2 user interference channel where there is an independent message from each transmitter to each receiver. There are three main results in this paper. First, we characterize the sum capacity of the deterministic X channel model under a symmetric setting. Second, we characterize the generalized degrees of freedom of the Gaussian X channel under a similar symmetric model. Third, we extend the noisy interference capacity characterization previously obtained for the interferen…
Aggregation of noisy observations involves a difficult tradeoff between observation quality, which can be increased by increasing the number of observations, and aggregation quality which decreases if the number of observations is too large. We clarify this behavior for a protypical system in which arbitrarily large numbers of observations exceeding the system capacity can be aggregated using lossy data compression. We show the existence of a scaling relation between the collective error and the system capacity, and show that large scale lossy aggregation can outperform lossless aggregation ab…
In this paper, the capacity region of the two-user linear deterministic (LD) interference channel with noisy output feedback (IC-NOF) is fully characterized. This result allows the identification of several asymmetric scenarios in which imple- menting channel-output feedback in only one of the transmitter- receiver pairs is as beneficial as implementing it in both links, in terms of achievable individual rate and sum-rate improvements w.r.t. the case without feedback. In other scenarios, the use of channel-output feedback in any of the transmitter-receiver pairs benefits only one of the two pa…
In part I, we reviewed how Shannon's classical notion of capacity is not sufficient to characterize a noisy communication channel if the channel is intended to be used as part of a feedback loop to stabilize an unstable scalar linear system. While classical capacity is not enough, a sense of capacity (parametrized by reliability) called "anytime capacity" is both necessary and sufficient for channel evaluation in this context. The rate required is the log of the open-loop system gain and the required reliability comes from the desired sense of stability. Sufficiency is maintained even in cases…
Motivated by a simple broadcast channel, we generalize the notions of a less noisy receiver and a more capable receiver to an essentially less noisy receiver and an essentially more capable receiver respectively. We establish the capacity regions of these classes by borrowing on existing techniques to obtain the characterization of the capacity region for certain new and interesting classes of broadcast channels. We also establish the relationships between the new classes and the existing classes.
We define capacity for group testing problems and deduce bounds for the capacity of a variety of noisy models, based on the capacity of equivalent noisy communication channels. For noiseless adaptive group testing we prove an information-theoretic lower bound which tightens a bound of Chan et al. This can be combined with a performance analysis of a version of Hwang's adaptive group testing algorithm, in order to deduce the capacity of noiseless and erasure group testing models.
We consider learning Ising tree models when the observations from the nodes are corrupted by independent but non-identically distributed noise with unknown statistics. Katiyar et al. (2020) showed that although the exact tree structure cannot be recovered, one can recover a partial tree structure; that is, a structure belonging to the equivalence class containing the true tree. This paper presents a systematic improvement of Katiyar et al. (2020). First, we present a novel impossibility result by deriving a bound on the necessary number of samples for partial recovery. Second, we derive a sign…
This paper is focused on the derivation of some universal properties of capacity-approaching low-density parity-check (LDPC) code ensembles whose transmission takes place over memoryless binary-input output-symmetric (MBIOS) channels. Properties of the degree distributions, graphical complexity and the number of fundamental cycles in the bipartite graphs are considered via the derivation of information-theoretic bounds. These bounds are expressed in terms of the target block/ bit error probability and the gap (in rate) to capacity. Most of the bounds are general for any decoding algorithm, and…
In this work, the role of noisy feedback in enhancing the secrecy capacity of the wiretap channel is investigated. A model is considered in which the feed-forward and feedback signals share the same noisy channel. More specifically, a discrete memoryless modulo-additive channel with a full-duplex destination node is considered first, and it is shown that a judicious use of feedback increases the perfect secrecy capacity to the capacity of the source-destination channel in the absence of the wiretapper. In the achievability scheme, the feedback signal corresponds to a private key, known only to…
In this paper, we consider the group testing problem with adaptive test designs and noisy outcomes. We propose a computationally efficient four-stage procedure with components including random binning, identification of bins containing defective items, 1-sparse recovery via channel codes, and a "clean-up" step to correct any errors from the earlier stages. We prove that the asymptotic required number of tests comes very close to the best known information-theoretic achievability bound (which is based on computationally intractable decoding), and approaches a capacity-based converse bound in th…
We develop several lower bounds on the capacity of binary input symmetric output channels with synchronization errors which also suffer from other types of impairments such as substitutions, erasures, additive white Gaussian noise (AWGN) etc. More precisely, we show that if the channel with synchronization errors can be decomposed into a cascade of two channels where only the first one suffers from synchronization errors and the second one is a memoryless channel, a lower bound on the capacity of the original channel in terms of the capacity of the synchronization error-only channel can be der…
Two new classes of quantum channels, which we call more capable and less noisy, are introduced. The more capable class consists of channels such that the quantum capacities of the complementary channels to the environments are zero. The less noisy class consists of channels such that the private capacities of the complementary channels to the environment are zero. For the more capable class, it is clarified that the private capacity and quantum capacity coincide. For the less noisy class, it is clarified that the private capacity and quantum capacity can be single letter characterized.
This work investigates the secrecy capacity of the Wiretap Broadcast Channel (WBC) with an external eavesdropper where a source wishes to communicate two private messages over a Broadcast Channel (BC) while keeping them secret from the eavesdropper. We derive a non-trivial outer bound on the secrecy capacity region of this channel which, in absence of security constraints, reduces to the best known outer bound to the capacity of the standard BC. An inner bound is also derived which follows the behavior of both the best known inner bound for the BC and the Wiretap Channel. These bounds are show…
This paper considers the transmission of confidential messages over noisy wireless ad hoc networks, where both background noise and interference from concurrent transmitters affect the received signals. For the random networks where the legitimate nodes and the eavesdroppers are distributed as Poisson point processes, we study the secrecy transmission capacity (STC), as well as the connection outage probability and secrecy outage probability, based on the physical layer security. We first consider the basic fixed transmission distance model, and establish a theoretical model of the STC. We the…
For any binary-input channel with perfect state information at the decoder, if the mutual information between the noisy state observation at the encoder and the true channel state is below a positive threshold determined solely by the state distribution, then the capacity is the same as that with no encoder side information. A complementary phenomenon is revealed for the generalized probing capacity. Extensions beyond binary-input channels are developed.
This letter investigates the role of index coding in the capacity of AWGN broadcast channels with receiver message side information. We first show that index coding is unnecessary where there are two receivers; multiplexing coding and superposition coding are sufficient to achieve the capacity region. We next show that, for more than two receivers, multiplexing coding and superposition coding alone can be suboptimal. We give an example where these two coding schemes alone cannot achieve the capacity region, but adding index coding can. This demonstrates that, in contrast to the two-receiver ca…
Decoding via sequentially guessing the error pattern in a received noisy sequence has received attention recently, and ORBGRAND has been proposed as one such decoding algorithm that is capable of utilizing the soft information embedded in the received noisy sequence. An information theoretic study is conducted for ORBGRAND, and it is shown that the achievable rate of ORBGRAND using independent and identically distributed random codebooks almost coincides with the channel capacity, for an additive white Gaussian noise channel under antipodal input. For finite-length codes, improved guessing sch…
We study a special class of diamond channels which was introduced by Schein in 2001. In this special class, each diamond channel consists of a transmitter, a noisy relay, a noiseless relay and a receiver. We prove the capacity of this class of diamond channels by providing an achievable scheme and a converse. The capacity we show is strictly smaller than the cut-set bound. Our result also shows the optimality of a combination of decode-and-forward (DAF) and compress-and-forward (CAF) at the noisy relay node. This is the first example where a combination of DAF and CAF is shown to be capacity a…
In this paper, a class of broadcast interference channels (BIC) is investigated, where one of the two broadcast receivers is subject to interference coming from a point-to-point transmission. For a general discrete memoryless broadcast interference channel (DM-BIC), an achievable scheme based on message splitting, superposition and binning is proposed and a concise representation of the corresponding achievable rate region R is obtained. Two partial-order broadcast conditions interference-oblivious less noisy and interference-cognizant less noisy are defined, thereby extending the usual less n…
We study the fundamental limits of biometric identification systems with chosen secret from an information theoretic perspective. Ignatenko and Willems (2015) characterized the capacity region of identification, secrecy, and privacy-leakage rates of the system provided that the enrollment channel is noiseless. In the enrollment process, however, it is highly considered that noise occurs when bio-data is scanned. Recently, Yachongka and Yagi (2019) characterized the capacity region of the system with generated secret considering noisy enrollment and template constraint. In this paper, we are in…
Consider a pair of terminals connected by two independent additive white Gaussian noise channels, and limited by individual power constraints. The first terminal would like to reliably send information to the second terminal, within a given error probability. We construct an explicit interactive scheme consisting of only (non-linear) scalar operations, by endowing the Schalkwijk-Kailath noiseless feedback scheme with modulo arithmetic. Our scheme achieves a communication rate close to the Shannon limit, in a small number of rounds. For example, for an error probability of $10^{-6}$, if the Sig…
