In this paper, we investigate the quantization of the output of a binary input discrete memoryless channel that maximizing the mutual information between the input and the quantized output under an entropy-constrained of the quantized output. A polynomial time algorithm is introduced that can find the truly global optimal quantizer. These results hold for binary input channels with an arbitrary number of quantized output. Finally, we extend these results to binary input continuous output channels and show a sufficient condition such that a single threshold quantizer is an optimal quantizer. Both theoretical results and numerical results are provided to justify our techniques.
Recently, low-resolution LDPC decoders have been introduced that perform mutual information maximizing signal processing. However, the optimal quantization in variable and check nodes requires expensive non-uniform operations. Instead, we propose to use uniform quantization with a simple hardware structure, which reduces the complexity of individual node operations approximately by half and shortens the decoding delay significantly. Our analysis shows that the loss of preserved mutual information resulting from restriction to uniform quantization is very small. Furthermore, the error rate simu…
In this paper, we propose a class of finite alphabet iterative decoder (FAID), called mutual information-maximizing quantized belief propagation (MIM-QBP) decoder, for decoding regular low-density parity-check (LDPC) codes. Our decoder follows the reconstruction-calculation-quantization (RCQ) decoding architecture that is widely used in FAIDs. We present the first complete and systematic design framework for the RCQ parameters, and prove that our design with sufficient precision at node update is able to maximize the mutual information between coded bits and exchanged messages. Simulation resu…
ADCs sit at the interface of the analog and digital worlds and fundamentally determine what information is available in the digital domain for processing. This paper shows that a configurable ADC can be designed for signals with non constant information as a function of frequency such that within a fixed power budget the ADC maximizes the information in the converted signal by frequency shaping the quantization noise. Quantization noise shaping can be realized via loop filter design for a single channel delta sigma ADC and extended to common time and frequency interleaved multi channel structu…
We investigate iterative low-resolution message-passing algorithms for quasi-cyclic LDPC codes with horizontal and vertical layered schedules. Coarse quantization and layered scheduling are highly relevant for hardware implementations to reduce the bit width of messages and the number of decoding iterations. As a novelty, this paper compares the two scheduling variants in combination with mutual information maximizing compression operations in variable and check nodes. We evaluate the complexity and error rate performance for various configurations. Dedicated hardware architectures for regular…
In this letter, we propose a two-stage design method to construct memory efficient mutual information-maximizing quantized min-sum (MIM-QMS) decoder for rate-compatible low-density parity-check (LDPC) codes. We first develop a modified density evolution to design a unique set of lookup tables (LUTs) that can be used for rate-compatible LDPC codes. The constructed LUTs are optimized based on their discrepancy values and a merge function to reduce the memory requirement. Numerical results show that the proposed rate-compatible MIM-QMS decoder can reduce the memory requirement for decoding by up …
This paper considers joint beamformer design towards maximizing the mutual information in a coherent wireless sensor network with noisy observation and multiple antennae. Leveraging the weighted minimum mean square error and block coordinate ascent (BCA) framework, we propose two new and efficient methods: batch-mode BCA and cyclic multi-block BCA. The existing batch-mode approaches require stringent conditions such as diagonal channel matrices and positive definite second-order matrices, and are therefore inapplicable to our problem. Our match-mode BCA overcomes the previous limitations via a…
Rutgers University（罗格斯大学）；Toyota Technological Institute at Chicago（芝加哥丰田工业大学）
We propose learning discrete structured representations from unlabeled data by maximizing the mutual information between a structured latent variable and a target variable. Calculating mutual information is intractable in this setting. Our key technical contribution is an adversarial objective that can be used to tractably estimate mutual information assuming only the feasibility of cross entropy calculation. We develop a concrete realization of this general formulation with Markov distributions over binary encodings. We report critical and unexpected findings on practical aspects of the objec…
High-capacity NAND flash memories use multi-level cells (MLCs) to store multiple bits per cell and achieve high storage densities. Higher densities cause increased raw bit error rates (BERs), which demand powerful error correcting codes. Low-density parity-check (LDPC) codes are a well-known class of capacity-approaching codes in AWGN channels. However, LDPC codes traditionally use soft information while the flash read channel provides only hard information. Low resolution soft information may be obtained by performing multiple reads per cell with distinct word-line voltages. We select the val…
We investigate the benefits of mutual coupling effects between the passive elements of intelligent reconfigurable surfaces (IRSs) on maximizing the achievable rate of downlink Internet-of-Things (IoT) networks. In this paper, we present an electromagnetic (EM) coupling model for IRSs whose elements are connected minimum scattering antennas (i.e., dipoles). Using Chu's theory, we incorporate the finite antenna size constraint on each element of the IRS to obtain the IRS mutual impedance matrix. By maximizing the IRS phase shiters using the gradient ascent procedure, our numerical results show t…
The multiterminal secret key agreement problem by public discussion is formulated with an additional source compression step where, prior to the public discussion phase, users independently compress their private sources to filter out strongly correlated components for generating a common secret key. The objective is to maximize the achievable key rate as a function of the joint entropy of the compressed sources. Since the maximum achievable key rate captures the total amount of information mutual to the compressed sources, an optimal compression scheme essentially maximizes the multivariate m…
In this paper, we propose a framework of the mutual information-maximizing (MIM) quantized decoding for low-density parity-check (LDPC) codes by using simple mappings and fixed-point additions. Our decoding method is generic in the sense that it can be applied to LDPC codes with arbitrary degree distributions, and can be implemented based on either the belief propagation (BP) algorithm or the min-sum (MS) algorithm. In particular, we propose the MIM density evolution (MIM-DE) to construct the lookup tables (LUTs) for the node updates. The computational complexity and memory requirements are di…
Single-user multiple-input / multiple-output (SU-MIMO) communication systems have been successfully used over the years and have provided a significant increase on a wireless link's capacity by enabling the transmission of multiple data streams. Assuming channel knowledge at the transmitter, the maximization of the mutual information of a MIMO link is achieved by finding the optimal power allocation under a given sum-power constraint, which is in turn obtained by the water-filling (WF) algorithm. However, in spectrum sharing setups, such as Licensed Shared Access (LSA), where a primary link (P…
In this paper, a new four-dimensional 64-ary polarization ring switching (4D-64PRS) modulation format with a spectral efficiency of 6 bit/4D-sym is introduced. The format is designed by maximizing the generalized mutual information (GMI) and by imposing a constant-modulus on the 4D structure. The proposed format yields an improved performance with respect to state-of-the-art geometrically shaped modulation formats for bit-interleaved coded modulation systems at the same spectral efficiency. Unlike previously published results, the coordinates of the constellation points and the binary labeling…
We investigate the sets of joint probability distributions that maximize the average multi-information over a collection of margins. These functionals serve as proxies for maximizing the multi-information of a set of variables or the mutual information of two subsets of variables, at a lower computation and estimation complexity. We describe the maximizers and their relations to the maximizers of the multi-information and the mutual information.
We consider a channel with a binary input X being corrupted by a continuous-valued noise that results in a continuous-valued output Y. An optimal binary quantizer is used to quantize the continuous-valued output Y to the final binary output Z to maximize the mutual information I(X; Z). We show that when the ratio of the channel conditional density r(y) = P(Y=y|X=0)/ P(Y =y|X=1) is a strictly increasing/decreasing function of y, then a quantizer having a single threshold can maximize mutual information. Furthermore, we show that an optimal quantizer (possibly with multiple thresholds) is the on…
Let $X$ and $Y$ be dependent random variables. This paper considers the problem of designing a scalar quantizer for $Y$ to maximize the mutual information between the quantizer's output and $X$, and develops fundamental properties and bounds for this form of quantization, which is connected to the log-loss distortion criterion. The main focus is the regime of low $I(X;Y)$, where it is shown that, if $X$ is binary, a constant fraction of the mutual information can always be preserved using $\mathcal{O}(\log(1/I(X;Y)))$ quantization levels, and there exist distributions for which this many quant…
We consider a continuous-time bandlimited additive white Gaussian noise channel with 1-bit output quantization. On such a channel the information is carried by the temporal distances of the zero-crossings of the transmit signal. The set of input signals is constrained by the bandwidth of the channel and an average power constraint. We derive a lower bound on the capacity by lower-bounding the mutual information rate for a given set of waveforms with exponentially distributed zero-crossing distances, where we focus on the behavior in the mid to high signal-to-noise ratio regime. We find that in…
We build a natural connection between the learning problem, co-training, and forecast elicitation without verification (related to peer-prediction) and address them simultaneously using the same information theoretic approach. In co-training/multiview learning, the goal is to aggregate two views of data into a prediction for a latent label. We show how to optimally combine two views of data by reducing the problem to an optimization problem. Our work gives a unified and rigorous approach to the general setting. In forecast elicitation without verification we seek to design a mechanism that eli…
The one-bit spatial Sigma-Delta concept has recently been proposed as an approach for achieving low distortion and low power consumption for massive multi-input multi-output systems. The approach exploits users located in known angular sectors or spatial oversampling to shape the quantization noise away from desired directions of arrival. While reducing the antenna spacing alleviates the adverse impact of quantization noise, it can potentially deteriorate the performance of the massive array due to excessive mutual coupling. In this paper, we analyze the impact of mutual coupling on the uplink…
Toyota Technological Institute at Chicago（芝加哥丰田技术研究院）；Rutgers University（罗格斯大学）
Measuring mutual information from finite data is difficult. Recent work has considered variational methods maximizing a lower bound. In this paper, we prove that serious statistical limitations are inherent to any method of measuring mutual information. More specifically, we show that any distribution-free high-confidence lower bound on mutual information estimated from N samples cannot be larger than O(ln N ).
The quantization of the output of a binary-input discrete memoryless channel to a smaller number of levels is considered. An algorithm which finds an optimal quantizer, in the sense of maximizing mutual information between the channel input and the quantizer output is given. This result holds for arbitrary channels, in contrast to previous results for restricted channels or a restricted number of quantizer outputs. In the worst case, the algorithm complexity is cubic $M^3$ in the number of channel outputs $M$. Optimality is proved using the theorem of Burshtein, Della Pietra, Kanevsky, and Nád…
Consider statistical learning (e.g. discrete distribution estimation) with local $ε$-differential privacy, which preserves each data provider's privacy locally, we aim to optimize statistical data utility under the privacy constraints. Specifically, we study maximizing mutual information between a provider's data and its private view, and give the exact mutual information bound along with an attainable mechanism: $k$-subset mechanism as results. The mutual information optimal mechanism randomly outputs a size $k$ subset of the original data domain with delicate probability assignment, where $k…
We consider a model nondispersive nonlinear optical fiber channel with additive white Gaussian noise at large $\mathrm{SNR}$ (signal-to-noise ratio) in the intermediate power region. Using Feynman path-integral technique we for the first time find the optimal input signal distribution maximizing the channel's per-sample mutual information. The finding of the optimal input signal distribution allows us to improve previously known estimates for the channel capacity. The output signal entropy, conditional entropy, and per-sample mutual information are calculated for Gaussian, half-Gaussian and mo…
Multivariate mutual information provides a conceptual framework for characterizing higher-order interactions in complex systems. Two well-known measures of multivariate information---total correlation and dual total correlation---admit a spectrum of measures with varying sensitivity to intermediate orders of dependence. Unfortunately, these intermediate measures have not received much attention due to their opaque representation of information. Here we draw on results from matroid theory to show that these measures are closely related to error-correcting codes. This connection allows us to der…
In this paper, under a general cost function $C$, we present a dynamic programming (DP) method to obtain an optimal sequential deterministic quantizer (SDQ) for $q$-ary input discrete memoryless channel (DMC). The DP method has complexity $O(q (N-M)^2 M)$, where $N$ and $M$ are the alphabet sizes of the DMC output and quantizer output, respectively. Then, starting from the quadrangle inequality, two techniques are applied to reduce the DP method's complexity. One technique makes use of the Shor-Moran-Aggarwal-Wilber-Klawe (SMAWK) algorithm and achieves complexity $O(q (N-M) M)$. The other tech…
This paper is devoted to the design of precoders maximizing the ergodic mutual information (EMI) of bi-correlated flat fading MIMO systems equiped with MMSE receivers. The channel state information and the second order statistics of the channel are assumed available at the receiver side and at the transmitter side respectively. As the direct maximization of the EMI needs the use of non attractive algorithms, it is proposed to optimize an approximation of the EMI, introduced recently, obtained when the number of transmit and receive antennas $t$ and $r$ converge to $\infty$ at the same rate. It…
We describe a novel extension of soft actor-critics for hierarchical Deep Q-Networks (HDQN) architectures using mutual information metric. The proposed extension provides a suitable framework for encouraging explorations in such hierarchical networks. A natural utilization of this framework is an adversarial setting, where meta-controller and controller play minimax over the mutual information objective but cooperate on maximizing expected rewards.
Wireless positioning systems that are implemented by means of a Cloud Radio Access Networks (C-RANs) may provide cost-effective solutions, particularly for indoor localization. In a C-RAN, the baseband processing, including localization, is carried out at a centralized control unit (CU) based on quantized baseband signals received from the RUs over finite-capacity fronthaul links. In this paper, the problem of maximizing the localization accuracy over fronthaul quantization/compression is formulated by adopting the Cramér-Rao bound (CRB) on the localization accuracy as the performance metric o…
This paper investigates the application of low-density parity-check (LDPC) codes to Flash memories. Multiple cell reads with distinct word-line voltages provide limited-precision soft information for the LDPC decoder. The values of the word-line voltages (also called reference voltages) are optimized by maximizing the mutual information (MI) between the input and output of the multiple-read channel. Constraining the maximum mutual-information (MMI) quantization to enforce a constant-ratio constraint provides a significant simplification with no noticeable loss in performance. Our simulation re…
Quantized massive multiple-input-multiple-output (MIMO) systems are gaining more interest due to their power efficiency. We present a new precoding technique to mitigate the multi-user interference and the quantization distortions in a downlink multi-user (MU) multiple-input-single-output (MISO) system with 1-bit quantization at the transmitter. This work is restricted to PSK modulation schemes. The transmit signal vector is optimized for every desired received vector taking into account the 1-bit quantization. The optimization is based on maximizing the safety margin to the decision threshold…
In phase retrieval problems, a signal of interest (SOI) is reconstructed based on the magnitude of a linear transformation of the SOI observed with additive noise. The linear transform is typically referred to as a measurement matrix. Many works on phase retrieval assume that the measurement matrix is a random Gaussian matrix, which, in the noiseless scenario with sufficiently many measurements, guarantees invertability of the transformation between the SOI and the observations, up to an inherent phase ambiguity. However, in many practical applications, the measurement matrix corresponds to an…
Training machine learning models that are robust against adversarial inputs poses seemingly insurmountable challenges. To better understand adversarial robustness, we consider the underlying problem of learning robust representations. We develop a notion of representation vulnerability that captures the maximum change of mutual information between the input and output distributions, under the worst-case input perturbation. Then, we prove a theorem that establishes a lower bound on the minimum adversarial risk that can be achieved for any downstream classifier based on its representation vulner…
1 Department of Electrical Engineering；Computer Science, MIT, Cambridge, MA, USA 2 Department of Aeronautics；Astronautics, MIT, Cambridge, MA, USA
Exploration tasks are embedded in many robotics applications, such as search and rescue and space exploration. Information-based exploration algorithms aim to find the most informative trajectories by maximizing an information-theoretic metric, such as the mutual information between the map and potential future measurements. Unfortunately, most existing information-based exploration algorithms are plagued by the computational difficulty of evaluating the Shannon mutual information metric. In this paper, we consider the fundamental problem of evaluating Shannon mutual information between the ma…
We consider a channel with discrete binary input X that is corrupted by a given continuous noise to produce a continuous-valued output Y. A quantizer is then used to quantize the continuous-valued output Y to the final binary output Z. The goal is to design an optimal quantizer Q* and also find the optimal input distribution p*(X) that maximizes the mutual information I(X; Z) between the binary input and the binary quantized output. A linear time complexity searching procedure is proposed. Based on the properties of the optimal quantizer and the optimal input distribution, we reduced the searc…
We present a two-stage approach for learning dictionaries for object classification tasks based on the principle of information maximization. The proposed method seeks a dictionary that is compact, discriminative, and generative. In the first stage, dictionary atoms are selected from an initial dictionary by maximizing the mutual information measure on dictionary compactness, discrimination and reconstruction. In the second stage, the selected dictionary atoms are updated for improved reconstructive and discriminative power using a simple gradient ascent algorithm on mutual information. Experi…
In this paper, we propose a novel pooling layer for graph neural networks based on maximizing the mutual information between the pooled graph and the input graph. Since the maximum mutual information is difficult to compute, we employ the Shannon capacity of a graph as an inductive bias to our pooling method. More precisely, we show that the input graph to the pooling layer can be viewed as a representation of a noisy communication channel. For such a channel, sending the symbols belonging to an independent set of the graph yields a reliable and error-free transmission of information. We show …
We propose a new method for learning image attention masks in a semi-supervised setting based on the Information Bottleneck principle. Provided with a set of labeled images, the mask generation model is minimizing mutual information between the input and the masked image while maximizing the mutual information between the same masked image and the image label. In contrast with other approaches, our attention model produces a Boolean rather than a continuous mask, entirely concealing the information in masked-out pixels. Using a set of synthetic datasets based on MNIST and CIFAR10 and the SVHN …
Graph representation learning based on graph neural networks (GNNs) can greatly improve the performance of downstream tasks, such as node and graph classification. However, the general GNN models do not aggregate node information in a hierarchical manner, and can miss key higher-order structural features of many graphs. The hierarchical aggregation also enables the graph representations to be explainable. In addition, supervised graph representation learning requires labeled data, which is expensive and error-prone. To address these issues, we present an unsupervised graph representation learn…
We use the language of uninformative Bayesian prior choice to study the selection of appropriately simple effective models. We advocate for the prior which maximizes the mutual information between parameters and predictions, learning as much as possible from limited data. When many parameters are poorly constrained by the available data, we find that this prior puts weight only on boundaries of the parameter manifold. Thus it selects a lower-dimensional effective theory in a principled way, ignoring irrelevant parameter directions. In the limit where there is sufficient data to tightly constra…
A privacy-constrained information extraction problem is considered where for a pair of correlated discrete random variables $(X,Y)$ governed by a given joint distribution, an agent observes $Y$ and wants to convey to a potentially public user as much information about $Y$ as possible without compromising the amount of information revealed about $X$. To this end, the so-called {\em rate-privacy function} is introduced to quantify the maximal amount of information (measured in terms of mutual information) that can be extracted from $Y$ under a privacy constraint between $X$ and the extracted inf…
