In this contribution, the Gaussian channel is represented as an equivalent thermal system allowing to express its input-output mutual information in terms of thermodynamic quantities. This thermodynamic description of the mutual information is based upon a generalization of the $2^{nd}$ thermodynamic law and provides an alternative proof to the Guo-Shamai-Verdú theorem, giving an intriguing connection between this remarkable theorem and the most fundamental laws of nature - the laws of thermodynamics.
Based on Planck's blackbody equation it is argued that a single mode light pulse, with a large number of photons, carries one entropy unit. Similarly, an empty radiation mode carries no entropy. In this case, the calculated entropy that a coded sequence of light pulses is carrying is simply the Gibbs mixing entropy, which is identical to the logical Shannon information. This approach is supported by a demonstration that information transmission and amplification, by a sequence of light pulses in an optical fiber, is a classic Carnot machine comprising of two isothermals and two adiabatic. Ther…
Similar to the energy flowing process in traditional heat engines, information could be considered to flow in the communication systems with the form of energy and entropy. Combining the thermodynamic Carnot machine and the classical Shannon information theory, a generalized thermodynamic MIMO (multiple input multiple outputs) communication system is established to analyze the channel capacity using forward error correction codes. Based on the concepts of freedom and entropy in the communication system, the generalized channel capacity is proposed under the thermodynamic theory. Furthermore, t…
We discuss why MMSE estimation arises in lattice-based schemes for approaching the capacity of linear Gaussian channels, and comment on its properties.
We continue to discuss why MMSE estimation arises in coding schemes that approach the capacity of linear Gaussian channels. Here we consider schemes that involve successive decoding, such as decision-feedback equalization or successive cancellation.
The Gray-Wyner network subject to a fidelity criterion is studied. Upper and lower bounds for the trade-offs between the private sum-rate and the common rate are obtained for arbitrary sources subject to mean-squared error distortion. The bounds meet exactly, leading to the computation of the rate region, when the source is jointly Gaussian. They meet partially when the sources are modeled via an additive Gaussian "channel". The bounds are inspired from the Shannon bounds on the rate-distortion problem.
We propose a novel tensor-based formalism for inferring causal structures from time series. An information theoretical analysis of transfer entropy, shows that transfer entropy results from transmission of information over a set of communication channels. Tensors are the mathematical equivalents of these multi-channel causal channels. A multi-channel causal channel is a generalization of a discrete memoryless channel. Investigation of a system comprising three variables shows that in our formalism, bivariate analysis suffices to differentiate between direct and indirect relations. For this to …
Using ideas from Chu and Bode/Fano theories, we characterize the maximum achievable rate over the single-input single-output wireless communication channels under a restriction on the antenna size at the receiver. By employing circuit-theoretic multiport models for radio communication systems, we derive the information-theoretic limits of compact antennas. We first describe an equivalent Chu's antenna circuit under the physical realizability conditions of its reflection coefficient. Such a design allows us to subsequently compute the achievable rate for a given receive antenna size thereby pro…
Transfer entropy is capable of capturing nonlinear source-destination relations between multi-variate time series. It is a measure of association between source data that are transformed into destination data via a set of linear transformations between their probability mass functions. The resulting tensor formalism is used to show that in specific cases, e.g., in the case the system consists of three stochastic processes, bivariate analysis suffices to distinguish true relations from false relations. This allows us to determine the causal structure as far as encoded in the probability mass fu…
The interference channel is the simplest communication scenario where multiple autonomous users compete for shared resources. We combine game theory and information theory to define a notion of a Nash equilibrium region of the interference channel. The notion is game theoretic: it captures the selfish behavior of each user as they compete. The notion is also information theoretic: it allows each user to use arbitrary communication strategies as it optimizes its own performance. We give an exact characterization of the Nash equilibrium region of the two-user linear deterministic interference ch…
We explore two fundamental questions at the intersection of sampling theory and information theory: how channel capacity is affected by sampling below the channel's Nyquist rate, and what sub-Nyquist sampling strategy should be employed to maximize capacity. In particular, we derive the capacity of sampled analog channels for three prevalent sampling strategies: sampling with filtering, sampling with filter banks, and sampling with modulation and filter banks. These sampling mechanisms subsume most nonuniform sampling techniques applied in practice. Our analyses illuminate interesting connecti…
A unification of thermodynamics and information theory is proposed. It is argued that similarly to the randomness due to collisions in thermal systems, the quenched randomness that exists in data files in informatics systems contributes to entropy. Therefore, it is possible to define equilibrium and to calculate temperature for informatics systems. The obtained temperature yields correctly the Shannon information balance in informatics systems and is consistent with the Clausius inequality and the Carnot cycle.
Let $W$ be a binary-input memoryless symmetric (BMS) channel with Shannon capacity $I(W)$ and fix any $α> 0$. We construct, for any sufficiently small $δ> 0$, binary linear codes of block length $O(1/δ^{2+α})$ and rate $I(W)-δ$ that enable reliable communication on $W$ with quasi-linear time encoding and decoding. Shannon's noisy coding theorem established the \emph{existence} of such codes (without efficient constructions or decoding) with block length $O(1/δ^2)$. This quadratic dependence on the gap $δ$ to capacity is known to be best possible. Our result thus yields a constructive version o…
In this paper, we characterize the information-theoretic capacity scaling of wireless ad hoc networks with $n$ randomly distributed nodes. By using an exact channel model from Maxwell's equations, we successfully resolve the conflict in the literature between the linear capacity scaling by Özgür et al. and the degrees of freedom limit given as the ratio of the network diameter and the wavelength $λ$ by Franceschetti et al. In dense networks where the network area is fixed, the capacity scaling is given as the minimum of $n$ and the degrees of freedom limit $λ^{-1}$ to within an arbitrarily sma…
We consider the problem of communicating the state of a dynamical system via a Shannon Gaussian channel. The receiver, which acts as both a decoder and estimator, observes the noisy measurement of the channel output and makes an optimal estimate of the state of the dynamical system in the minimum mean square sense. The transmitter observes a possibly noisy measurement of the state of the dynamical system. These measurements are then used to encode the message to be transmitted over a noisy Gaussian channel, where a per sample power constraint is imposed on the transmitted message. Thus, we get…
We study a setting where a receiver must design a questionnaire to recover a sequence of symbols known to strategic sender, whose utility may not be incentive compatible. We allow the receiver the possibility of selecting the alternatives presented in the questionnaire, and thereby linking decisions across the components of the sequence. We show that, despite the strategic sender and the noise in the channel, the receiver can recover exponentially many sequences, but also that exponentially many sequences are unrecoverable even by the best strategy. We define the growth rate of the number of r…
The capacity of finite state channels (FSCs) has been established as the limit of a sequence of multi-letter expressions only and, despite tremendous effort, a corresponding finite-letter characterization remains unknown to date. This paper analyzes the capacity of FSCs from a fundamental, algorithmic point of view by studying whether or not the corresponding achievability and converse bounds on the capacity can be computed algorithmically. For this purpose, the concept of Turing machines is used which provide the fundamental performance limits of digital computers. To this end, computable con…
This paper introduces a novel framework and corresponding methods for sampling and reconstruction of sparse signals in shift-invariant (SI) spaces. We reinterpret the random demodulator, a system that acquires sparse bandlimited signals, as a system for the acquisition of linear combinations of the samples in the SI setting with the box function as the sampling kernel. The sparsity assumption is exploited by the compressive sensing (CS) paradigm for a recovery of the SI samples from a reduced set of measurements. The SI samples are subsequently filtered by a discrete-time correction filter to …
We study conditional linear information inequalities, i.e., linear inequalities for Shannon entropy that hold for distributions whose entropies meet some linear constraints. We prove that some conditional information inequalities cannot be extended to any unconditional linear inequalities. Some of these conditional inequalities hold for almost entropic points, while others do not. We also discuss some counterparts of conditional information inequalities for Kolmogorov complexity.
The locking effect is a phenomenon which is unique to quantum information theory and represents one of the strongest separations between the classical and quantum theories of information. The Fawzi-Hayden-Sen (FHS) locking protocol harnesses this effect in a cryptographic context, whereby one party can encode n bits into n qubits while using only a constant-size secret key. The encoded message is then secure against any measurement that an eavesdropper could perform in an attempt to recover the message, but the protocol does not necessarily meet the composability requirements needed in quantum…
