We consider the processing of statistical samples $X\sim P_θ$ by a channel $p(y|x)$, and characterize how the statistical information from the samples for estimating the parameter $θ\in\mathbb{R}^d$ can scale with the mutual information or capacity of the channel. We show that if the statistical model has a sub-Gaussian score function, then the trace of the Fisher information matrix for estimating $θ$ from $Y$ can scale at most linearly with the mutual information between $X$ and $Y$. We apply this result to obtain minimax lower bounds in distributed statistical estimation problems, and obtain a tight preconstant for Gaussian mean estimation. We then show how our Fisher information bound can also imply mutual information or Jensen-Shannon divergence based distributed strong data processing inequalities.
I present several new relations between mutual information (MI) and statistical estimation error for a system that can be regarded simultaneously as a communication channel and as an estimator of an input parameter. I first derive a second-order result between MI and Fisher information (FI) that is valid for sufficiently narrow priors, but arbitrary channels. A second relation furnishes a lower bound on the MI in terms of the minimum mean-squared error (MMSE) on the Bayesian estimation of the input parameter from the channel output, one that is valid for arbitrary channels and priors. The exis…
The universality for the local spiked eigenvalues is a powerful tool to deal with the problems of the asymptotic law for the bulks of spiked eigenvalues of high-dimensional generalized Fisher matrices. In this paper, we focus on a more generalized spiked Fisher matrix, where $Σ_1Σ_2^{-1}$ is free of the restriction of diagonal independence, and both of the spiked eigenvalues and the population 4th moments are not necessary required to be bounded. By reducing the matching four moments constraint to a tail probability, we propose a Generalized Four Moment Theorem (G4MT) for the bulks of spiked e…
We introduce a method---called Fisher exact scanning (FES)---for testing and identifying variable dependency that generalizes Fisher's exact test on $2\times 2$ contingency tables to $R\times C$ contingency tables and continuous sample spaces. FES proceeds through scanning over the sample space using windows in the form of $2\times 2$ tables of various sizes, and on each window completing a Fisher's exact test. Based on a factorization of Fisher's multivariate hypergeometric (MHG) likelihood into the product of the univariate hypergeometric likelihoods, we show that there exists a coarse-to-fi…
The Nile problem by Ronald Fisher may be interpreted as the problem of making statistical inference for a special curved exponential family when the minimal sufficient statistic is incomplete. The problem itself and its versions for general curved exponential families pose a mathematical-statistical challenge: studying the subalgebras of ancillary statistics within the $σ$-algebra of the (incomplete) minimal sufficient statistics and closely related questions of the structure of UMVUEs. In this paper a new method is developed that, in particular, proves that in the classical Nile problem no st…
Two new proofs of the Fisher information inequality (FII) using data processing inequalities for mutual information and conditional variance are presented.
Although Shannon mutual information has been widely used, its effective calculation is often difficult for many practical problems, including those in neural population coding. Asymptotic formulas based on Fisher information sometimes provide accurate approximations to the mutual information but this approach is restricted to continuous variables because the calculation of Fisher information requires derivatives with respect to the encoded variables. In this paper, we consider information-theoretic bounds and approximations of the mutual information based on Kullback--Leibler divergence and Ré…
Within the framework of linear vector Gaussian channels with arbitrary signaling, closed-form expressions for the Jacobian of the minimum mean square error and Fisher information matrices with respect to arbitrary parameters of the system are calculated in this paper. Capitalizing on prior research where the minimum mean square error and Fisher information matrices were linked to information-theoretic quantities through differentiation, closed-form expressions for the Hessian of the mutual information and the differential entropy are derived. These expressions are then used to assess the conca…
We study the convexity of mutual information along the evolution of the heat equation. We prove that if the initial distribution is log-concave, then mutual information is always a convex function of time. We also prove that if the initial distribution is either bounded, or has finite fourth moment and Fisher information, then mutual information is eventually convex, i.e., convex for all large time. Finally, we provide counterexamples to show that mutual information can be nonconvex at small time.
Recently proposed methods in data subset selection, that is active learning and active sampling, use Fisher information, Hessians, similarity matrices based on gradients, and gradient lengths to estimate how informative data is for a model's training. Are these different approaches connected, and if so, how? We revisit the fundamentals of Bayesian optimal experiment design and show that these recently proposed methods can be understood as approximations to information-theoretic quantities: among them, the mutual information between predictions and model parameters, known as expected informatio…
This paper considers the model of an arbitrary distributed signal x observed through an added independent white Gaussian noise w, y=x+w. New relations between the minimal mean square error of the non-causal estimator and the likelihood ratio between y and ωare derived. This is followed by an extended version of a recently derived relation between the mutual information I(x;y) and the minimal mean square error. These results are applied to derive infinite dimensional versions of the Fisher information and the de Bruijn identity. The derivation of the results is based on the Malliavin calculus.
We consider the problem of learning high-dimensional, nonparametric and structured (e.g. Gaussian) distributions in distributed networks, where each node in the network observes an independent sample from the underlying distribution and can use $k$ bits to communicate its sample to a central processor. We consider three different models for communication. Under the independent model, each node communicates its sample to a central processor by independently encoding it into $k$ bits. Under the more general sequential or blackboard communication models, nodes can share information interactively …
We study the convexity of mutual information as a function of time along the flow of the Ornstein-Uhlenbeck process. We prove that if the initial distribution is strongly log-concave, then mutual information is eventually convex, i.e., convex for all large time. In particular, if the initial distribution is sufficiently strongly log-concave compared to the target Gaussian measure, then mutual information is always a convex function of time. We also prove that if the initial distribution is either bounded or has finite fourth moment and Fisher information, then mutual information is eventually …
It is proved that in a non-Bayesian parametric estimation problem, if the Fisher information matrix (FIM) is singular, unbiased estimators for the unknown parameter will not exist. Cramer-Rao bound (CRB), a popular tool to lower bound the variances of unbiased estimators, seems inapplicable in such situations. In this paper, we show that the Moore-Penrose generalized inverse of a singular FIM can be interpreted as the CRB corresponding to the minimum variance among all choices of minimum constraint functions. This result ensures the logical validity of applying the Moore-Penrose generalized in…
While most useful information theoretic inequalities can be deduced from the basic properties of entropy or mutual information, Shannon's entropy power inequality (EPI) seems to be an exception: available information theoretic proofs of the EPI hinge on integral representations of differential entropy using either Fisher's information (FI) or minimum mean-square error (MMSE). In this paper, we first present a unified view of proofs via FI and MMSE, showing that they are essentially dual versions of the same proof, and then fill the gap by providing a new, simple proof of the EPI, which is sole…
We study the relationship between information- and estimation-theoretic quantities in time-evolving systems. We focus on the Fokker-Planck channel defined by a general stochastic differential equation, and show that the time derivatives of entropy, KL divergence, and mutual information are characterized by estimation-theoretic quantities involving an appropriate generalization of the Fisher information. Our results vastly extend De Bruijn's identity and the classical I-MMSE relation.
We develop data processing inequalities that describe how Fisher information from statistical samples can scale with the privacy parameter $\varepsilon$ under local differential privacy constraints. These bounds are valid under general conditions on the distribution of the score of the statistical model, and they elucidate under which conditions the dependence on $\varepsilon$ is linear, quadratic, or exponential. We show how these inequalities imply order optimal lower bounds for private estimation for both the Gaussian location model and discrete distribution estimation for all levels of pri…
In this paper, we introduce new Stein identities for gamma target distribution as well as a new non-linear channel specifically designed for gamma inputs. From these two ingredients, we derive an explicit and simple formula for the derivative of the input-output mutual information of this non-linear channel with respect to the channel quality parameter. This relation is reminiscent of the well-known link between the derivative of the input-output mutual information of additive Gaussian noise channel with respect to the signal-to-noise ratio and the minimum mean-square error. The proof relies o…
While most useful information theoretic inequalities can be deduced from the basic properties of entropy or mutual information, up to now Shannon's entropy power inequality (EPI) is an exception: Existing information theoretic proofs of the EPI hinge on representations of differential entropy using either Fisher information or minimum mean-square error (MMSE), which are derived from de Bruijn's identity. In this paper, we first present an unified view of these proofs, showing that they share two essential ingredients: 1) a data processing argument applied to a covariance-preserving linear tran…
We investigate the benefits of mutual coupling effects between the passive elements of intelligent reconfigurable surfaces (IRSs) on maximizing the achievable rate of downlink Internet-of-Things (IoT) networks. In this paper, we present an electromagnetic (EM) coupling model for IRSs whose elements are connected minimum scattering antennas (i.e., dipoles). Using Chu's theory, we incorporate the finite antenna size constraint on each element of the IRS to obtain the IRS mutual impedance matrix. By maximizing the IRS phase shiters using the gradient ascent procedure, our numerical results show t…
We present two extended forms of Fisher information that fit well in the context of nonextensive thermostatistics. We show that there exists an interplay between these generalized Fisher information, the generalized $q$-Gaussian distributions and the $q$-entropies. The minimum of the generalized Fisher information among distributions with a fixed moment, or with a fixed $q$-entropy is attained, in both cases, by a generalized $q$-Gaussian distribution. This complements the fact that the $q$-Gaussians maximize the $q$-entropies subject to a moment constraint, and yields new variational characte…
Differential privacy is a precise mathematical constraint meant to ensure privacy of individual pieces of information in a database even while queries are being answered about the aggregate. Intuitively, one must come to terms with what differential privacy does and does not guarantee. For example, the definition prevents a strong adversary who knows all but one entry in the database from further inferring about the last one. This strong adversary assumption can be overlooked, resulting in misinterpretation of the privacy guarantee of differential privacy. Herein we give an equivalent definiti…
Single-user multiple-input / multiple-output (SU-MIMO) communication systems have been successfully used over the years and have provided a significant increase on a wireless link's capacity by enabling the transmission of multiple data streams. Assuming channel knowledge at the transmitter, the maximization of the mutual information of a MIMO link is achieved by finding the optimal power allocation under a given sum-power constraint, which is in turn obtained by the water-filling (WF) algorithm. However, in spectrum sharing setups, such as Licensed Shared Access (LSA), where a primary link (P…
In the present paper, we would like to draw attention to a possible generalized Fisher information that fits well in the formalism of nonextensive thermostatistics. This generalized Fisher information is defined for densities on $\mathbb{R}^{n}.$ Just as the maximum Rényi or Tsallis entropy subject to an elliptic moment constraint is a generalized q-Gaussian, we show that the minimization of the generalized Fisher information also leads a generalized q-Gaussian. This yields a generalized Cramér-Rao inequality. In addition, we show that the generalized Fisher information naturally pops up in a …
Category learning in deep neural networks: Information content and geometry of internal representations
Centre d’Analyse et de Mathématique Sociales (CAMS) EHESS, CNRS École des Hautes Études en Sciences Sociales（分析与数学社会中心（CAMS）EHESS，CNRS高等社会科学学院）；Laboratoire de Physique de l’École Normale Supérieure (LPENS), ENS, Université PSL, CNRS, Sorbonne Université, Université Paris Cité École Normale Supérieure（高等师范物理实验室（LPENS），ENS，PSL大学，CNRS，索邦大学，巴黎城市大学高等师范学院）
In this communication, we describe some interrelations between generalized $q$-entropies and a generalized version of Fisher information. In information theory, the de Bruijn identity links the Fisher information and the derivative of the entropy. We show that this identity can be extended to generalized versions of entropy and Fisher information. More precisely, a generalized Fisher information naturally pops up in the expression of the derivative of the Tsallis entropy. This generalized Fisher information also appears as a special case of a generalized Fisher information for estimation probl…
A Mutual Information Lower Bound for Multimodal Regression Active Learning
Graduate Group on Applied Mathematics & Computational Science（应用数学与计算科学研究生组）；Department of Computer and Information Science（计算机与信息科学系）；University of Pennsylvania（宾夕法尼亚大学）；Department of Mechanical Engineering and Applied Mechanics（机械工程与应用力学系）
In this paper, we consider the joint design of the transceivers for a multiple access Multiple Input and Multiple Output (MIMO) system having Inter-Symbol Interference (ISI) channels. The system we consider is equipped with the Minimum Mean Square Error (MMSE) Decision-Feedback (DF) detector. Traditionally, transmitter designs for this system have been based on constraints of either the transmission power or the signal-to-interference-and-noise ratio (SINR) for each user. Here, we explore a novel perspective and examine a transceiver design which is based on a fixed sum mutual information cons…
The performance of the generalized belief propagation algorithm for computing the noiseless capacity and mutual information rates of finite-size two-dimensional and three-dimensional run-length limited constraints is investigated. For each constraint, a method is proposed to choose the basic regions and to construct the region graph. Simulation results for the capacity of different constraints as a function of the size of the channel and mutual information rates of different constraints as a function of signal-to-noise ratio are reported. Convergence to the Shannon capacity is also discussed.
A robust prediction model invoking the Takens embedding theorem, whose \textit{working hypothesis} is obtained via an inference procedure based on the minimum Fisher information principle, is presented. The coefficients of the ansatz, central to the \textit{working hypothesis} satisfy a time independent Schrödinger-like equation in a vector setting. The inference of i) the probability density function of the coefficients of the \textit{working hypothesis} and ii) the establishing of constraint driven pseudo-inverse condition for the modeling phase of the prediction scheme, is made, for the cas…
