Estimating the entropy of a discrete random variable is a fundamental problem in information theory and related fields. This problem has many applications in various domains, including machine learning, statistics and data compression. Over the years, a variety of estimation schemes have been suggested. However, despite significant progress, most methods still struggle when the sample is small, compared to the variable's alphabet size. In this work, we introduce a practical solution to this problem, which extends the work of McAllester and Statos (2020). The proposed scheme uses the generalization abilities of cross-entropy estimation in deep neural networks (DNNs) to introduce improved entropy estimation accuracy. Furthermore, we introduce a family of estimators for related information-theoretic measures, such as conditional entropy and mutual information. We show that these estimators are strongly consistent and demonstrate their performance in a variety of use-cases. First, we consider large alphabet entropy estimation. Then, we extend the scope to mutual information estimation. Next, we apply the proposed scheme to conditional mutual information estimation, as we focus on independence testing tasks. Finally, we study a transfer entropy estimation problem. The proposed estimators demonstrate improved performance compared to existing methods in all tested setups.
This paper focuses on the extreme-value problem for Shannon entropy of the joint distribution with given marginals. It is proved that the minimum-entropy coupling must be of order-preserving, while the maximum-entropy coupling coincides with the independent one. Note that in this sense, we interpret entropy as a measure of system disorder.
This note provides a succinct proof of a 1973 theorem of Lieb that establishes the concavity of a certain trace function. The development relies on a deep result from quantum information theory, the joint convexity of quantum relative entropy, as well as a recent argument due to Carlen and Lieb.
Department of Electrical Engineering（电气工程系）；Information and Decision System Group（信息与决策系统小组）；University of Chile（智利大学）
The problem of Shannon entropy estimation in countable infinite alphabets is addressed from the study and use of convergence results of the entropy functional, which is known to be discontinuous with respect to the total variation distance in $\infty$-alphabets. Sufficient conditions for the convergence of the entropy are used, including scenarios with both finitely and infinitely supported assumptions on the distributions. From this new perspective, four plug-in histogram-based estimators are studied showing that convergence results are instrumental to derive new strong consistency and rate o…
We show that various formulations (e.g., dual and Kullback-Csiszar iterations) of estimation of maximum entropy (ME) models can be transformed to solving systems of polynomial equations in several variables for which one can use celebrated Grobner bases methods. Posing of ME estimation as solving polynomial equations is possible, in the cases where feature functions (sufficient statistic) that provides the information about the underlying random variable in the form of expectations are integer valued.
The existing upper and lower bounds between entropy and error are mostly derived through an inequality means without linking to joint distributions. In fact, from either theoretical or application viewpoint, there exists a need to achieve a complete set of interpretations to the bounds in relation to joint distributions. For this reason, in this work we propose a new approach of deriving the bounds between entropy and error from a joint distribution. The specific case study is given on binary classifications, which can justify the need of the proposed approach. Two basic types of classificatio…
We consider a pair of correlated processes {Z_n} and {S_n} (two sided), where the former is observable and the later is hidden. The uncertainty in the estimation of Z_n upon its finite past history is H(Z_n|Z_0^{n-1}), and for estimation of S_n upon this observation is H(S_n|Z_0^{n-1}), which are both sequences of n. The limits of these sequences (and their existence) are of practical and theoretical interest. The first limit, if exists, is the entropy rate. We call the second limit the estimation entropy. An example of a process jointly correlated to another one is the hidden Markov process. …
The min-entropy is a widely used metric to quantify the randomness of generated random numbers in cryptographic applications; it measures the difficulty of guessing the most likely output. An important min-entropy estimator is the compression estimator of NIST Special Publication (SP) 800-90B, which relies on Maurer's universal test. In this paper, we propose two kinds of min-entropy estimators to improve computational complexity and estimation accuracy by leveraging two variations of Maurer's test: Coron's test (for Shannon entropy) and Kim's test (for Renyi entropy). First, we propose a min-…
Low complexity joint estimation of synchronization impairments and channel in a single-user MIMO-OFDM system is presented in this letter. Based on a system model that takes into account the effects of synchronization impairments such as carrier frequency offset, sampling frequency offset, and symbol timing error, and channel, a Maximum Likelihood (ML) algorithm for the joint estimation is proposed. To reduce the complexity of ML grid search, the number of received signal samples used for estimation need to be reduced. The conventional channel estimation methods using Least-Squares (LS) fail fo…
A joint communication and channel state estimation problem is investigated, in which reliable information transmission over a noisy channel, and high-fidelity estimation of the channel state, are simultaneously sought. The tradeoff between the achievable information rate and the estimation distortion is quantified by formulating the problem as a constrained channel coding problem, and the resulting capacity-distortion function characterizes the fundamental limit of the joint communication and channel estimation problem. The analytical results are illustrated through case studies, and further i…
Joint estimation of carrier frequency and direction of arrival (DOA) for multiple signals has been found in many practical applications such as Cognitive Radio (CR). However, Nyquist sampling mechanism is costly or implemented due to wide spectrum range. Taking advantage of sub-Nyquist sampling technology, some array receiver architectures are proposed to realize joint estimation of carrier frequency and DOA. To further decrease equivalent sampling rate and hardware complexity, we propose a simplifying receiver architecture based on our previous work. We come up with joint DOA and frequency es…
In this work, a new Bayesian framework for OFDM channel estimation is proposed. Using Jaynes' maximum entropy principle to derive prior information, we successively tackle the situations when only the channel delay spread is a priori known, then when it is not known. Exploitation of the time-frequency dimensions are also considered in this framework, to derive the optimal channel estimation associated to some performance measure under any state of knowledge. Simulations corroborate the optimality claim and always prove as good or better in performance than classical estimators.
Boston University（波士顿大学）；Northeastern University（东北大学）；University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）；UC Berkeley（加州大学伯克利分校）；Stanford University（斯坦福大学）
Recent work of Acharya et al. (NeurIPS 2019) showed how to estimate the entropy of a distribution $\mathcal D$ over an alphabet of size $k$ up to $\pmε$ additive error by streaming over $(k/ε^3) \cdot \text{polylog}(1/ε)$ i.i.d. samples and using only $O(1)$ words of memory. In this work, we give a new constant memory scheme that reduces the sample complexity to $(k/ε^2)\cdot \text{polylog}(1/ε)$. We conjecture that this is optimal up to $\text{polylog}(1/ε)$ factors.
Accurate indoor localization has long been a challenging problem due to the presence of multipath. Joint direction-of-arrival (DOA) and time delay (TD) estimation is a promising technique for accurate indoor Localization in next generation WiFi and 5G, as it has the capability of separating the line-of-sight (LOS) signal from multipath signals in the TD space. Although the benefit of joint DOA and TD estimation over DOA-only estimation has been empirically shown long ago, it has not been theoretically justified yet. In this paper, we provide a theoretical proof of the benefit of joint DOA and …
The design and analysis of diffusive molecular communication systems generally requires knowledge of the environment's physical and chemical properties. Furthermore, prospective applications might rely on the timely detection of changes in the local system parameters. This paper studies the local estimation of channel parameters for diffusive molecular communication when a transmitter releases molecules that are observed by a receiver. The Fisher information matrix of the joint parameter estimation problem is derived so that the Cramer-Rao lower bound on the variance of locally unbiased estima…
In this paper, we propose a joint pilot design and channel estimation scheme based on the deep learning (DL) technique for multiuser multiple-input multiple output (MIMO) channels. To this end, we construct a pilot designer using two-layer neural networks (TNNs) and a channel estimator using deep neural networks (DNNs), which are jointly trained to minimize the mean square error (MSE) of channel estimation. To effectively reduce the interference among the multiple users, we also use the successive interference cancellation (SIC) technique in the channel estimation process. The numerical result…
The joint convexity of the map $(X,A) \mapsto X^* A^{-1} X$, an integral representation of operator convex functions, and an observation of Ando are used to obtain a simple proof of both the joint convexity of relative entropy and a trace convexity result of Lieb. The latter was the key ingredient in the original proof of the strong subadditivity of quantum entropy.
The minimum error entropy (MEE) criterion has been successfully used in fields such as parameter estimation, system identification and the supervised machine learning. There is in general no explicit expression for the optimal MEE estimate unless some constraints on the conditional distribution are imposed. A recent paper has proved that if the conditional density is conditionally symmetric and unimodal (CSUM), then the optimal MEE estimate (with Shannon entropy) equals the conditional median. In this study, we extend this result to the generalized MEE estimation where the optimality criterion…
A physically unclonable function (PUF) is an electronic circuit that produces an intrinsic identifier in response to a challenge. These identifiers depend on uncontrollable variations of the manufacturing process, which make them hard to predict or to replicate. Various security protocols leverage on such intrinsic randomness for authentification, cryptographic key generation, anti-counterfeiting, etc. Evaluating the entropy of PUFs (for all possible challenges) allows one to assess the security properties of such protocols. In this paper, we estimate the probability distribution of certain ki…
We propose a general framework for solving quantum state estimation problems using the minimum relative entropy criterion. A convex optimization approach allows us to decide the feasibility of the problem given the data and, whenever necessary, to relax the constraints in order to allow for a physically admissible solution. Building on these results, the variational analysis can be completed ensuring existence and uniqueness of the optimum. The latter can then be computed by standard, efficient standard algorithms for convex optimization, without resorting to approximate methods or restrictive…
In this paper, we investigate the partition inequality, joint convexity, and Pinsker's inequality, for a divergence that generalizes the Tsallis Relative Entropy and Kullback-Leibler divergence. The generalized divergence is defined in terms of a deformed exponential function, which replaces the Tsallis $q$-exponential. We also constructed a family of probability distributions related to the generalized divergence. We found necessary and sufficient conditions for the partition inequality to be satisfied. A sufficient condition for the joint convexity was established. We proved that the general…
Entropy Estimation is an important problem with many applications in cryptography, statistic,machine learning. Although the estimators optimal with respect to the sample complexity have beenrecently developed, there are still some challenges we address in this paper.The contribution is a novel estimator which is built directly on the birthday paradox. Theanalysis turns out to be considerably simpler and offer superior confidence bounds with explicitconstants. We also discuss how streaming algorithm can be used to massively improve memoryconsumption. Last but not least, we study the problem of …
Institute of Telecommunications, TU Wien（维也纳工业大学电信研究所）；Université Paris-Saclay, CNRS, CentraleSupélec, Laboratoire des Signaux et Systèmes（巴黎萨克雷大学、法国国家科学研究中心、中央理工-高等电力学院、信号与系统实验室）；Montreal Institute for Learning Algorithms (Mila), Université de Montréal（蒙特利尔学习算法研究所（Mila）、蒙特利尔大学）；Acoustics Research Institute, Austrian Academy of Sciences（奥地利科学院声学研究所）
The estimation of information measures of continuous distributions based on samples is a fundamental problem in statistics and machine learning. In this paper, we analyze estimates of differential entropy in $K$-dimensional Euclidean space, computed from a finite number of samples, when the probability density function belongs to a predetermined convex family $\mathcal{P}$. First, estimating differential entropy to any accuracy is shown to be infeasible if the differential entropy of densities in $\mathcal{P}$ is unbounded, clearly showing the necessity of additional assumptions. Subsequently,…
Conventional correlation-based frame synchronization techniques can suffer significant performance degradation over multi-path frequency-selective channels. As a remedy, in this paper we consider joint frame synchronization and channel estimation. This, however, increases the length of the resulting combined channel and its estimation becomes more challenging. On the other hand, since the combined channel is a sparse vector, sparse channel estimation methods can be applied. We propose a joint frame synchronization and channel estimation method using the orthogonal matching pursuit (OMP) algori…
The min-entropy is a widely used metric to quantify the randomness of generated random numbers, which measures the difficulty of guessing the most likely output. It is difficult to accurately estimate the min-entropy of a non-independent and identically distributed (non-IID) source. Hence, NIST Special Publication (SP) 800-90B adopts ten different min-entropy estimators and then conservatively selects the minimum value among ten min-entropy estimates. Among these estimators, the longest repeated substring (LRS) estimator estimates the collision entropy instead of the min-entropy by counting th…
There is no closed form analytical equation or quick method to calculate probabilities based only on the entropy of a signal or process. Except in the cases where there are constraints on the state probabilities, one must typically derive the underlying probabilities through search algorithms. These become more computationally expensive as entropies of higher orders are investigated. In this paper, a method to calculate a joint probability matrix based on the entropy for any order is elaborated. With this method, only first order entropies need to be successfully calculated while the others ar…
Maximum Likelihood (ML) algorithms, for the joint estimation of synchronization impairments and channel in Multiple Input Multiple Output-Orthogonal Frequency Division Multiplexing (MIMO-OFDM) system, are investigated in this work. A system model that takes into account the effects of carrier frequency offset, sampling frequency offset, symbol timing error, and channel impulse response is formulated. Cramér-Rao Lower Bounds for the estimation of continuous parameters are derived, which show the coupling effect among different impairments and the significance of the joint estimation. We propose…
Given two discrete random variables $X$ and $Y$, with probability distributions ${\bf p} =(p_1, \ldots , p_n)$ and ${\bf q}=(q_1, \ldots , q_m)$, respectively, denote by ${\cal C}({\bf p}, {\bf q})$ the set of all couplings of ${\bf p}$ and ${\bf q}$, that is, the set of all bivariate probability distributions that have ${\bf p}$ and ${\bf q}$ as marginals. In this paper, we study the problem of finding the joint probability distribution in ${\cal C}({\bf p}, {\bf q})$ of minimum entropy (equivalently, the joint probability distribution that maximizes the mutual information between $X$ and $Y$…
This paper presents a class of new fast non-trainable entropy-based confidence estimation methods for automatic speech recognition. We show how per-frame entropy values can be normalized and aggregated to obtain a confidence measure per unit and per word for Connectionist Temporal Classification (CTC) and Recurrent Neural Network Transducer (RNN-T) models. Proposed methods have similar computational complexity to the traditional method based on the maximum per-frame probability, but they are more adjustable, have a wider effective threshold range, and better push apart the confidence distribut…
W. M. Keck Science Department of Pitzer, Scripps, and Claremont McKenna College（皮泽学院、斯克里普斯学院和克莱蒙特·麦肯纳学院W.M.凯克科学系）；University of California at Davis（加州大学戴维斯分校）
Inferring models, predicting the future, and estimating the entropy rate of discrete-time, discrete-event processes is well-worn ground. However, a much broader class of discrete-event processes operates in continuous-time. Here, we provide new methods for inferring, predicting, and estimating them. The methods rely on an extension of Bayesian structural inference that takes advantage of neural network's universal approximation power. Based on experiments with complex synthetic data, the methods are competitive with the state-of-the-art for prediction and entropy-rate estimation.
Estimating the entropy rate of discrete time series is a challenging problem with important applications in numerous areas including neuroscience, genomics, image processing and natural language processing. A number of approaches have been developed for this task, typically based either on universal data compression algorithms, or on statistical estimators of the underlying process distribution. In this work, we propose a fully-Bayesian approach for entropy estimation. Building on the recently introduced Bayesian Context Trees (BCT) framework for modelling discrete time series as variable-memo…
This paper investigates parametric direction-of-arrival (DOA) estimation in a particular context: i) each sensor is characterized by an unknown complex gain and ii) the array consists of a collection of subarrays which are substantially separated from each other leading to a structured noise covariance matrix. We propose two iterative algorithms based on the maximum likelihood (ML) estimation method adapted to the context of joint array calibration and DOA estimation. Numerical simulations reveal that the two proposed schemes, the iterative ML (IML) and the modified iterative ML (MIML) algorit…
Consider the problem of estimating the Shannon entropy of a distribution over $k$ elements from $n$ independent samples. We show that the minimax mean-square error is within universal multiplicative constant factors of $$\Big(\frac{k }{n \log k}\Big)^2 + \frac{\log^2 k}{n}$$ if $n$ exceeds a constant factor of $\frac{k}{\log k}$; otherwise there exists no consistent estimator. This refines the recent result of Valiant-Valiant \cite{VV11} that the minimal sample size for consistent entropy estimation scales according to $Θ(\frac{k}{\log k})$. The apparatus of best polynomial approximation plays…
We investigate state estimation of linear systems over channels having a finite state not known by the transmitter or receiver. We show that similar to memoryless channels, zero-error capacity is the right figure of merit for achieving bounded estimation errors. We then consider finite-state, worst-case versions of the common erasure and additive noise channels models, in which the noise is governed by a finite-state machine without any statistical structure. Upper and lower bounds on their zero-error capacities are derived, revealing a connection with the {\em topological entropy} of the chan…
We compare an entropy estimator $H_z$ recently discussed in [10] with two estimators $H_1$ and $H_2$ introduced in [6][7]. We prove the identity $H_z \equiv H_1$, which has not been taken into account in [10]. Then, we prove that the statistical bias of $H_1$ is less than the bias of the ordinary likelihood estimator of entropy. Finally, by numerical simulation we verify that for the most interesting regime of small sample estimation and large event spaces, the estimator $H_2$ has a significant smaller statistical error than $H_z$.
Entropy estimation, due in part to its connection with mutual information, has seen considerable use in the study of time series data including causality detection and information flow. In many cases, the entropy is estimated using $k$-nearest neighbor (Kozachenko-Leonenko) based methods. However, analytic results on this estimator are limited to independent data. In the article, we show rigorous bounds on the rate of decay of the bias in the number of samples, $N$, assuming they are drawn from a stationary process which satisfies a suitable mixing condition. Numerical examples are presented w…
In this paper, we propose a novel deep learning based approach for joint channel estimation and signal detection in orthogonal frequency division multiplexing (OFDM) systems by exploring the time and frequency correlation of wireless fading channels. Specifically, a Channel Estimation Network (CENet) is designed to replace the conventional interpolation procedure in pilot-aided estimation scheme. Then, based on the outcome of the CENet, a Channel Conditioned Recovery Network (CCRNet) is designed to recover the transmit signal. Experimental results demonstrate that CENet and CCRNet achieve supe…
We consider sensor scheduling as the optimal observability problem for partially observable Markov decision processes (POMDP). This model fits to the cases where a Markov process is observed by a single sensor which needs to be dynamically adjusted or by a set of sensors which are selected one at a time in a way that maximizes the information acquisition from the process. Similar to conventional POMDP problems, in this model the control action is based on all past measurements; however here this action is not for the control of state process, which is autonomous, but it is for influencing the …
Differential entropy and log determinant of the covariance matrix of a multivariate Gaussian distribution have many applications in coding, communications, signal processing and statistical inference. In this paper we consider in the high dimensional setting optimal estimation of the differential entropy and the log-determinant of the covariance matrix. We first establish a central limit theorem for the log determinant of the sample covariance matrix in the high dimensional setting where the dimension $p(n)$ can grow with the sample size $n$. An estimator of the differential entropy and the lo…
In the context of state estimation under communication constraints, several notions of dynamical entropy play a fundamental role, among them: topological entropy and restoration entropy. In this paper, we present a theorem which demonstrates that for most dynamical systems restoration entropy strictly exceeds topological entropy. This implies that robust estimation policies in general require a higher rate of data transmission than non-robust ones. The proof of our theorem is quite short, but uses sophisticated tools from the theory of smooth dynamical systems.
KTH Royal Institute of Technology（瑞典皇家理工学院）；Tokyo Institute of Technology（东京工业大学）；New York University（纽约大学）
In this paper, we obtain generic bounds on the variances of estimation and prediction errors in time series analysis via an information-theoretic approach. It is seen in general that the error bounds are determined by the conditional entropy of the data point to be estimated or predicted given the side information or past observations. Additionally, we discover that in order to achieve the prediction error bounds asymptotically, the necessary and sufficient condition is that the "innovation" is asymptotically white Gaussian. When restricted to Gaussian processes and 1-step prediction, our boun…
We discuss algorithms for estimating the Shannon entropy h of finite symbol sequences with long range correlations. In particular, we consider algorithms which estimate h from the code lengths produced by some compression algorithm. Our interest is in describing their convergence with sequence length, assuming no limits for the space and time complexities of the compression algorithms. A scaling law is proposed for extrapolation from finite sample lengths. This is applied to sequences of dynamical systems in non-trivial chaotic regimes, a 1-D cellular automaton, and to written English texts.
Three Laws of Multivariable Feedback Systems, Extended Spectral Flatness (Extended Wiener Entropy), 'Uncertainty Principles' in Variance Minimization, and Performance Limitations in Minimum Variance Estimation/Filtering
