This paper reports a new reading for wavelets, which is based on the classical 'De Broglie' principle. The wave-particle duality principle is adapted to wavelets. Every continuous basic wavelet is associated with a proper probability density, allowing defining the Shannon entropy of a wavelet. Further entropy definitions are considered, such as Jumarie or Renyi entropy of wavelets. We proved that any wavelet of the same family has the same Shannon entropy of its mother wavelet. Finally, the Shannon entropy for a few standard wavelet families is determined.
We derive expressions for the Shannon and Rényi entropy rates of stationary vector valued Gaussian random processes using the block matrix version of Szegö's theorem.
By replacing linear averaging in Shannon entropy with Kolmogorov-Nagumo average (KN-averages) or quasilinear mean and further imposing the additivity constraint, Rényi proposed the first formal generalization of Shannon entropy. Using this recipe of Rényi, one can prepare only two information measures: Shannon and Rényi entropy. Indeed, using this formalism Rényi characterized these additive entropies in terms of axioms of quasilinear mean. As additivity is a characteristic property of Shannon entropy, pseudo-additivity of the form $x \oplus_{q} y = x + y + (1-q)x y$ is a characteristic proper…
Inria（法国国家信息与自动化研究所）；École Polytechnique（巴黎综合理工学院）；Università di Siena（锡耶纳大学）
Feature selection, in the context of machine learning, is the process of separating the highly predictive feature from those that might be irrelevant or redundant. Information theory has been recognized as a useful concept for this task, as the prediction power stems from the correlation, i.e., the mutual information, between features and labels. Many algorithms for feature selection in the literature have adopted the Shannon-entropy-based mutual information. In this paper, we explore the possibility of using Rényi min-entropy instead. In particular, we propose an algorithm based on a notion o…
Existing polarization theories have mostly been concerned with Shannon's information measures, such as Shannon entropy and mutual information, and some related measures such as the Bhattacharyya parameter. In this work, we extend polarization theories to a more general information measure, namely, the Rényi entropy. Our study shows that under conditional Rényi entropies of different orders, the same synthetic sub-channel may exhibit opposite extremal states. This result reveals more insights into the polarization phenomenon on the micro scale (probability pairs) rather than on the average scal…
The LMC complexity, an indicator of complexity based on a probabilistic description, is revisited. A straightforward approach allows us to establish the time evolution of this indicator in a near-equilibrium situation and gives us a new insight for interpreting the LMC complexity for a general non equilibrium system. Its relationship with the Renyi entropies is also explained. One of the advantages of this indicator is that its calculation does not require a considerable computational effort in many cases of physical and biological interest.
We investigate quantum Rényi entropic quantities, specifically those derived from 'sandwiched' divergence. This divergence is one of several proposed Rényi generalisations of the quantum relative entropy. We may define Rényi generalisations of the quantum conditional entropy and mutual information in terms of this divergence, from which they inherit many desirable properties. However, these quantities lack some of the convenient structure of their Shannon and von Neumann counterparts. We attempt to bridge this gap by establishing divergence inequalities for valid combinations of Rényi order wh…
We associate to the p-th Rényi entropy a definition of entropy power, which is the natural extension of Shannon's entropy power and exhibits a nice behaviour along solutions to the p-nonlinear heat equation in $R^n$. We show that the Rényi entropy power of general probability densities solving such equations is always a concave function of time, whereas it has a linear behaviour in correspondence to the Barenblatt source-type solutions. We then shown that the p-th Rényi entropy power of a probability density which solves the nonlinear diffusion of order p, is a concave function of time. This r…
This paper addresses the correspondence between linear inequalities of Shannon entropy and differential entropy for sums of independent group-valued random variables. We show that any balanced (with the sum of coefficients being zero) linear inequality of Shannon entropy holds if and only if its differential entropy counterpart also holds; moreover, any linear inequality for differential entropy must be balanced. In particular, our result shows that recently proved differential entropy inequalities by Kontoyiannis and Madiman \cite{KM14} can be deduced from their discrete counterparts due to T…
We prove that the average smooth Renyi entropy rate will approach the entropy rate of a stationary, ergodic information source, which is equal to the Shannon entropy rate for a classical information source and the von Neumann entropy rate for a quantum information source.
Algorithmic entropy and Shannon entropy are two conceptually different information measures, as the former is based on size of programs and the later in probability distributions. However, it is known that, for any recursive probability distribution, the expected value of algorithmic entropy equals its Shannon entropy, up to a constant that depends only on the distribution. We study if a similar relationship holds for Rényi and Tsallis entropies of order $α$, showing that it only holds for Rényi and Tsallis entropies of order 1 (i.e., for Shannon entropy). Regarding a time bounded analogue rel…
In this paper, we examine the Renyi entropy rate of stationary ergodic processes. For a special class of stationary ergodic processes, we prove that the Renyi entropy rate always exists and can be polynomially approximated by its defining sequence; moreover, using the Markov approximation method, we show that the Renyi entropy rate can be exponentially approximated by that of the Markov approximating sequence, as the Markov order goes to infinity. For the general case, by constructing a counterexample, we disprove the conjecture that the Renyi entropy rate of a general stationary ergodic proce…
We introduce a variant of the Rényi entropy definition that aligns it with the well-known Hölder mean: in the new formulation, the r-th order Rényi Entropy is the logarithm of the inverse of the r-th order Hölder mean. This brings about new insights into the relationship of the Rényi entropy to quantities close to it, like the information potential and the partition function of statistical mechanics. We also provide expressions that allow us to calculate the Rényi entropies from the Shannon cross-entropy and the escort probabilities. Finally, we discuss why shifting the Rényi entropy is fruitf…
Recently, Savaré-Toscani proved that the Rényi entropy power of general probability densities solving the $p$-nonlinear heat equation in $\mathbb{R}^n$ is always a concave function of time, which extends Costa's concavity inequality for Shannon's entropy power to Rényi entropies. In this paper, we give a generalization of Savaré-Toscani's result by giving a class of sufficient conditions of the parameters under which the concavity of the Rényi entropy power is still valid. These conditions are quite general and include the parameter range given by Savaré-Toscani as special cases. Also, the con…
The paper examines relationships between the Shannon entropy and the $\ell_α$-norm for $n$-ary probability vectors, $n \ge 2$. More precisely, we investigate the tight bounds of the $\ell_α$-norm with a fixed Shannon entropy, and vice versa. As applications of the results, we derive the tight bounds between the Shannon entropy and several information measures which are determined by the $\ell_α$-norm, e.g., Rényi entropy, Tsallis entropy, the $R$-norm information, and some diversity indices. Moreover, we apply these results to uniformly focusing channels. Then, we show the tight bounds of Gall…
The paper examines relationships between the conditional Shannon entropy and the expectation of $\ell_α$-norm for joint probability distributions. More precisely, we investigate the tight bounds of the expectation of $\ell_α$-norm with a fixed conditional Shannon entropy, and vice versa. As applications of the results, we derive the tight bounds between the conditional Shannon entropy and several information measures which are determined by the expectation of $\ell_α$-norm, e.g., the conditional Rényi entropy and the conditional $R$-norm information. Moreover, we apply these results to discret…
Building on Shannon's lead, let's consider a more malleable expression for tracking uncertainty, and states of "knowledge available" vs. "knowledge missing," to better practice innovation, improve risk management, and successfully measure progress of intractable undertakings. Shannon's formula, and its common replacements (Renyi, Tsallis) compute to increased knowledge whenever two competing choices, however marginal, exchange probability measures. Such and other distortions are corrected by anchoring knowledge to a reference challenge. Entropy then expresses progress towards meeting that chal…
Shannon entropy is the most crucial foundation of Information Theory, which has been proven to be effective in many fields such as communications. Renyi entropy and Chernoff information are other two popular measures of information with wide applications. The mutual information is effective to measure the channel information for the fact that it reflects the relation between output variables and input variables. In this paper, we reexamine these channel information measures in big data viewpoint by means of ACE algorithm. The simulated results show us that decomposition results of Shannon and …
How hard is it guess a password? Massey showed that that the Shannon entropy of the distribution from which the password is selected is a lower bound on the expected number of guesses, but one which is not tight in general. In a series of subsequent papers under ever less restrictive stochastic assumptions, an asymptotic relationship as password length grows between scaled moments of the guesswork and specific Rényi entropy was identified. Here we show that, when appropriately scaled, as the password length grows the logarithm of the guesswork satisfies a Large Deviation Principle (LDP), provi…
We discuss the interest of escort distributions and Rényi entropy in the context of source coding. We first recall a source coding theorem by Campbell relating a generalized measure of length to the Rényi-Tsallis entropy. We show that the associated optimal codes can be obtained using considerations on escort-distributions. We propose a new family of measure of length involving escort-distributions and we show that these generalized lengths are also bounded below by the Rényi entropy. Furthermore, we obtain that the standard Shannon codes lengths are optimum for the new generalized lengths mea…
This paper gives improved Rényi entropy power inequalities (R-EPIs). Consider a sum $S_n = \sum_{k=1}^n X_k$ of $n$ independent continuous random vectors taking values on $\mathbb{R}^d$, and let $α\in [1, \infty]$. An R-EPI provides a lower bound on the order-$α$ Rényi entropy power of $S_n$ that, up to a multiplicative constant (which may depend in general on $n, α, d$), is equal to the sum of the order-$α$ Rényi entropy powers of the $n$ random vectors $\{X_k\}_{k=1}^n$. For $α=1$, the R-EPI coincides with the well-known entropy power inequality by Shannon. The first improved R-EPI is obtain…
The Renyi entropies constitute a family of information measures that generalizes the well-known Shannon entropy, inheriting many of its properties. They appear in the form of unconditional and conditional entropies, relative entropies or mutual information, and have found many applications in information theory and beyond. Various generalizations of Renyi entropies to the quantum setting have been proposed, most notably Petz's quasi-entropies and Renner's conditional min-, max- and collision entropy. Here, we argue that previous quantum extensions are incompatible and thus unsatisfactory. We p…
The direct spreading measures of the Laguerre polynomials, which quantify the distribution of its Rakhmanov probability density along the positive real line in various complementary and qualitatively different ways, are investigated. These measures include the familiar root-mean-square or standard deviation and the information-theoretic lengths of Fisher, Renyi and Shannon types. The Fisher length is explicitly given. The Renyi length of order q (such that 2q is a natural number) is also found in terms of the polynomials parameters by means of two error-free computing approaches; one makes use…
Tsallis and Rényi entropies, which are monotone transformations of each other, are deformations of the celebrated Shannon entropy. Maximization of these deformed entropies, under suitable constraints, leads to the $q$-exponential family which has applications in non-extensive statistical physics, information theory and statistics. In previous information-geometric studies, the $q$-exponential family was analyzed using classical convex duality and Bregman divergence. In this paper, we show that a generalized $λ$-duality, where $λ= 1 - q$ is the constant information-geometric curvature, leads to…
Massachusetts Institute of Technology（麻省理工学院）；University of California, San Diego（加州大学圣地亚哥分校）；Indian Institute of Science（印度科学学院）
It was recently shown that estimating the Shannon entropy $H({\rm p})$ of a discrete $k$-symbol distribution ${\rm p}$ requires $Θ(k/\log k)$ samples, a number that grows near-linearly in the support size. In many applications $H({\rm p})$ can be replaced by the more general Rényi entropy of order $α$, $H_α({\rm p})$. We determine the number of samples needed to estimate $H_α({\rm p})$ for all $α$, showing that $α< 1$ requires a super-linear, roughly $k^{1/α}$ samples, noninteger $α>1$ requires a near-linear $k$ samples, but, perhaps surprisingly, integer $α>1$ requires only $Θ(k^{1-1/α})$ sam…
The Sharma-Mittal entropies generalize the celebrated Shannon, Rényi and Tsallis entropies. We report a closed-form formula for the Sharma-Mittal entropies and relative entropies for arbitrary exponential family distributions. We instantiate explicitly the formula for the case of the multivariate Gaussian distributions and discuss on its estimation.
École polytechnique fédérale de Lausanne (EPFL)（洛桑联邦理工学院）
The dual-tree complex wavelet transform (DT-CWT) is known to exhibit better shift-invariance than the conventional discrete wavelet transform. We propose an amplitude-phase representation of the DT-CWT which, among other things, offers a direct explanation for the improvement in the shift-invariance. The representation is based on the shifting action of the group of fractional Hilbert transform (fHT) operators, which extends the notion of arbitrary phase-shifts from sinusoids to finite-energy signals (wavelets in particular). In particular, we characterize the shiftability of the DT-CWT in ter…
