We show that various formulations (e.g., dual and Kullback-Csiszar iterations) of estimation of maximum entropy (ME) models can be transformed to solving systems of polynomial equations in several variables for which one can use celebrated Grobner bases methods. Posing of ME estimation as solving polynomial equations is possible, in the cases where feature functions (sufficient statistic) that provides the information about the underlying random variable in the form of expectations are integer valued.
Let $K = \{0,1,...,q-1\}$. We use a special class of translation invariant measures on $K^\mathbb{Z}$ called algebraic measures to study the entropy rate of a hidden Markov processes. Under some irreducibility assumptions of the Markov transition matrix we derive exact formulas for the entropy rate of a general $q$ state hidden Markov process derived from a Markov source corrupted by a specific noise model. We obtain upper bounds on the error when using an approximation to the formulas and numerically compute the entropy rates of two and three state hidden Markov models.
The asymptotic convergence of probability density function (pdf) and convergence of differential entropy are examined for the non-stationary processes that follow the maximum entropy principle (MaxEnt) and maximum entropy production principle (MEPP). Asymptotic convergence of pdf provides new justification of MEPP while convergence of differential entropy is important in asymptotic analysis of communication systems. A set of equations describing the dynamics of pdf under mass conservation and energy conservation constraints is derived. It is shown that for pdfs with compact carrier the limit p…
Two maximization problems of Rényi entropy rate are investigated: the maximization over all stochastic processes whose marginals satisfy a linear constraint, and the Burg-like maximization over all stochastic processes whose autocovariance function begins with some given values. The solutions are related to the solutions to the analogous maximization problems of Shannon entropy rate.
Model selection is central to statistics, and many learning problems can be formulated as model selection problems. In this paper, we treat the problem of selecting a maximum entropy model given various feature subsets and their moments, as a model selection problem, and present a minimum description length (MDL) formulation to solve this problem. For this, we derive normalized maximum likelihood (NML) codelength for these models. Furthermore, we prove that the minimax entropy principle is a special case of maximum entropy model selection, where one assumes that complexity of all the models ar…
University of Electronic Science and Technology of China（电子科技大学）；Shannxi Normal University（陕西师范大学）；Japan Advanced Institute of Science and Technology（日本北陆先端科学技术大学院大学）；ETH Zurich（苏黎世联邦理工学院）
Recently, a new type of set, named as random permutation set (RPS), is proposed by considering all the permutations of elements in a certain set. For measuring the uncertainty of RPS, the entropy of RPS is presented. However, the maximum entropy principle of RPS entropy has not been discussed. To address this issue, in this paper, the maximum entropy of RPS is presented. The analytical solution for maximum entropy of RPS and its corresponding PMF condition are respectively proofed and discussed. Numerical examples are used to illustrate the maximum entropy RPS. The results show that the maximu…
Algebraic statistics is a recently evolving field, where one would treat statistical models as algebraic objects and thereby use tools from computational commutative algebra and algebraic geometry in the analysis and computation of statistical models. In this approach, calculation of parameters of statistical models amounts to solving set of polynomial equations in several variables, for which one can use celebrated Grobner bases theory. Owing to the important role of information theory in statistics, this paper as a first step, explores the possibility of describing maximum and minimum entrop…
In this paper, following standard arguments, the maximum Renyi entropy problem for the weighted case is analyzed. We verify that under some constrains on weight function, the Student-r and Student-t distributions maximize the weighted Renyi entropy. Furthermore, an extended version of the Hadamard inequality is derived.
University of North Carolina Asheville（北卡罗来纳大学阿什维尔分校）
The principle of maximum entropy is a broadly applicable technique for computing a distribution with the least amount of information possible constrained to match empirical data, for instance, feature expectations. We seek to generalize this principle to scenarios where the empirical feature expectations cannot be computed because the model variables are only partially observed, which introduces a dependency on the learned model. Generalizing the principle of latent maximum entropy, we introduce uncertain maximum entropy and describe an expectation-maximization based solution to approximately …
University of North Carolina Asheville（北卡罗来纳大学阿什维尔分校）；University of Georgia（佐治亚大学）
The principle of maximum entropy is a broadly applicable technique for computing a distribution with the least amount of information possible while constrained to match empirically estimated feature expectations. However, in many real-world applications that use noisy sensors computing the feature expectations may be challenging due to partial observation of the relevant model variables. For example, a robot performing apprenticeship learning may lose sight of the agent it is learning from due to environmental occlusion. We show that in generalizing the principle of maximum entropy to these ty…
On a compact group the Haar probability measure plays the role of uniform distribution. The entropy and rate distortion theory for this uniform distribution is studied. New results and simplified proofs on convergence of convolutions on compact groups are presented and they can be formulated as entropy increases to its maximum. Information theoretic techniques and Markov chains play a crucial role. The convergence results are also formulated via rate distortion functions. The rate of convergence is shown to be exponential.
This paper describes the Elliptical Quartic Exponential distribution in $\mathbb{R}^D$, obtained via a maximum entropy construction by imposing second and fourth moment constraints. I discuss relationships to related work, analytical expressions for the normalization constant and the entropy, and the conditional and marginal distributions.
This is a preliminary article stating and proving a new maximum entropy theorem. The entropies that we consider can be used as measures of biodiversity. In that context, the question is: for a given collection of species, which frequency distribution(s) maximize the diversity? The theorem provides the answer. The chief surprise is that although we are dealing with not just a single entropy, but a one-parameter family of entropies, there is a single distribution maximizing all of them simultaneously.
What is the relationship between plausibility logic and the principle of maximum entropy? When does the principle give unreasonable or wrong results? When is it appropriate to use the rule `expectation = average'? Can plausibility logic give the same answers as the principle, and better answers if those of the principle are unreasonable? To try to answer these questions, this study offers a numerical collection of plausibility distributions given by the maximum-entropy principle and by plausibility logic for a set of fifteen simple problems: throwing dice.
This paper modifies Jaynes's axioms of plausible reasoning and derives the minimum relative entropy principle, Bayes's rule, as well as maximum likelihood from first principles. The new axioms, which I call the Optimum Information Principle, is applicable whenever the decision maker is given the data and the relevant background information. These axioms provide an answer to the question "why maximize entropy when faced with incomplete information?"
The possibility of reconciliation between canonical probability distributions obtained from the $q$-maximum entropy principle with predictions from the law of large numbers when empirical samples are held to the same constraints, is investigated into. Canonical probability distributions are constrained by both: $(i)$ the additive duality of generalized statistics and $(ii)$ normal averages expectations. Necessary conditions to establish such a reconciliation are derived by appealing to a result concerning large deviation properties of conditional measures. The (dual) $q^*$-maximum entropy prin…
