Hilhorst and Schehr recently presented a straight forward computation of limit distributions of sufficiently correlated random numbers \cite{hilhorst}. Here we present the analytical form of entropy which --under the maximum entropy principle (with ordinary constraints)-- provides these limit distributions. These distributions are not $q$-Gaussians and can not be obtained with Tsallis entropy.
It is possible to derive the maximum entropy principle from thermodynamic stability requirements. Using as a starting point the equilibrium probability distribution, currently used in non-extensive thermostatistics, it turns out that the relevant entropy function is Renyi's alpha-entropy, and not Tsallis' entropy.
The maximum entropy approach operating with quite general entropy measure and constraint is considered. It is demonstrated that for a conditional or parametrized probability distribution $f(x|μ)$ there is a "universal" relation among the entropy rate and the functions appearing in the constraint. It is shown that the recently proposed variational formulation of the entropic functional can be obtained as a consequence of this relation, that is from the maximum entropy principle. This resolves certain puzzling points appeared in the variational approach.
With use of the Second Inverse Maximum Entropy Principle we find entropy functions for systems with fractal distribution functions with order parameter $q$. We compare these entropy functions with those given by the Bose-Einstein and Fermi-Dirac cases.
After the justification of the maximum entropy approach for equilibrium thermodynamic system, and of a maximum path entropy algorithm for nonequilibrium thermodynamic systems by virtue of the principle of virtual work, we present in this paper another application of the principle to thermodynamic systems out of equilibrium. Unlike the justification of maximum path entropy for the motion trajectories during a period of time, this work is on the maximum of the entropy defined as a measure of the momentary dynamical uncertainty as a function of the probability distribution over the microstates of…
It is shown that the laws of thermodynamics are extremely robust under generalizations of the form of entropy. Using the Bregman-type relative entropy, the Clausius inequality is proved to be always valid. This implies that thermodynamics is highly universal and does not rule out consistent generalization of the maximum entropy method.
Sanov's Theorem and the Conditional Limit Theorem (CoLT) are established for a multicolor Polya Eggenberger urn sampling scheme, giving the Polya divergence and the Polya extension to the Maximum Relative Entropy (MaxEnt) method. Polya MaxEnt includes the standard MaxEnt as a special case. The universality of standard MaxEnt - advocated by an axiomatic approach to inference for inverse problems - is challenged, in favor of a probabilistic approach based on CoLT and the Maximum Probability principle.
It is shown that Onsager's principle of the least dissipation of energy is equivalent to the maximum entropy production principle. It is known that solutions of the linearized Boltzmann equation make extrema of entropy production. It is argued, in the case of stationary processes, that this extremum is a maximum rather than a minimum.
We will discuss the maximum entropy production (MaxEP) principle based on Jaynes' information theoretical arguments, as was done by Dewar (2003, 2005). With the help of a simple mathematical model of a non-equilibrium system, we will show how to derive minimum and maximum entropy production. Furthermore, the model will help us to clarify some confusing points and to see differences between some MaxEP studies in the literature.
We derive the microcanonical ensemble from the Maximum Entropy Principle (MEP) using the phase space volume entropy of P. Hertz. Maximizing this entropy with respect to the probability distribution with the constraints of normalization and average energy, we obtain the condition of constant energy. This approach is complementary to the traditional derivation of the microcanonical ensemble from the MEP using Shannon entropy and assuming a priori that the energy is constant which results in equal probabilities.
In contrast to the standard derivation of Kirchhoff's loop law, which invokes electric potential, we show, for the linear planar electric network in a stationary state at the fixed temperature,that loop law can be derived from the maximum entropy production principle. This means that the currents in network branches are distributed in such a way as to achieve the state of maximum entropy production.
We give a new proof of the theorems on the maximum entropy principle in Tsallis statistics. That is, we show that the $q$-canonical distribution attains the maximum value of the Tsallis entropy, subject to the constraint on the $q$-expectation value and the $q$-Gaussian distribution attains the maximum value of the Tsallis entropy, subject to the constraint on the $q$-variance, as applications of the nonnegativity of the Tsallis relative entropy, without using the Lagrange multipliers method. In addition, we define a $q$-Fisher information and then prove a $q$-Cramér-Rao inequality that the $q…
We consider relaxation of an isolated system to the equilibrium using detailed balance condition and Onsager's fluctuation approximation. There is a small deviation from the equilibrium in two parameters. For this system, explicit expressions both for the dependence of trajectory entropy on random thermodynamic fluxes and for the dependence of entropy production on the most probable thermodynamic fluxes are obtained. Onsager's linear relations are obtained for the considered model using two methods (maximization of trajectory entropy and Ziegler's maximization of entropy production). Two exist…
The Maximum Entropy (MaxEnt) technique is applied to the derivation of the Gaussian Dispersion Plume Model as well as to more complex transport phenomena such as the one-dimensional advection equation, the one-dimensional diffusion equation, the one dimensional advection-diffusion equation, and finally to the multi-dimensional advection-diffusion equation. Further application is discussed.
We apply the Principle of Maximum Entropy to the study of a general class of deterministic fractal sets. The scaling laws peculiar to these objects are accounted for by means of a constraint concerning the average content of information in those patterns. This constraint allows for a new statistical characterization of fractal objects and fractal dimension.
We propose an extension of the principle of virtual work of mechanics to random dynamics of mechanical systems. The total virtual work of the interacting forces and inertial forces on every particle of the system is calculated by considering the motion of each particle. Then according to the principle of Lagrange-d'Alembert for dynamical equilibrium, the vanishing ensemble average of the virtual work gives rise to the thermodynamic equilibrium state with maximization of thermodynamic entropy. This approach establishes a close relationship between the maximum entropy approach for statistical me…
Maximum entropy method is a constructive criterion for setting up a probability distribution maximally non-committal to missing information on the basis of partial knowledge, usually stated as constrains on expectation values of some functions. In connection with experiments sample average of those functions are used as surrogate of the expectation values. We address sampling bias in maximum entropy approaches with finite data sets without forcedly equating expectation values to corresponding experimental average values. Though we rise the approach in a general formulation, the equations are u…
The random matrix ensembles (RME) of quantum statistical Hamiltonian operators, e.g. Gaussian random matrix ensembles (GRME) and Ginibre random matrix ensembles (Ginibre RME), are applied to following quantum statistical systems: nuclear systems, molecular systems, and two-dimensional electron systems (Wigner-Dyson electrostatic analogy). Measures of quantum chaos and quantum integrability with respect to eigenergies of quantum systems are defined and calculated. Quantum statistical information functional is defined as negentropy (opposite of entropy or minus entropy). The distribution functio…
In this paper, we develop a general theory for the estimation of the transition probabilities of reversible Markov chains using the maximum entropy principle. A broad range of physical models can be studied within this approach. We use one-dimensional classical spin systems to illustrate the theoretical ideas. The examples studied in this paper are: the Ising model, the Potts model and the Blume-Emery-Griffiths model.
The random matrix ensembles (RME) of quantum statistical Hamiltonians, e.g. Gaussian random matrix ensembles (GRME) and Ginibre random matrix ensembles (Ginibre RME), are applied in literature to following quantum statistical systems: molecular systems, nuclear systems, disordered materials, random Ising spin systems, and two-dimensional electron systems (Wigner-Dyson electrostatic analogy). Measures of quantum chaos and quantum integrability with respect to eigenergies of quantum systems are defined and calculated. Quantum statistical information functional is defined as negentropy (opposite …
The maximum entropy principle in Tsallis statistics is reformulated in the mathematical framework of the q-product, which results in the unique non self-referential q-canonical distribution. As one of the applications of the present formalism, we theoretically derive the physical temperature which coincides with that already obtained in accordance with the generalized zeroth law of thermodynamics.
We generalize the usual exponential Boltzmann factor to any reasonable and potentially observable distribution function, $B(E)$. By defining generalized logarithms $Λ$ as inverses of these distribution functions, we are led to a generalization of the classical Boltzmann-Gibbs entropy, $S_{BG}= -\int d εω(ε) B(ε) \log B(ε)$ to the expression $S\equiv -\int d εω(ε) \int_0^{B(ε)} dx Λ(x)$, which contains the classical entropy as a special case. We demonstrate that this entropy has two important features: First, it describes the correct thermodynamic relations of the system, and second, the observ…
Random matrix ensembles (RME) of quantum statistical Hamiltonian operators, {\em e.g.} Gaussian random matrix ensembles (GRME) and Ginibre random matrix ensembles (Ginibre RME), found applications in literature in study of following quantum statistical systems: molecular systems, nuclear systems, disordered materials, random Ising spin systems, quantum chaotic systems, and two-dimensional electron systems (Wigner-Dyson electrostatic analogy). Measures of quantum chaos and quantum integrability with respect to eigenergies of quantum systems are defined and calculated. Quantum statistical inform…
The equilibrium distributions of probabilities providing maximality of Renyi and Tsallis entropies are rederived. New S-forms of them are found which are normalised with corresponding entropies in contrast to the usual Z-forms normalised with partition functions.
Maximum entropy (maxEnt) inference of state probabilities using state-dependent constraints is popular in the study of complex systems. In stochastic dynamical systems, the effect of state space topology and path-dependent constraints on the inferred state probabilities is unknown. To that end, we derive the transition probabilities and the stationary distribution of a maximum {\it path} entropy Markov process subject to state- and path-dependent constraints. The stationary distribution reflects a competition between path multiplicity and imposed constraints and is significantly different from…
We introduce a class of maximum-entropy states that naturally includes within it all of the major continuous-time stochastic processes that have been applied to animal movement, including Brownian motion, Ornstein-Uhlenbeck motion, integrated Ornstein-Uhlenbeck motion, a recently discovered hybrid of the previous models, and a new model that describes central-place foraging. We are also able to predict a further hierarchy of new models that will emerge as data quality improves to better resolve the underlying continuity of animal movement. Finally, we also show that Langevin equations must obe…
A stochastic action principle for stochastic dynamics is revisited. We present first numerical diffusion experiments showing that the diffusion path probability depend exponentially on average Lagrangian action. This result is then used to derive an uncertainty measure defined in a way mimicking the heat or entropy in the first law of thermodynamics. It is shown that the path uncertainty (or path entropy) can be measured by the Shannon information and that the maximum entropy principle and the least action principle of classical mechanics can be unified into a concise form. It is argued that t…
We deal with a generalized statistical description of nonequilibrium complex systems based on least biased distributions given some prior information. A maximum entropy principle is introduced that allows for the determination of the distribution of the fluctuating intensive parameter $β$ of a superstatistical system, given certain constraints on the complex system under consideration. We apply the theory to three examples: The superstatistical quantum mechanical harmonic oscillator, the superstatistical classical ideal gas, and velocity time series as measured in a turbulent Taylor-Couette fl…
The problem of determining the joint probability distributions for correlated random variables with pre-specified marginals is considered. When the joint distribution satisfying all the required conditions is not unique, the "most unbiased" choice corresponds to the distribution of maximum entropy. The calculation of the maximum entropy distribution requires the solution of rather complicated nonlinear coupled integral equations, exact solutions to which are obtained for the case of Gaussian marginals; otherwise, the solution can be expressed as a perturbation around the product of the margina…
We have calculated the Tsallis entropy and Fisher information matrix (entropy) of spatially-correlated nonextensive systems, by using an analytic non-Gaussian distribution obtained by the maximum entropy method. Effects of the correlated variability on the Fisher information matrix are shown to be different from those on the Tsallis entropy. The Fisher information is increased (decreased) by a positive (negative) correlation, whereas the Tsallis entropy is decreased with increasing an absolute magnitude of the correlation independently of its sign. This fact arises from the difference in their…
I show that the maximum entropy principle can be replaced by a more natural assumption, that there exists a phenomenological function of entropy consistent with the microscopic model. The requirement of existence provides then a unique construction of the related probability density. I conclude the letter with an axiomatic formulation of the notion of entropy, which is suitable for exploration of the non-equilibrium phenomena.
We apply the maximum entropy principle to economic systems in equilibrium and find the density function for the market's wealth. This is the same as price density which is used for insurance pricing. The risk aversion parameter of the agent then it's utility function with respect to this density is derived.
We show that the Jaynes principle is indeed a proper inference scheme when applied to compound systems and will correctly produce the entangled maximum entropy states compatible with appropriate data. This is accomplished by including the dispersion of the entanglement along with its mean value as constraints and an application of the uncertainty principle. We also construct a "thermodynamic-like" description of the entanglement arising out of the maximum entropy principle.
We propose an information-theoretical measure, the \textit{relative cluster entropy} $\mathcal{D_{C}}[P \| Q] $, to discriminate among cluster partitions characterised by probability distribution functions $P$ and $Q$. The measure is illustrated with the clusters generated by pairs of fractional Brownian motions with Hurst exponents $H_1$ and $H_2$ respectively. For subdiffusive, normal and superdiffusive sequences, the relative entropy sensibly depends on the difference between $H_1$ and $H_2$. By using the \textit{minimum relative entropy} principle, cluster sequences characterized by differ…
Recently, the conditional maximum-entropy method (abbreviated as C-MaxEnt) has been proposed for selecting priors in Bayesian statistics in a very simple way. Here, it is examined for extreme-value statistics. For the Weibull type as an explicit example, it is shown how C-MaxEnt can give rise to a prior satisfying Jeffreys' rule.
Here I investigate some mathematical aspects of the maximum entropy theory of ecology (METE). In particular I address the geometrical structure of METE endowed by information geometry. As novel results, the macrostate entropy is calculated analytically by the Legendre transformation of the log-normalizer in METE. This result allows for the calculation of the metric terms in the information geometry arising from METE and, by consequence, the covariance matrix between METE variables.
When a liquid is cooled below its melting temperature, if crystallization is avoided, it forms a glass. This phenomenon, called glass transition, is characterized by a marked increase of viscosity, about 14 orders of magnitude, in a narrow temperature interval. The microscopic mechanism behind the glass transition is still poorly understood. However, recently, great advances have been made in the identification of cooperative rearranging regions, or dynamical heterogeneities, i.e. domains of the liquid whose relaxation is highly correlated. The growth of the size of these domains is now believ…
We show that the naive application of the maximum entropy principle can yield answers which depend on the level of description, i.e. the result is not invariant under coarse-graining. We demonstrate that the correct approach, even for discrete systems, requires maximization of the relative entropy with a suitable reference probability, which in some instances can be deduced from the symmetry properties of the dynamics. We present simple illustrations of this crucial yet surprising feature in examples of classical and quantum statistical mechanics, as well as in the field of ecology.
In this study we apply the maximum entropy principle to derive the properly scaled velocity distribution function of Boltzmann equations for mixtures, which leads to a non-isothermal Maxwell-Stefan diffusion model. We also analyze the entropy balance law and derive the kinetic entropy production from the scaled distribution function.
We derive rigorous results on the link between the principle of maximum entropy production and the principle of maximum Kolmogorov-Sinai entropy using a Markov model of the passive scalar diffusion called the Zero Range Process. We show analytically that both the entropy production and the Kolmogorov-Sinai entropy seen as functions of f admit a unique maximum denoted fmaxEP and fmaxKS. The behavior of these two maxima is explored as a function of the system disequilibrium and the system resolution N. The main result of this article is that fmaxEP and fmaxKS have the same Taylor expansion at _r…
ConIII (pronounced CON-ee) is an open-source Python project providing a simple interface to solving the pairwise and higher order Ising model and a base for extension to other maximum entropy models. We describe the maximum entropy problem and give an overview of the algorithms that are implemented as part of ConIII (https://github.com/eltrompetero/coniii) including Monte Carlo histogram, pseudolikelihood, minimum probability flow, a regularized mean field method, and a cluster expansion method. Our goal is to make a variety of maximum entropy techniques accessible to those unfamiliar with the…
We tried to explain our point of view on this principle and answer a number of critical questions in this comment. It is the first time we have clearly presented all the existing restrictions of MEPP application and briefly explained them; this will help to avoid misunderstandings connected with its use.
The block entanglement entropy and fluctuations are investigated in one dimension in finite size correlated electron systems using the Gutzwiller wave function as a prototype correlated electron state. Entanglement entropy shows logarithmic divergence for all values of the correlation projection parameter $g$, as predicted by conformal field theories for critical systems, but the central charge requires finite size corrections. There is an infinite correlation length corresponding to correlation between same kinds of spins, for all values of $g$. A scaling form for the block entropy, as a func…
We discuss a characterization of complexity based on successive approximations of the probability density describing a system by means of maximum entropy methods, thereby quantifying the respective role played by different orders of interaction. This characterization is applied on simple cellular automata in order to put it in perspective with the usual notion of complexity for such systems based on Wolfram classes. The overlap is shown to be good, but not perfect. This suggests that complexity in the sense of Wolfram emerges as an intermediate regime of maximum entropy-based complexity, but a…
In this letter we show that the Shore--Johnson axioms for Maximum Entropy Principle in statistical estimation theory account for a considerably wider class of entropic functional than previously thought. Apart from a formal side of the proof, we substantiate our point by analyzing the effect of weak correlations and discuss two pertinent examples: $2$-qubit quantum system and strongly interacting nuclear systems.
The possibility of reconciliation between canonical probability distributions obtained from the $q$-maximum entropy principle with predictions from the law of large numbers when empirical samples are held to the same constraints, is investigated into. Canonical probability distributions are constrained by both: $(i)$ the additive duality of generalized statistics and $(ii)$ normal averages expectations. Necessary conditions to establish such a reconciliation are derived by appealing to a result concerning large deviation properties of conditional measures. The (dual) $q^*$-maximum entropy prin…
