Explicit Bounds for Entropy Concentration under Linear Constraints
Kostas N. Oikonomou, Peter D. Grunwald
备注:
1) An error affecting sec. 3 has been corrected: the parameters delta and theta cannot be chosen independently. Sec. 3 has been revised up to Theorem 3.15 in sec. 3.6. 2) Some minor updates in sec. 4. 3) Some proofs used in both sec. 3 and sec. 4 have been unified (This version to appear in IEEE Transactions on Information Theory, December 2015)
Consider the set of all sequences of $n$ outcomes, each taking one of $m$ values, that satisfy a number of linear constraints. If $m$ is fixed while $n$ increases, most sequences that satisfy the constraints result in frequency vectors whose entropy approaches that of the maximum entropy vector satisfying the constraints. This well-known "entropy concentration" phenomenon underlies the maximum entropy method.
Existing proofs of the concentration phenomenon are based on limits or asymptotics and unrealistically assume that constraints hold precisely, supporting maximum entropy inference more in principle than in practice. We present, for the first time, non-asymptotic, explicit lower bounds on $n$ for a number of variants of the concentration result to hold to any prescribed accuracies, with the constraints holding up to any specified tolerance, taking into account the fact that allocations of discrete units can satisfy constraints only approximately. Again unlike earlier results, we measure concentration not by deviation from the maximum entropy value, but by the $\ell_1$ and $\ell_2$ distances from the maximum entropy-achieving frequency vector. One of our results holds independently of the alphabet size $m$ and is based on a novel proof technique using the multi-dimensional Berry-Esseen theorem. We illustrate and compare our results using various detailed examples.
Entropy Functions on Two-Dimensional Faces of Polymatroidal Region of Degree Four: Part II: Information Theoretic Constraints Breed New Combinatorial Structures
Xidian University(西安电子科技大学);Guangxi Normal University(广西师范大学)
Maximum Entropy Rate of Markov Sources for Systems With Non-regular Constraints
使用离散无噪声信道的概念,Shannon在A Mathematical Theory of Communication中表明,如果约束定义了正则语言,则用于约束系统的编码器的最终性能受到系统的组合容量的限制。在目前的工作中,它表明,这不是一个固有的性质的规律性,但持有一般。为了证明这一点,约束系统描述的生成函数和随机行走的树。