可数离散分布的贝叶斯熵估计
Bayesian Entropy Estimation for Countable Discrete Distributions
摘要：
我们考虑的问题估计香农熵$H$从离散数据，在可能的符号的数量是未知的，甚至可数无限的情况下。Pitman-Yor过程是Dirichlet过程的推广，它提供了可数无限离散分布空间上的一个易于处理的先验分布，在贝叶斯非参数统计和机器学习中有着重要的应用。在这里，我们表明，它也提供了一个自然的家庭先验贝叶斯熵估计，由于这一事实，即在$H$的诱导后验分布的时刻可以计算分析。我们推导出公式的后验均值（贝叶斯最小二乘估计）和方差Dirichlet和Pitman-Yor过程先验。此外，我们发现，一个固定的Dirichlet或Pitman-Yor过程的先验意味着一个狭窄的先验分布在$H$，这意味着先验强有力地决定了熵估计在欠采样制度。我们得到一个家庭的连续混合措施，使Pitman-Yor过程的混合物产生一个近似平坦的先验$H$。我们表明，由此产生的Pitman-Yor混合（PYM）熵估计是一致的一大类分布。我们探索的理论性质的估计，并表明，它表现良好，无论是在模拟和实际数据的应用。
英文摘要：
We consider the problem of estimating Shannon's entropy $H$ from discrete data, in cases where the number of possible symbols is unknown or even countably infinite. The Pitman-Yor process, a generalization of Dirichlet process, provides a tractable prior distribution over the space of countably infinite discrete distributions, and has found major applications in Bayesian non-parametric statistics and machine learning. Here we show that it also provides a natural family of priors for Bayesian entropy estimation, due to the fact that moments of the induced posterior distribution over $H$ can be computed analytically. We derive formulas for the posterior mean (Bayes' least squares estimate) and variance under Dirichlet and Pitman-Yor process priors. Moreover, we show that a fixed Dirichlet or Pitman-Yor process prior implies a narrow prior distribution over $H$, meaning the prior strongly determines the entropy estimate in the under-sampled regime. We derive a family of continuous mixing measures such that the resulting mixture of Pitman-Yor processes produces an approximately flat prior over $H$. We show that the resulting Pitman-Yor Mixture (PYM) entropy estimator is consistent for a large class of distributions. We explore the theoretical properties of the resulting estimator, and show that it performs well both in simulation and in application to real data.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
