arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1702.01666cs.ITcs.CCmath.IT
收藏

论Renyi分歧估计的复杂性

On the Complexity of Estimating Renyi Divergences

Maciej Skorski

摘要:

本文研究了离散分布的Renyi发散估计的复杂性:$p$从样本中观察到的和基线分布$q$已知\emph{a priori}。扩展Acharya等人的结果。(SODA'15)估计Renyi熵,我们提出了改进的估计技术,以及上界和下界的样本复杂性。我们发现,相反,估计Renyi熵的次线性(在字母表大小)的样本数就足够了,样本的复杂性是严重依赖于\emph{events occurring unlikely}在$q$,一般是无界的(无论使用什么估计技术)。对于大于$1$阶的任何发散,我们提供了依赖于$p$和$q$概率的样本数量的上界和下界。我们的结论是,最坏情况下的样本复杂度是多项式的字母表大小当且仅当概率的$q$是不可忽略的。这给出了理论上的见解,在应用论文中使用的算法来处理数值不稳定性,这发生在小概率的$q$。我们的结果解释了小概率应该小心处理,不仅因为数值问题,而且因为在样本复杂性的爆炸。

英文摘要:

This paper studies the complexity of estimating Renyi divergences of discrete distributions: $p$ observed from samples and the baseline distribution $q$ known \emph{a priori}. Extending the results of Acharya et al. (SODA'15) on estimating Renyi entropy, we present improved estimation techniques together with upper and lower bounds on the sample complexity. We show that, contrarily to estimating Renyi entropy where a sublinear (in the alphabet size) number of samples suffices, the sample complexity is heavily dependent on \emph{events occurring unlikely} in $q$, and is unbounded in general (no matter what an estimation technique is used). For any divergence of order bigger than $1$, we provide upper and lower bounds on the number of samples dependent on probabilities of $p$ and $q$. We conclude that the worst-case sample complexity is polynomial in the alphabet size if and only if the probabilities of $q$ are non-negligible. This gives theoretical insights into heuristics used in applied papers to handle numerical instability, which occurs for small probabilities of $q$. Our result explains that small probabilities should be handled with care not only because of numerical issues, but also because of a blow up in sample complexity.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑