arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2008.00903stat.AP
收藏

Zipf定律估计量的偏差

Bias in Zipf's Law Estimators

Charlie Pilgrim, Thomas T Hills

摘要:

从秩-频数据推断幂律模型的流行极大似然估计是有偏的。这种偏差的来源是不适当的似然函数。正确的似然函数的推导和证明是计算上棘手的。探讨了一种计算效率更高的近似贝叶斯计算方法。这种方法被证明有较少的偏差,从理想化的秩-频率齐普菲分布产生的数据。然而,现有的估计和ABC估计这里描述的假设,词是从一个简单的概率分布,而语言是一个复杂得多的过程。我们表明,这种错误的假设会导致持续的偏见时,将这些方法应用到自然语言估计Zipf指数。我们建议研究人员在调查秩频数据中的幂律时注意这些偏差。

英文摘要:

The prevailing maximum likelihood estimators for inferring power law models from rank-frequency data are biased. The source of this bias is an inappropriate likelihood function. The correct likelihood function is derived and shown to be computationally intractable. A more computationally efficient method of approximate Bayesian computation (ABC) is explored. This method is shown to have less bias for data generated from idealised rank-frequency Zipfian distributions. However, the existing estimators and the ABC estimator described here assume that words are drawn from a simple probability distribution, while language is a much more complex process. We show that this false assumption leads to continued biases when applying any of these methods to natural language to estimate Zipf exponents. We recommend that researchers be aware of these biases when investigating power laws in rank-frequency data.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑