英语文本中齐普夫定律的大规模分析
Large-scale analysis of Zipf's law in English texts
摘要：
尽管齐普夫定律是定量语言学的一个范例，但它存在三个主要问题：它的表述是模糊的，它的有效性没有从统计的角度进行严格的检验，它没有被大量的文本所面对。因此，我们可以将目前Zipf定律在文本中的支持总结为轶事。我们试图解决这些问题，通过研究三个不同版本的齐普夫定律，并适合他们的古腾堡项目数据库（包括30000多个文本）中的所有可用的英文文本。为此，我们在拟合和拟合优度测试中使用了最先进的工具，并根据文本统计的特殊性进行了精心定制。值得注意的是，齐普夫定律的三个版本之一，由一个纯幂律形式的互补累积分布函数的词频，是能够适应超过40%的文本在数据库中（在0.05的显着性水平），为整个域的频率（从1到最大值），只有一个自由参数（指数）。
英文摘要：
Despite being a paradigm of quantitative linguistics, Zipf's law for words suffers from three main problems: its formulation is ambiguous, its validity has not been tested rigorously from a statistical point of view, and it has not been confronted to a representatively large number of texts. So, we can summarize the current support of Zipf's law in texts as anecdotic. We try to solve these issues by studying three different versions of Zipf's law and fitting them to all available English texts in the Project Gutenberg database (consisting of more than 30000 texts). To do so we use state-of-the art tools in fitting and goodness-of-fit tests, carefully tailored to the peculiarities of text statistics. Remarkably, one of the three versions of Zipf's law, consisting of a pure power-law form in the complementary cumulative distribution function of word frequencies, is able to fit more than 40% of the texts in the database (at the 0.05 significance level), for the whole domain of frequencies (from 1 to the maximum value) and with only one free parameter (the exponent).
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
