arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1407.4610physics.soc-phcs.CL
收藏

从句子构成中的样本空间塌陷理解齐普夫词频定律

Understanding Zipf's law of word frequencies through sample-space collapse in sentence formation

Medical University of Vienna(维也纳医科大学);Santa Fe Institute(圣塔菲研究所);IIASA(国际应用系统分析研究所)

Stefan Thurner, Rudolf Hanel, Bo Liu, Bernat Corominas-Murtra

摘要:

句子的形成是一个高度结构化和历史依赖的过程。在句子中使用特定单词的概率很大程度上取决于该句子中单词使用的“历史”。我们研究了一个简单的历史依赖模型的文本生成假设的样本空间的词的使用减少沿句子的形成,平均。我们首先表明,该模型解释了近似的Zipf定律发现在词频作为样本空间减少的直接后果。然后,我们实证量化的样本空间减少量的句子中的10本著名的英语书籍,通过分析相应的单词转换表,捕捉哪些单词可以遵循任何给定的单词在一个文本中。我们发现一个高度嵌套的结构,在这些过渡表,并表明这种“嵌套”是密切相关的幂律指数所观察到的词频分布。利用所提出的模型,可以理解文本的嵌套性可以是实际标度指数的起源,并且可以通过逐本书的嵌套程度的变化来理解与精确的齐普夫定律的偏差。在理论层面上,我们能够证明,在嵌套较弱的情况下,齐普夫定律会在快速转变中崩溃。与以前试图理解语言中的齐普夫定律不同,样本空间缩减模型不是基于语言形成背后的乘法、偏好或自组织批判机制的假设,而是简单地使用经验上可量化的参数“嵌套性”来理解词频的统计。

英文摘要:

The formation of sentences is a highly structured and history-dependent process. The probability of using a specific word in a sentence strongly depends on the 'history' of word-usage earlier in that sentence. We study a simple history-dependent model of text generation assuming that the sample-space of word usage reduces along sentence formation, on average. We first show that the model explains the approximate Zipf law found in word frequencies as a direct consequence of sample-space reduction. We then empirically quantify the amount of sample-space reduction in the sentences of ten famous English books, by analysis of corresponding word-transition tables that capture which words can follow any given word in a text. We find a highly nested structure in these transition tables and show that this `nestedness' is tightly related to the power law exponents of the observed word frequency distributions. With the proposed model it is possible to understand that the nestedness of a text can be the origin of the actual scaling exponent, and that deviations from the exact Zipf law can be understood by variations of the degree of nestedness on a book-by-book basis. On a theoretical level we are able to show that in case of weak nesting, Zipf's law breaks down in a fast transition. Unlike previous attempts to understand Zipf's law in language the sample-space reducing model is not based on assumptions of multiplicative, preferential, or self-organised critical mechanisms behind language formation, but simply used the empirically quantifiable parameter 'nestedness' to understand the statistics of word frequencies.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑