文本混合塑造了等级频率分布的解剖学:自然语言的现代齐普菲机制
Text mixing shapes the anatomy of rank-frequency distributions: A modern Zipfian mechanics for natural language
摘要:
自然语言充满了规则和例外。最著名的数量规则之一是齐普夫定律,它指出一个词出现的频率与它的等级近似成反比。虽然这种排名的“法律”已被发现持有不同的文本和形式的数据,越来越大的语料库在过去15年的分析表明,存在两个缩放制度。到目前为止,这些制度已经解释了一个假设,表明语言分为核心和非核心词汇的分离。在这里,我们提出并捍卫另一种假设,这两个缩放制度的聚合文本的行为的结果。我们观察到,文本混合导致一个有效的衰减的字介绍,我们表明,提供了准确的预测的位置和严重程度的休息缩放。在研究古腾堡电子书项目(电子书)中10种语言的大型语料库后,我们发现了对我们主张的普遍性的实证支持。
英文摘要:
Natural languages are full of rules and exceptions. One of the most famous quantitative rules is Zipf's law which states that the frequency of occurrence of a word is approximately inversely proportional to its rank. Though this `law' of ranks has been found to hold across disparate texts and forms of data, analyses of increasingly large corpora over the last 15 years have revealed the existence of two scaling regimes. These regimes have thus far been explained by a hypothesis suggesting a separability of languages into core and non-core lexica. Here, we present and defend an alternative hypothesis, that the two scaling regimes result from the act of aggregating texts. We observe that text mixing leads to an effective decay of word introduction, which we show provides accurate predictions of the location and severity of breaks in scaling. Upon examining large corpora from 10 languages in the Project Gutenberg eBooks collection (eBooks), we find emphatic empirical support for the universality of our claim.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。