arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1406.5181cs.CLphysics.soc-ph
收藏

齐普夫定律适用于短语,而不是单词

Zipf's law holds for phrases, not words

The University of Vermont(佛蒙特大学);University of Colorado, Boulder(科罗拉多大学博尔德分校);Massachusetts Institute of Technology(麻省理工学院)

Jake Ryland Williams, Paul R. Lessard, Suma Desu, Eric Clark, James P. Bagrow, Christopher M. Danforth, Peter Sheridan Dodds

摘要:

由于齐普夫定律最初也是最著名的词频定律,它对人类语言的适用性令人惊讶地有限,在尺度上出现明显的突破之前,它的适用性不超过三到四个数量级。在这里,建立在简单的观察,一个或多个单词的短语组成的最连贯的语言单位的意义,我们经验表明,齐普夫定律的短语扩展到多达九个数量级的等级。在这样做的过程中,我们开发了一个原则性和可扩展的随机文本分区的统计机械方法,它开辟了严格的文本分析通过混合长度短语的排名排序丰富的前沿。

英文摘要:

With Zipf's law being originally and most famously observed for word frequency, it is surprisingly limited in its applicability to human language, holding over no more than three to four orders of magnitude before hitting a clear break in scaling. Here, building on the simple observation that phrases of one or more words comprise the most coherent units of meaning in language, we show empirically that Zipf's law for phrases extends over as many as nine orders of rank magnitude. In doing so, we develop a principled and scalable statistical mechanical method of random text partitioning, which opens up a rich frontier of rigorous text analysis via a rank ordering of mixed length phrases.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑