arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1502.03296physics.soc-phcs.LGphysics.data-an
收藏

语言学中的统计规律

Statistical laws in linguistics

Max Planck Institute for the Physics of Complex Systems(马克斯·普朗克复杂系统物理研究所)

Eduardo G. Altmann, Martin Gerlach

摘要:

齐普夫定律只是众多用语言描述统计学的普适定律之一。在这里,我们回顾和批判性地讨论如何这些法律可以统计解释,拟合和测试(证伪)。现代的大型书面文本数据库的可用性允许测试具有前所未有的统计准确性,也可以表征典型行为周围的波动。我们发现,波动通常比基于简化统计假设的预期大得多(例如,这些简化也出现在通常的统计检验中,因此大的波动可能被错误地解释为对定律的证伪。相反,在这里我们认为,语言规律只有在伴随着一个可以计算波动的模型时才有意义(可证伪)(例如,文本的生成模型)。我们报告的大幅波动表明,语言规律对文本生成的创造性过程施加的约束并不像人们所期望的那样严格。

英文摘要:

Zipf's law is just one out of many universal laws proposed to describe statistical regularities in language. Here we review and critically discuss how these laws can be statistically interpreted, fitted, and tested (falsified). The modern availability of large databases of written text allows for tests with an unprecedent statistical accuracy and also a characterization of the fluctuations around the typical behavior. We find that fluctuations are usually much larger than expected based on simplifying statistical assumptions (e.g., independence and lack of correlations between observations).These simplifications appear also in usual statistical tests so that the large fluctuations can be erroneously interpreted as a falsification of the law. Instead, here we argue that linguistic laws are only meaningful (falsifiable) if accompanied by a model for which the fluctuations can be computed (e.g., a generative model of the text). The large fluctuations we report show that the constraints imposed by linguistic laws on the creativity process of text generation are not as tight as one could expect.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑