Zipf's law is just one out of many universal laws proposed to describe statistical regularities in language. Here we review and critically discuss how these laws can be statistically interpreted, fitted, and tested (falsified). The modern availability of large databases of written text allows for tests with an unprecedent statistical accuracy and also a characterization of the fluctuations around the typical behavior. We find that fluctuations are usually much larger than expected based on simplifying statistical assumptions (e.g., independence and lack of correlations between observations).These simplifications appear also in usual statistical tests so that the large fluctuations can be erroneously interpreted as a falsification of the law. Instead, here we argue that linguistic laws are only meaningful (falsifiable) if accompanied by a model for which the fluctuations can be computed (e.g., a generative model of the text). The large fluctuations we report show that the constraints imposed by linguistic laws on the creativity process of text generation are not as tight as one could expect.
Technical University of Madrid(马德里理工大学);Queen Mary University of London(伦敦玛丽女王大学);Telefonica Research(西班牙电信研究院);Universitat Politècnica de Catalunya(加泰罗尼亚理工大学)
Statistical physics of language change inferred from time evolving maps
School of Mathematics and Physics, University of Portsmouth(朴茨茅斯大学数学与物理学院);Department of Theoretical & Applied Linguistics, University of Cambridge(剑桥大学理论与应用语言学系)
Statistical Laws of Human Mobility under Large-Scale Behavioural Shock
University of Bologna(博洛尼亚大学);Fondazione Bruno Kessler(布鲁诺·凯斯勒基金会);University of Padua(帕多瓦大学);Istituto Nazionale di Fisica Nucleare, Sez. Padova(意大利国家核物理研究所帕多瓦分部);Cuebiq Inc.(Cuebiq公司);University of California, Berkeley(加州大学伯克利分校)
Department of Computational Biology(计算生物学系);Infosys Center for Artificial Intelligence(Infosys人工智能中心);Center of Excellence in Healthcare(医疗卓越中心);Department of Mathematics(数学系);Department of Computer Science(计算机科学系);Indraprastha Institute of Information Technology Delhi (IIIT-Delhi)(德里印度教徒普拉斯塔信息技术学院(IIIT-Delhi));Foodoscope Technologies Private Limited(Foodoscope技术私人有限公司)
G. K.齐普夫对词频与其他词的特征之间关系的研究导致了各种语言学规律的制定。最流行的是Zipf的词频定律。在这里,我们集中在两个法律,已经研究得不太深入:意义频率的法律,即更频繁的词是多义的趋势,和缩写的法律,即更频繁的词是更短的趋势。在以前的工作中,我们测试了这些Zipfian定律对英语的鲁棒性,粗略地测量了字长的字符数,并区分了成人和儿童的语音。在本文中,我们将我们的研究扩展到其他语言(荷兰语和西班牙语),并介绍了两个额外的长度措施:音节长度和音素长度。相关分析表明,意义频率规律和缩略语规律在所有被分析的语言中总体上是一致的。
Individual Biases, Cultural Evolution, and the Statistical Nature of Language Universals: The Case of Colour Naming Systems
City University London(伦敦城市大学);Sapienza Università di Roma(罗马大学);Institute for Scientific Interchange (ISI)(科学交流研究所(ISI));SONY Computer Science Lab (SONY-CSL)(索尼计算机科学实验室(SONY-CSL));CNR-ISC(意大利国家研究委员会复杂系统研究所(CNR-ISC))