Beyond Zipf's law: Modeling the structure of human language
IFISC (CSIC-UIB) Instituto de Física Interdisciplinar y Sistemas Complejos（IFISC（西班牙国家研究委员会-巴利阿里群岛大学）跨学科物理与复杂系统研究所）；Indiana University（印第安纳大学）；ISI Foundation（ISI基金会）
M. Angeles Serrano, Alessandro Flammini, Filippo Menczer
Human language, the most powerful communication system in history, is closely associated with cognition. Written text is one of the fundamental manifestations of language, and the study of its universal regularities can give clues about how our brains process information and how we, as a society, organize and share it. Still, only classical patterns such as Zipf's law have been explored in depth. In contrast, other basic properties like the existence of bursts of rare words in specific documents, the topical organization of collections, or the sublinear growth of vocabulary size with the length of a document, have only been studied one by one and mainly applying heuristic methodologies rather than basic principles and general mechanisms. As a consequence, there is a lack of understanding of linguistic processes as complex emergent phenomena. Beyond Zipf's law for word frequencies, here we focus on Heaps' law, burstiness, and the topicality of document collections, which encode correlations within and across documents absent in random null models. We introduce and validate a generative model that explains the simultaneous emergence of all these patterns from simple rules. As a result, we find a connection between the bursty nature of rare words and the topical organization of texts and identify dynamic word ranking and memory across documents as key mechanisms explaining the non trivial organization of written text. Our research can have broad implications and practical applications in computer science, cognitive science, and linguistics.
On the class of coding optimality of human languages and the origins of Zipf's law
Quantitative, Mathematical and Computational Linguistics Research Group, Department of Computer Science, Universitat Politècnica de Catalunya, 08034 Barcelona, Catalonia (Spain)（计算机科学系量化、数学和计算语言学研究组，巴塞罗那理工大学，西班牙加泰罗尼亚地区08034）
Communication University of China（中国传媒大学）；Anhui Jianzhu University（安徽建筑大学）；Zhejiang University（浙江大学）；Ningbo Institute of Technology, Zhejiang University（浙江大学宁波理工学院）
Zipf's law has been found in many human-related fields, including language, where the frequency of a word is persistently found as a power law function of its frequency rank, known as Zipf's law. However, there is much dispute whether it is a universal law or a statistical artifact, and little is known about what mechanisms may have shaped it. To answer these questions, this study conducted a large scale cross language investigation into Zipf's law. The statistical results show that Zipf's laws in 50 languages all share a 3-segment structural pattern, with each segment demonstrating distinctiv…
We demonstrate that large texts, representing human (English, Russian, Ukrainian) and artificial (C++, Java) languages, display quantitative patterns characterized by the Benford-like and Zipf laws. The frequency of a word following the Zipf law is inversely proportional to its rank, whereas the total numbers of a certain word appearing in the text generate the uneven Benford-like distribution of leading numbers. Excluding the most popular words essentially improves the correlation of actual textual data with the Zipfian distribution, whereas the Benford distribution of leading numbers (arisin…
Universitat Politècnica de Catalunya（加泰罗尼亚理工大学）；University of Kansas（堪萨斯大学）
In his pioneering research, G. K. Zipf observed that more frequent words tend to have more meanings, and showed that the number of meanings of a word grows as the square root of its frequency. He derived this relationship from two assumptions: that words follow Zipf's law for word frequencies (a power law dependency between frequency and rank) and Zipf's law of meaning distribution (a power law dependency between number of meanings and rank). Here we show that a single assumption on the joint probability of a word and a meaning suffices to infer Zipf's meaning-frequency law or relaxed versions…
Zipf's law for word frequencies: word forms versus lemmas in long texts
Centre de Recerca Matemàtica（数学研究中心）；Universitat Autònoma de Barcelona（巴塞罗那自治大学）；Universitat Pompeu Fabra（庞培法布拉大学）；Universitat Politècnica de Catalunya（加泰罗尼亚理工大学）
In his pioneering research, G. K. Zipf formulated a couple of statistical laws on the relationship between the frequency of a word with its number of meanings: the law of meaning distribution, relating the frequency of a word and its frequency rank, and the meaning-frequency law, relating the frequency of a word with its number of meanings. Although these laws were formulated more than half a century ago, they have been only investigated in a few languages. Here we present the first study of these laws in Catalan. We verify these laws in Catalan via the relationship among their exponents and t…
Central China Normal University（华中师范大学）；Yerevan Physics Institute（埃里温物理研究所）
Zipf's law is the main regularity of quantitative linguistics. Despite of many works devoted to foundations of this law, it is still unclear whether it is only a statistical regularity, or it has deeper relations with information-carrying structures of the text. This question relates to that of distinguishing a meaningful text (written in an unknown system) from a meaningless set of symbols that mimics statistical features of a text. Here we contribute to resolving these questions by comparing features of the first half of a text (from the beginning to the middle) to its second half. This comp…
Pre-trained Models Perform the Best When Token Distributions Follow Zipf's Law
School of Mathematical Sciences, Peking University（北京大学数学科学学院）；Department of Computer Science and Engineering, University of Notre Dame（圣母大学计算机科学与工程系）；College of Computer Science, Nankai University（南开大学计算机学院）
Australian Institute of Health Innovation, Macquarie University（麦考瑞大学澳大利亚健康创新研究所）；Centre for Big Data Research in Health, UNSW（新南威尔士大学健康大数据研究中心）；University of Sydney（悉尼大学）；University of Technology Sydney（悉尼科技大学）；Swinburne University of Technology（斯威本科技大学）
Bayesian modelling and statistical text analysis rely on informed probability priors to encourage good solutions. This paper empirically analyses whether text in medical discharge reports follow Zipf's law, a commonly assumed statistical property of language where word frequency follows a discrete power law distribution. We examined 20,000 medical discharge reports from the MIMIC-III dataset. Methods included splitting the discharge reports into tokens, counting token frequency, fitting power law distributions to the data, and testing whether alternative distributions--lognormal, exponential, …
The pioneering research of G. K. Zipf on the relationship between word frequency and other word features led to the formulation of various linguistic laws. The most popular is Zipf's law for word frequencies. Here we focus on two laws that have been studied less intensively: the meaning-frequency law, i.e. the tendency of more frequent words to be more polysemous, and the law of abbreviation, i.e. the tendency of more frequent words to be shorter. In a previous work, we tested the robustness of these Zipfian laws for English, roughly measuring word length in number of characters and distinguis…
A Zipf's Law-based Text Generation Approach for Addressing Imbalance in Entity Extraction
School of Information Resource Management, Renmin University of China（中国人民大学信息资源管理学院）；College of Information Science and Technology, Beijing University of Chemical Technology（北京化工大学信息科学与技术学院）
Novosibirsk State Technical University（新西伯利亚国立技术大学）；Novosibirsk State University（新西伯利亚国立大学）；Nankai University（南开大学）；Sobolev Institute of Mathematics（索博列夫数学研究所）
We analyse correspondence of a text to a simple probabilistic model. The model assumes that the words are selected independently from an infinite dictionary. The probability distribution correspond to the Zipf---Mandelbrot law. We count sequentially the numbers of different words in the text and get the process of the numbers of different words. Then we estimate Zipf---Mandelbrot law parameters using the same sequence and construct an estimate of the expectation of the number of different words in the text. Then we subtract the corresponding values of the estimate from the sequence and normali…
Zoomi, Inc（Zoomi公司）；College of William & Mary（威廉与玛丽学院）；Peking University（北京大学）；Purdue University（普渡大学）
Despite the recent successes of deep learning in natural language processing (NLP), there remains widespread usage of and demand for techniques that do not rely on machine learning. The advantage of these techniques is their interpretability and low cost when compared to frequently opaque and expensive machine learning models. Although they may not be be as performant in all cases, they are often sufficient for common and relatively simple problems. In this paper, we aim to modernize these older methods while retaining their advantages by extending approaches from categorical or bag-of-words r…
The performance of deep learning in natural language processing has been spectacular, but the reasons for this success remain unclear because of the inherent complexity of deep learning. This paper provides empirical evidence of its effectiveness and of a limitation of neural networks for language engineering. Precisely, we demonstrate that a neural language model based on long short-term memory (LSTM) effectively reproduces Zipf's law and Heaps' law, two representative statistical properties underlying natural language. We discuss the quality of reproducibility and the emergence of Zipf's law…
The frequencies at which individual words occur across languages follow power law distributions, a pattern of findings known as Zipf's law. A vast literature argues over whether this serves to optimize the efficiency of human communication, however this claim is necessarily post hoc, and it has been suggested that Zipf's law may in fact describe mixtures of other distributions. From this perspective, recent findings that Sinosphere first (family) names are geometrically distributed are notable, because this is actually consistent with information theoretic predictions regarding optimal coding.…
National Research University Higher School of Economics（国立研究型大学高等经济学院）
This paper studies the limits of language models' statistical learning in the context of Zipf's law. First, we demonstrate that Zipf-law token distribution emerges irrespective of the chosen tokenization. Second, we show that Zipf distribution is characterized by two distinct groups of tokens that differ both in terms of their frequency and their semantics. Namely, the tokens that have a one-to-one correspondence with one semantic concept have different statistical properties than those with semantic ambiguity. Finally, we demonstrate how these properties interfere with statistical learning pr…
Laboratoire de Physique Statistique et Systèmes Complexes, ISMANS（ISMANS统计物理与复杂系统实验室）；Hua-Zhong Normal University（华中师范大学）；Université du Maine（曼恩大学）；Yerevan Physics Institute（埃里温物理研究所）；University of Heilongjiang（黑龙江大学）
