Word frequency is assumed to correlate with word familiarity, but the strength of this correlation has not been thoroughly investigated. In this paper, we report on our analysis of the correlation between a word familiarity rating list obtained through a psycholinguistic experiment and the log-frequency obtained from various corpora of different kinds and sizes (up to the terabyte scale) for English and Japanese. Major findings are threefold: First, for a given corpus, familiarity is necessary for a word to achieve high frequency, but familiar words are not necessarily frequent. Second, correlation increases with the corpus data size. Third, a corpus of spoken language correlates better than one of written language. These findings suggest that cognitive familiarity ratings are correlated to frequency, but more highly to that of spoken rather than written language.
Indian Institute of Science (IISc)（印度科学学院）；IIIT Hyderabad（印度海得拉巴国际信息技术学院）
In this study, listeners of varied Indian nativities are asked to listen and recognize TIMIT utterances spoken by American speakers. We have three kinds of responses from each listener while they recognize an utterance: 1. Sentence difficulty ratings, 2. Speaker difficulty ratings, and 3. Transcription of the utterance. From these transcriptions, word error rate (WER) is calculated and used as a metric to evaluate the similarity between the recognized and the original sentences.The sentences selected in this study are categorized into three groups: Easy, Medium and Hard, based on the frequency…
Investigating the Frequency Distortion of Word Embeddings and Its Impact on Bias Metrics
Instituto de Investigación en Ciencias de la Computación, CONICET-UBA（CONICET-布宜诺斯艾利斯大学计算机科学研究所）；Universidad de Buenos Aires (UBA)（布宜诺斯艾利斯大学（UBA））；Departamento de Computación, FCEyN, UBA（布宜诺斯艾利斯大学理学院计算机系）
How does word frequency in pre-training data affect the behavior of similarity metrics in contextualized BERT embeddings? Are there systematic ways in which some word relationships are exaggerated or understated? In this work, we explore the geometric characteristics of contextualized word embeddings with two novel tools: (1) an identity probe that predicts the identity of a word using its embedding; (2) the minimal bounding sphere for a word's contextualized representations. Our results reveal that words of high and low frequency differ significantly with respect to their representational geo…
National Institute of Information and Communications Technology（国立信息与通信技术研究所）
Let $f (\cdot)$ be the absolute frequency of words and $r$ be the rank of words in decreasing order of frequency, then the following function can fit the rank-frequency relation \[ f (r;s,t) = \left(\frac{r_{\tt max}}{r}\right)^{1-s} \left(\frac{r_{\tt max}+t \cdot r_{\tt exp}}{r+t \cdot r_{\tt exp}}\right)^{1+(1+t)s} \] where $r_{\tt max}$ and $r_{\tt exp}$ are the maximum and the expectation of the rank, respectively; $s>0$ and $t>0$ are parameters estimated from data. On well-behaved data, there should be $s<1$ and $s \cdot t < 1$.
University College London（伦敦大学学院）；Tilburg University（蒂尔堡大学）
Recent efforts have shown that neural text processing models are vulnerable to adversarial examples, but the nature of these examples is poorly understood. In this work, we show that adversarial attacks against CNN, LSTM and Transformer-based classification models perform word substitutions that are identifiable through frequency differences between replaced words and their corresponding substitutions. Based on these findings, we propose frequency-guided word substitutions (FGWS), a simple algorithm exploiting the frequency properties of adversarial word substitutions for the detection of adve…
Instituto de Física Enrique Gaviola, Consejo Nacional de Investigaciones Científicas y Técnicas（恩里克·加维奥拉物理研究所，国家科学技术研究委员会）；Universidad Nacional de Córdoba（科尔多瓦国立大学）；Centro Atómico Bariloche（巴里洛切原子能中心）；Instituto Balseiro（巴尔塞iro研究所）；Comisión Nacional de Energía Atómica（国家原子能委员会）；Universidad Nacional de Cuyo（库约国立大学）；Consejo Nacional de Investigaciones Científicas y Técnicas（国家科学技术研究委员会）
We analyze the frequency-rank relationship in sub-vocabularies corresponding to three different grammatical classes (nouns, verbs, and others) in a collection of literary works in English, whose words have been automatically tagged according to their grammatical role. Comparing with a null hypothesis which assumes that words belonging to each class are uniformly distributed across the frequency-ranked vocabulary of the whole work, we disclose statistically significant differences between the three classes. This results point to the fact that frequency-rank relationships may reflect linguistic …
National Engineering Research Center for Software Engineering, Peking University（北京大学国家软件工程研究中心）；Alibaba Group（阿里巴巴集团）
Low-frequency word prediction remains a challenge in modern neural machine translation (NMT) systems. Recent adaptive training methods promote the output of infrequent words by emphasizing their weights in the overall training objectives. Despite the improved recall of low-frequency words, their prediction precision is unexpectedly hindered by the adaptive objectives. Inspired by the observation that low-frequency words form a more compact embedding space, we tackle this challenge from a representation learning perspective. Specifically, we propose a frequency-aware token-level contrastive lea…
Restricted Recurrent Neural Tensor Networks: Exploiting Word Frequency and Compositionality
Institute of Informatics, Federal University of Rio Grande do Sul（联邦南里奥格兰德大学信息学院）；School of Computer Science and Electronic Engineering, University of Essex（埃塞克斯大学计算机科学与电子工程学院）
HHMI Janelia Research Campus（HHMI珍妮亚研究园区）；Queen’s University Belfast（贝尔法斯特女王大学）
Quite often, words from one language are adopted within a different language without translation; these words appear in transliterated form in text written in the latter language. This phenomenon is particularly widespread within Indian languages where many words are loaned from English. In this paper, we address the task of identifying loanwords automatically and in an unsupervised manner, from large datasets of words from agglutinative Dravidian languages. We target two specific languages from the Dravidian family, viz., Malayalam and Telugu. Based on familiarity with the languages, we outli…
Universidad de Buenos Aires（布宜诺斯艾利斯大学）；CONICET（阿根廷国家科学与技术研究理事会）；Academia Argentina de Letras（阿根廷文学院）
The task of detecting regionalisms (expressions or words used in certain regions) has traditionally relied on the use of questionnaires and surveys, and has also heavily depended on the expertise and intuition of the surveyor. The irruption of Social Media and its microblogging services has produced an unprecedented wealth of content, mainly informal text generated by users, opening new opportunities for linguists to extend their studies of language variation. Previous work on automatic detection of regionalisms depended mostly on word frequencies. In this work, we present a novel metric based…
Neural language models learn, to varying degrees of accuracy, the grammatical properties of natural languages. In this work, we investigate whether there are systematic sources of variation in the language models' accuracy. Focusing on subject-verb agreement and reflexive anaphora, we find that certain nouns are systematically understood better than others, an effect which is robust across grammatical tasks and different language models. Surprisingly, we find that across four orders of magnitude, corpus frequency is unrelated to a noun's performance on grammatical tasks. Finally, we find that …
Improving Retrieval Modeling Using Cross Convolution Networks And Multi Frequency Word Embedding
Department of Computer Science, CUNY Graduate Center, USA（纽约市立大学研究生中心计算机科学系，美国）；Department of Mathematics and Computer Science, York College (CUNY), USA（纽约市立大学约克学院数学与计算机科学系，美国）；Department of Electrical Engineering, Columbia University, USA（哥伦比亚大学电气工程系，美国）；OATH Inc, USA（OATH公司，美国）
ETH Zürich（苏黎世联邦理工学院）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）
Languages are continuously undergoing changes, and the mechanisms that underlie these changes are still a matter of debate. In this work, we approach language evolution through the lens of causality in order to model not only how various distributional factors associate with language change, but how they causally affect it. In particular, we study slang, which is an informal language that is typically restricted to a specific group or social setting. We analyze the semantic change and frequency shift of slang words and compare them to those of standard, nonslang words. With causal discovery an…
Understanding how words change their meanings over time is key to models of language and cultural evolution, but historical data on meaning is scarce, making theories hard to develop and test. Word embeddings show promise as a diachronic tool, but have not been carefully evaluated. We develop a robust methodology for quantifying semantic change by evaluating word embeddings (PPMI, SVD, word2vec) against known historical changes. We then use this methodology to reveal statistical laws of semantic evolution. Using six historical corpora spanning four languages and two centuries, we propose two q…
ICC (UBA-CONICET)（ICC（布宜诺斯艾利斯大学-国家科学技术研究委员会））；Maestría en Data Mining (UBA)（数据挖掘硕士项目（布宜诺斯艾利斯大学））；Escuela IDAES (UNSAM)（IDAES学院（圣马丁将军大学））
Numerous works use word embedding-based metrics to quantify societal biases and stereotypes in texts. Recent studies have found that word embeddings can capture semantic similarity but may be affected by word frequency. In this work we study the effect of frequency when measuring female vs. male gender bias with word embedding-based bias quantification methods. We find that Skip-gram with negative sampling and GloVe tend to detect male bias in high frequency words, while GloVe tends to return female bias in low frequency words. We show these behaviors still exist when words are randomly shuffl…
Word forms - not just their lengths- are optimized for efficient communication
University of California, Berkeley（加州大学伯克利分校）
一个词的长度和它的使用频率之间的反比关系，最早由G. K. Zipf在1935年提出，是一个经典的经验法则，适用于各种人类语言。我们证明，长度是一个更普遍的属性的一个方面的话：他们是如何独特的相对于其他词在一种语言。区别性在识别流利语音中的单词中起着关键作用，因为它反映了在为模糊信号选择最佳候选时潜在竞争者的实力。语音信息量是在一种语言的声音或字符序列的统计模型下对单词的字符串概率的测量，它简明地捕捉了独特性。通过对13种语言的大规模语料库的分析，我们发现独特性在预测频率方面明显优于字长。这一发现提供了证据表明，听众的处理限制形状细粒度方面的词的形式跨越语言。
Stanford University（斯坦福大学）；University of Michigan（密歇根大学）
Cosine similarity of contextual embeddings is used in many NLP tasks (e.g., QA, IR, MT) and metrics (e.g., BERTScore). Here, we uncover systematic ways in which word similarities estimated by cosine over BERT embeddings are understated and trace this effect to training data frequency. We find that relative to human judgements, cosine similarity underestimates the similarity of frequent words with other instances of the same word or other words across contexts, even after controlling for polysemy and other factors. We conjecture that this underestimation of similarity for high frequency words i…
Stanford University（斯坦福大学）；Saarland University（萨尔大学）；University of Toronto（多伦多大学）
Languages vary considerably in syntactic structure. About 40% of the world's languages have subject-verb-object order, and about 40% have subject-object-verb order. Extensive work has sought to explain this word order variation across languages. However, the existing approaches are not able to explain coherently the frequency distribution and evolution of word order in individual languages. We propose that variation in word order reflects different ways of balancing competing pressures of dependency locality and information locality, whereby languages favor placing elements together when they …
The frequencies at which individual words occur across languages follow power law distributions, a pattern of findings known as Zipf's law. A vast literature argues over whether this serves to optimize the efficiency of human communication, however this claim is necessarily post hoc, and it has been suggested that Zipf's law may in fact describe mixtures of other distributions. From this perspective, recent findings that Sinosphere first (family) names are geometrically distributed are notable, because this is actually consistent with information theoretic predictions regarding optimal coding.…
The University of Sydney（悉尼大学）；Tencent AI Lab（腾讯人工智能实验室）；University of Macau（澳门大学）；JD Explore Academy, JD.com（京东探索研究院）
Knowledge distillation (KD) is commonly used to construct synthetic data for training non-autoregressive translation (NAT) models. However, there exists a discrepancy on low-frequency words between the distilled and the original data, leading to more errors on predicting low-frequency words. To alleviate the problem, we directly expose the raw data into NAT by leveraging pretraining. By analyzing directed alignments, we found that KD makes low-frequency source words aligned with targets more deterministically but fails to align sufficient low-frequency words from target to source. Accordingly,…
In his pioneering research, G. K. Zipf formulated a couple of statistical laws on the relationship between the frequency of a word with its number of meanings: the law of meaning distribution, relating the frequency of a word and its frequency rank, and the meaning-frequency law, relating the frequency of a word with its number of meanings. Although these laws were formulated more than half a century ago, they have been only investigated in a few languages. Here we present the first study of these laws in Catalan. We verify these laws in Catalan via the relationship among their exponents and t…
Language Technologies Institute, Carnegie Mellon University（卡内基梅隆大学语言技术研究所）；Department of Computer Science, University of Toronto（多伦多大学计算机科学系）；Computer Science and Engineering, University of California, San Diego（加利福尼亚大学圣地亚哥分校计算机科学与工程系）
We perform statistical analysis of the phenomenon of neology, the process by which new words emerge in a language, using large diachronic corpora of English. We investigate the importance of two factors, semantic sparsity and frequency growth rates of semantic neighbors, formalized in the distributional semantics paradigm. We show that both factors are predictive of word emergence although we find more support for the latter hypothesis. Besides presenting a new linguistic application of distributional semantics, this study tackles the linguistic question of the role of language-internal factor…
Institute for Natural Language Processing, University of Stuttgart（斯图加特大学自然语言处理研究所）
This paper presents a comparison of unsupervised methods of hypernymy prediction (i.e., to predict which word in a pair of words such as fish-cod is the hypernym and which the hyponym). Most importantly, we demonstrate across datasets for English and for German that the predictions of three methods (WeedsPrec, invCL, SLQS Row) strongly overlap and are highly correlated with frequency-based predictions. In contrast, the second-order method SLQS shows an overall lower accuracy but makes correct predictions where the others go wrong. Our study once more confirms the general need to check the freq…
Peking University（北京大学）；Key Laboratory of Machine Perception, MOE, School of EECS, Peking University（北京大学信息科学技术学院机器感知与智能教育部重点实验室）；Microsoft Research Asia（微软亚洲研究院）；Center for Data Science, Peking University（北京大学数据科学中心）；Beijing Institute of Big Data Research（北京大数据研究院）
Continuous word representation (aka word embedding) is a basic building block in many neural network-based models used in natural language processing tasks. Although it is widely accepted that words with similar semantics should be close to each other in the embedding space, we find that word embeddings learned in several tasks are biased towards word frequency: the embeddings of high-frequency and low-frequency words lie in different subregions of the embedding space, and the embedding of a rare word and a popular word can be far from each other even if they are semantically similar. This mak…
Pre-trained contextualized language models (PrLMs) have led to strong performance gains in downstream natural language understanding tasks. However, PrLMs can still be easily fooled by adversarial word substitution, which is one of the most challenging textual adversarial attack methods. Existing defence approaches suffer from notable performance loss and complexities. Thus, this paper presents a compact and performance-preserved framework, Anomaly Detection with Frequency-Aware Randomization (ADFAR). In detail, we design an auxiliary anomaly detection classifier and adopt a multi-task learnin…
IR models using a pretrained language model significantly outperform lexical approaches like BM25. In particular, SPLADE, which encodes texts to sparse vectors, is an effective model for practical use because it shows robustness to out-of-domain datasets. However, SPLADE still struggles with exact matching of low-frequency words in training data. In addition, domain shifts in vocabulary and word frequencies deteriorate the IR performance of SPLADE. Because supervision data are scarce in the target domain, addressing the domain shifts without supervision data is necessary. This paper proposes a…
Unsupervised Sentence Representations as Word Information Series: Revisiting TF--IDF
Universidad Nacional Autónoma de México (UNAM)（墨西哥国立自治大学）；Centro de ciencias genómicas (CCG–UNAM)（基因组科学中心（墨西哥国立自治大学））；Instituto de ingeniería (IIngen–UNAM)（工程研究所（墨西哥国立自治大学））；Laboratoire Informatique d’Avignon (LIA–UAPV)（阿维尼翁计算机科学实验室（阿维尼翁大学））；Centro de investigación en computación (CIC–IPN)（计算机研究中心（国立理工学院））
Peking University（北京大学）；Beijing Language and Culture University（北京语言大学）；Tencent（腾讯）；Institute for Clarity in Documentation（文档清晰度研究所）；Inria Paris-Rocquencourt（法国国家信息与自动化研究所巴黎-罗克库尔分部）；Rajiv Gandhi University（拉吉夫·甘地大学）；Tsinghua University（清华大学）；Palmer Research Laboratories（帕尔默研究实验室）
Online advertising is an important revenue source for many IT companies. In the search advertising scenario, advertisement text that meets the need of the search query would be more attractive to the user. However, the manual creation of query-variant advertisement texts for massive items is expensive. Traditional text generation methods tend to focus on the general searching needs with high frequency while ignoring the diverse personalized searching needs with low frequency. In this paper, we propose the query-variant advertisement text generation task that aims to generate candidate advertis…
SLoW: Select Low-frequency Words! Automatic Dictionary Selection for Translation on Large Language Models
The Chinese University of Hong Kong（香港中文大学）；Southeast University（东南大学）；FaceMind Corporation（FaceMind公司）；College of Computer Science and Technology, Jilin University（吉林大学计算机科学与技术学院）
The pioneering research of G. K. Zipf on the relationship between word frequency and other word features led to the formulation of various linguistic laws. The most popular is Zipf's law for word frequencies. Here we focus on two laws that have been studied less intensively: the meaning-frequency law, i.e. the tendency of more frequent words to be more polysemous, and the law of abbreviation, i.e. the tendency of more frequent words to be shorter. In a previous work, we tested the robustness of these Zipfian laws for English, roughly measuring word length in number of characters and distinguis…
