Although Perplexity is a widely used performance metric for language models, the values are highly dependent upon the number of words in the corpus and is useful to compare performance of the same corpus only. In this paper, we propose a new metric that can be used to evaluate language model performance with different vocabulary sizes. The proposed unigram-normalized Perplexity actually presents the performance improvement of the language models from that of simple unigram model, and is robust on the vocabulary size. Both theoretical analysis and computational experiments are reported.
Conditional Unigram Tokenization with Parallel Data
Institute of Formal and Applied Linguistics, Faculty of Mathematics and Physics, Charles University, Prague, Czech Republic（形式与应用语言学研究所，数学与物理系，查尔斯大学，布拉格，捷克共和国）
University of Cambridge（剑桥大学）；Harvard University（哈佛大学）；MPI for Evolutionary Anthropology（马克斯·普朗克进化人类学研究所）；HSE University（高等经济大学）；ETH Zürich（苏黎世联邦理工学院）
The unigram distribution is the non-contextual probability of finding a specific word form in a corpus. While of central importance to the study of language, it is commonly approximated by each word's sample frequency in the corpus. This approach, being highly dependent on sample size, assigns zero probability to any out-of-vocabulary (oov) word form. As a result, it produces negatively biased probabilities for any oov word form, while positively biased probabilities to in-corpus words. In this work, we argue in favor of properly modeling the unigram distribution -- claiming it should be a cen…
Multilingual Tokenization through the Lens of Indian Languages: Challenges and Insights
Department of CSE, Indian Institute of Technology Hyderabad（印度理工学院海得拉巴分校计算机科学与工程系）；Department of CSE, Indian Institute of Technology Bombay（印度理工学院孟买分校计算机科学与工程系）；SCEE, Indian Institute of Technology Mandi（印度理工学院曼迪分校软件工程系）；Adobe Research（Adobe研究）
Recently, continuous cache models were proposed as extensions to recurrent neural network language models, to adapt their predictions to local changes in the data distribution. These models only capture the local context, of up to a few thousands tokens. In this paper, we propose an extension of continuous cache models, which can scale to larger contexts. In particular, we use a large scale non-parametric memory component that stores all the hidden activations seen in the past. We leverage recent advances in approximate nearest neighbor search and quantization algorithms to store millions of r…
In this paper, we detail novel strategies for interpolating personalized language models and methods to handle out-of-vocabulary (OOV) tokens to improve personalized language models. Using publicly available data from Reddit, we demonstrate improvements in offline metrics at the user level by interpolating a global LSTM-based authoring model with a user-personalized n-gram model. By optimizing this approach with a back-off to uniform OOV penalty and the interpolation coefficient, we observe that over 80% of users receive a lift in perplexity, with an average of 5.2% in perplexity lift per user…
In applications involving conversational speech, data sparsity is a limiting factor in building a better language model. We propose a simple, language-independent method to quickly harvest large amounts of data from Twitter to supplement a smaller training set that is more closely matched to the domain. The techniques lead to a significant reduction in perplexity on four low-resource languages even though the presence on Twitter of these languages is relatively small. We also find that the Twitter text is more useful for learning word classes than the in-domain text and that use of these word …
To Each Language Its Tokenizer: Modular Tokenizers for Efficient Multilingual LLMs
Kyutai；Sorbonne Université（索邦大学）；CNRS（法国国家科学研究中心）；Institute of Intelligent Systems and Robotics（智能系统与机器人研究所）；Univ. Grenoble Alpes（格勒诺布尔阿尔卑斯大学）；Grenoble INP（格勒诺布尔国立理工学院）；LIG（格勒诺布尔信息学实验室）
In this work we explore recent advances in Recurrent Neural Networks for large scale Language Modeling, a task central to language understanding. We extend current models to deal with two key challenges present in this task: corpora and vocabulary sizes, and complex, long term structure of language. We perform an exhaustive study on techniques such as character Convolutional Neural Networks or Long-Short Term Memory, on the One Billion Word Benchmark. Our best single model significantly improves state-of-the-art perplexity from 51.3 down to 30.0 (whilst reducing the number of parameters by a f…
The University of Tokyo（东京大学）；Graduate School of Frontier Sciences（前沿科学研究生院）；Research Center for Advanced Science and Technology（先进科学技术研究中心）
Language models have primarily been evaluated with perplexity. While perplexity quantifies the most comprehensible prediction performance, it does not provide qualitative information on the success or failure of models. Another approach for evaluating language models is thus proposed, using the scaling properties of natural language. Five such tests are considered, with the first two accounting for the vocabulary population and the other three for the long memory of natural language. The following models were evaluated with these tests: n-grams, probabilistic context-free grammar (PCFG), Simon…
Perplexity (per word) is the most widely used metric for evaluating language models. Despite this, there has been no dearth of criticism for this metric. Most of these criticisms center around lack of correlation with extrinsic metrics like word error rate (WER), dependence upon shared vocabulary for model comparison and unsuitability for unnormalized language model evaluation. In this paper, we address the last problem and propose a new discriminative entropy based intrinsic metric that works for both traditional word level models and unnormalized language models like sentence level models. W…
Leibniz Institute for the German Language（莱布尼茨德语语言研究所）
The coronavirus pandemic may be the largest crisis the world has had to face since World War II. It does not come as a surprise that it is also having an impact on language as our primary communication tool. We present three inter-connected resources that are designed to capture and illustrate these effects on a subset of the German language: An RSS corpus of German-language newsfeeds (with freely available untruncated unigram frequency lists), a static but continuously updated HTML page tracking the diversity of the used vocabulary and a web application that enables other researchers and the …
Paloma: A Benchmark for Evaluating Language Model Fit
Allen Institute for Artificial Intelligence（艾伦人工智能研究所）；Paul G. Allen School of Computer Science & Engineering, University of Washington（华盛顿大学保罗·G·艾伦计算机科学与工程学院）
How Much is Enough? The Diminishing Returns of Tokenization Training Data
Kensho Technologies, Cambridge, MA（Kensho技术公司，马萨诸塞州剑桥）；Ben-Gurion University of the Negev, Beer Sheva, Israel（贝内-阿尔-加利利大学，以色列贝 Sheva）；Massachusetts Institute of Technology, Cambridge, MA（麻省理工学院，马萨诸塞州剑桥）
Wright State University（莱特州立大学）；SVAIL, Baidu Research（百度研究院硅谷人工智能实验室）；University of Missouri（密苏里大学）；Yitu. Inc（依图科技）
Recurrent neural language models are the state-of-the-art models for language modeling. When the vocabulary size is large, the space taken to store the model parameters becomes the bottleneck for the use of recurrent neural language models. In this paper, we introduce a simple space compression method that randomly shares the structured parameters at both the input and output embedding layers of the recurrent neural language models to significantly reduce the size of model parameters, but still compactly represent the original input and output embedding layers. The method is easy to implement …
To mitigate the problem of having to traverse over the full vocabulary in the softmax normalization of a neural language model, sampling-based training criteria are proposed and investigated in the context of large vocabulary word-based neural language models. These training criteria typically enjoy the benefit of faster training and testing, at a cost of slightly degraded performance in terms of perplexity and almost no visible drop in word error rate. While noise contrastive estimation is one of the most popular choices, recently we show that other sampling-based criteria can also perform we…
