This work addresses the problem of measuring how many languages a person "effectively" speaks given that some of the languages are close to each other. In other words, to assign a meaningful number to her language portfolio. Intuition says that someone who speaks fluently Spanish and Portuguese is linguistically less proficient compared to someone who speaks fluently Spanish and Chinese since it takes more effort for a native Spanish speaker to learn Chinese than Portuguese. As the number of languages grows and their proficiency levels vary, it gets even more complicated to assign a score to a language portfolio. In this article we propose such a measure ("linguistic quotient" - LQ) that can account for these effects.
We define the properties that such a measure should have. They are based on the idea of coherent risk measures from the mathematical finance. Having laid down the foundation, we propose one such a measure together with the algorithm that works on languages classification tree as input.
The algorithm together with the input is available online at lingvometer.com
Computational measures of linguistic diversity help us understand the linguistic landscape using digital language data. The contribution of this paper is to calibrate measures of linguistic diversity using restrictions on international travel resulting from the COVID-19 pandemic. Previous work has mapped the distribution of languages using geo-referenced social media and web data. The goal, however, has been to describe these corpora themselves rather than to make inferences about underlying populations. This paper shows that a difference-in-differences method based on the Herfindahl-Hirschman…
CSIRO Data61（澳大利亚联邦科学与工业研究组织数据61中心）；Kirby Institute, The University of New South Wales（柯比研究所、新南威尔士大学）
Epidemic intelligence deals with the detection of disease outbreaks using formal (such as hospital records) and informal sources (such as user-generated text on the web) of information. In this survey, we discuss approaches for epidemic intelligence that use textual datasets, referring to it as `text-based epidemic intelligence'. We view past work in terms of two broad categories: health mention classification (selecting relevant text from a large volume) and health event detection (predicting epidemic events from a collection of relevant text). The focus of our discussion is the underlying co…
Multiple studies have probed representations emerging in neural networks trained for end-to-end NLP tasks and examined what word-level linguistic information may be encoded in the representations. In classical probing, a classifier is trained on the representations to extract the target linguistic information. However, there is a threat of the classifier simply memorizing the linguistic labels for individual words, instead of extracting the linguistic abstractions from the representations, thus reporting false positive results. While considerable efforts have been made to minimize the memoriza…
Techniques are presented for defining models of computational linguistics theories. The methods of generalized diagrams that were developed by this author for modeling artificial intelligence planning and reasoning are shown to be applicable to models of computation of linguistics theories. It is shown that for extensional and intensional interpretations, models can be generated automatically which assign meaning to computations of linguistics theories for natural languages. Keywords: Computational Linguistics, Reasoning Models, G-diagrams For Models, Dynamic Model Implementation, Linguistics …
University of Amsterdam（阿姆斯特丹大学）；RWTH-Aachen University（亚琛工业大学）
There is a growing interest in the combined use of NLP and machine learning methods to predict gaze patterns during naturalistic reading. While promising results have been obtained through the use of transformer-based language models, little work has been undertaken to relate the performance of such models to general text characteristics. In this paper we report on experiments with two eye-tracking corpora of naturalistic reading and two language models (BERT and GPT-2). In all experiments, we test effects of a broad spectrum of features for predicting human reading behavior that fall into fiv…
Frequency counts are a measure of how much use a language makes of a linguistic unit, such as a phoneme or word. However, what is often important is not the units themselves, but the contrasts between them. A measure is therefore needed for how much use a language makes of a contrast, i.e. the functional load (FL) of the contrast. We generalize previous work in linguistics and speech recognition and propose a family of measures for the FL of several phonological contrasts, including phonemic oppositions, distinctive features, suprasegmentals, and phonological rules. We then test it for robustn…
We define general linguistic intelligence as the ability to reuse previously acquired knowledge about a language's lexicon, syntax, semantics, and pragmatic conventions to adapt to new tasks quickly. Using this definition, we analyze state-of-the-art natural language understanding models and conduct an extensive empirical investigation to evaluate them against these criteria through a series of experiments that assess the task-independence of the knowledge being acquired by the learning process. In addition to task performance, we propose a new evaluation metric based on an online encoding of …
Universiti Brunei Darussalam（文莱达鲁萨兰大学）；IIIT Hyderabad（印度海得拉巴国际信息技术学院）
Media news are making a large part of public opinion and, therefore, must not be fake. News on web sites, blogs, and social media must be analyzed before being published. In this paper, we present linguistic characteristics of media news items to differentiate between fake news and real news using machine learning algorithms. Neural fake news generation, headlines created by machines, semantic incongruities in text and image captions generated by machine are other types of fake news problems. These problems use neural networks which mainly control distributional features rather than evidence. …
RoCode: A Dataset for Measuring Code Intelligence from Problem Definitions in Romanian
University POLITEHNICA of Bucharest（布加勒斯特理工大学）；University of Bucharest, HTL Research Center（布加勒斯特大学HTL研究中心）；PRHLT Research Center, Universitat Politècnica de València（瓦伦西亚理工大学PRHLT研究中心）；ValgrAI - Valencian Graduate School and Research Network of Artificial Intelligence（ValgrAI - 瓦伦西亚人工智能研究生院与研究网络）
Facebook AI Research（Facebook人工智能研究院）；Virginia Tech（弗吉尼亚理工大学）；Microsoft Research（微软研究院）
As machines have become more intelligent, there has been a renewed interest in methods for measuring their intelligence. A common approach is to propose tasks for which a human excels, but one which machines find difficult. However, an ideal task should also be easy to evaluate and not be easily gameable. We begin with a case study exploring the recently popular task of image captioning and its limitations as a task for measuring machine intelligence. An alternative and more promising task is Visual Question Answering that tests a machine's ability to reason about language and vision. We descr…
LongEmotion: Measuring Emotional Intelligence of Large Language Models in Long-Context Interaction
Shenzhen MSU-BIT University（深圳MSU-BIT大学）；The University of Hong Kong（香港大学）；City University of Hong Kong（香港城市大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究所）；Beijing Normal University（北京师范大学）；University of Michigan, Ann Arbor（密歇根大学安娜堡分校）
Measuring, evaluating and reducing Gender Bias has come to the forefront with newer and improved language embeddings being released every few months. But could this bias vary from domain to domain? We see a lot of work to study these biases in various embedding models but limited work has been done to debias Indic languages. We aim to measure and study this bias in Hindi language, which is a higher-order language (gendered) with reference to English, a lower-order language. To achieve this, we study the variations across domains to quantify if domain embeddings allow us some insight into Gende…
On the Compatibility of Generative AI and Generative Linguistics
Department of Decision Sciences, HEC Montréal（决策科学系，蒙特利尔HEC商学院）；Mila - Quebec Artificial Intelligence Institute（魁北克人工智能研究所）；Department of Linguistics, University of California - Davis（语言学系，加州大学戴维斯分校）
Transformer architectures show significant promise for natural language processing. Given that a single pretrained model can be fine-tuned to perform well on many different tasks, these networks appear to extract generally useful linguistic features. A natural question is how such networks represent this information internally. This paper describes qualitative and quantitative investigations of one particularly effective model, BERT. At a high level, linguistic features seem to be represented in separate semantic and syntactic subspaces. We find evidence of a fine-grained geometric representat…
Modality is one of the important components of grammar in linguistics. It lets speaker to express attitude towards, or give assessment or potentiality of state of affairs. It implies different senses and thus has different perceptions as per the context. This paper presents an account showing the gap in the functionality of the current state of art Natural Language Processing (NLP) systems. The contextual nature of linguistic modality is studied. In this paper, the works and logical approaches employed by Natural Language Processing systems dealing with modality are reviewed. It sees human cog…
Cognitive Machine Learning (ENS - CNRS - EHESS - INRIA - PSL Research University)（认知机器学习（巴黎高等师范学院-法国国家科学研究中心-法国社会科学高等研究院-法国国家信息与自动化研究所-巴黎文理研究大学））；Facebook A.I. Research（Facebook人工智能研究院）；Laboratoire de Linguistique Formelle (CNRS - Paris Diderot - Sorbonne Paris Cite)（形式语言学实验室（法国国家科学研究中心-巴黎狄德罗大学-巴黎索邦大学））；University of Toronto（多伦多大学）
Vector space models of words have long been claimed to capture linguistic regularities as simple vector translations, but problems have been raised with this claim. We decompose and empirically analyze the classic arithmetic word analogy test, to motivate two new metrics that address the issues with the standard test, and which distinguish between class-wise offset concentration (similar directions between pairs of words drawn from different broad classes, such as France--London, China--Ottawa, ...) and pairing consistency (the existence of a regular transformation between correctly-matched pa…
Measuring research data reuse in scholarly publications using generative artificial intelligence: Open Science Indicator development and preliminary results
Facebook AI Research（Facebook AI 研究院）；University of Cambridge（剑桥大学）
Meaning has been called the "holy grail" of a variety of scientific disciplines, ranging from linguistics to philosophy, psychology and the neurosciences. The field of Artifical Intelligence (AI) is very much a part of that list: the development of sophisticated natural language semantics is a sine qua non for achieving a level of intelligence comparable to humans. Embodiment theories in cognitive science hold that human semantic representation depends on sensori-motor experience; the abundant evidence that human meaning representation is grounded in the perception of physical reality leads to…
Unveiling Linguistic Regions in Large Language Models
School of Computer Science, Fudan University（复旦大学计算机科学技术学院）；Institute of Modern Languages and Linguistics, Fudan University（复旦大学现代语言与语言学研究所）；Shanghai Collaborative Innovation Center of Intelligent Visual Computing（上海智能视觉计算协同创新中心）
The value-alignment problem for artificial intelligence (AI) asks how we can ensure that the 'values' (i.e., objective functions) of artificial systems are aligned with the values of humanity. In this paper, I argue that linguistic communication (natural language) is a necessary condition for robust value alignment. I discuss the consequences that the truth of this claim would have for research programmes that attempt to ensure value alignment for AI systems; or, more loftily, designing robustly beneficial or ethical artificial agents.
This position paper describes and critiques the Pretraining-Agnostic Identically Distributed (PAID) evaluation paradigm, which has become a central tool for measuring progress in natural language understanding. This paradigm consists of three stages: (1) pre-training of a word prediction model on a corpus of arbitrary size; (2) fine-tuning (transfer learning) on a training set representing a classification task; (3) evaluation on a test set drawn from the same distribution as that training set. This paradigm favors simple, low-bias architectures, which, first, can be scaled to process vast amo…
Institute for Information Technology（信息技术研究所）；National Research Council Canada（加拿大国家研究委员会）
Since many real-world concepts are associated with colour, for example danger with red, linguistic information is often complimented with the use of appropriate colours in information visualization and product marketing. Yet, there is no comprehensive resource that captures concept-colour associations. We present a method to create a large word-colour association lexicon by crowdsourcing. We focus especially on abstract concepts and emotions to show that even though they cannot be physically visualized, they too tend to have strong colour associations. Finally, we show how word-colour associat…
Large-scale language technologies are increasingly used in various forms of communication with humans across different contexts. One particular use case for these technologies is conversational agents, which output natural language text in response to prompts and queries. This mode of engagement raises a number of social and ethical questions. For example, what does it mean to align conversational agents with human norms or values? Which norms or values should they be aligned with? And how can this be accomplished? In this paper, we propose a number of steps that help answer these questions. W…
EHESS（高等社会科学研究院）；ENS（高等师范学校）；PSL Research University（巴黎文理研究大学）；CNRS（法国国家科学研究中心）；INRIA（法国国家信息与自动化研究所）
During their first years of life, infants learn the language(s) of their environment at an amazing speed despite large cross cultural variations in amount and complexity of the available language input. Understanding this simple fact still escapes current cognitive and linguistic theories. Recently, spectacular progress in the engineering science, notably, machine learning and wearable technology, offer the promise of revolutionizing the study of cognitive development. Machine learning offers powerful learning algorithms that can achieve human-like performance on many linguistic tasks. Wearabl…
University of Groningen（格罗宁根大学）；University of Malta（马耳他大学）
Contextualized word embeddings have been replacing standard embeddings as the representational knowledge source of choice in NLP systems. Since a variety of biases have previously been found in standard word embeddings, it is crucial to assess biases encoded in their replacements as well. Focusing on BERT (Devlin et al., 2018), we measure gender bias by studying associations between gender-denoting target words and names of professions in English and German, comparing the findings with real-world workforce statistics. We mitigate bias by fine-tuning BERT on the GAP corpus (Webster et al., 2018…
