Meaning is the foundation stone of intercultural communication. Languages are continuously changing, and words shift their meanings for various reasons. Semantic divergence in related languages is a key concern of historical linguistics. In this paper we investigate semantic divergence across languages by measuring the semantic similarity of cognate sets in multiple languages. The method that we propose is based on cross-lingual word embeddings. In this paper we implement and evaluate our method on English and five Romance languages, but it can be extended easily to any language pair, requiring only large monolingual corpora for the involved languages and a small bilingual dictionary for the pair. This language-agnostic method facilitates a quantitative analysis of cognates divergence -- by computing degrees of semantic similarity between cognate pairs -- and provides insights for identifying false friends. As a second contribution, we formulate a straightforward method for detecting false friends, and introduce the notion of "soft false friend" and "hard false friend", as well as a measure of the degree of "falseness" of a false friends pair. Additionally, we propose an algorithm that can output suggestions for correcting false friends, which could result in a very helpful tool for language learning or translation.
University of Trento（特伦托大学）；National University of Mongolia（蒙古国立大学）
The Universal Knowledge Core (UKC) is a large multilingual lexical database with a focus on language diversity and covering over a thousand languages. The aim of the database, as well as its tools and data catalogue, is to make the somewhat abstract notion of diversity visually understandable for humans and formally exploitable by machines. The UKC website lets users explore millions of individual words and their meanings, but also phenomena of cross-lingual convergence and divergence, such as shared interlingual meanings, lexicon similarities, cognate clusters, or lexical gaps. The UKC LiveLa…
University of Washington（华盛顿大学）；Allen Institute for Artificial Intelligence（艾伦人工智能研究院）；Stanford University（斯坦福大学）
As major progress is made in open-ended text generation, measuring how close machine-generated text is to human language remains a critical open problem. We introduce MAUVE, a comparison measure for open-ended text generation, which directly compares the learnt distribution from a text generation model to the distribution of human-written text using divergence frontiers. MAUVE scales up to modern text generation models by computing information divergences in a quantized embedding space. Through an extensive empirical study on three open-ended generation tasks, we find that MAUVE identifies kno…
Convergences and Divergences between Automatic Assessment and Human Evaluation: Insights from Comparing ChatGPT-Generated Translation and Neural Machine Translation
School of Foreign Languages, Shanghai Jiao Tong University（上海交通大学外国语学院）；Department of Computer Science and Engineering, Shanghai Jiao Tong University（上海交通大学计算机科学与工程系）；Institute of Corpus Studies and Applications, Shanghai International Studies University（上海外国语大学语料库研究与应用研究所）
State-of-the-art machine learning methods exhibit limited compositional generalization. At the same time, there is a lack of realistic benchmarks that comprehensively measure this ability, which makes it challenging to find and evaluate improvements. We introduce a novel method to systematically construct such benchmarks by maximizing compound divergence while guaranteeing a small atom divergence between train and test sets, and we quantitatively compare this method to other approaches for creating compositional generalization benchmarks. We present a large and realistic natural language quest…
Divergent Token Metrics: Measuring degradation to prune away LLM components -- and optimize quantization
Aleph Alpha；Technical University Darmstadt（达姆施塔特工业大学）；Hessian Center for Artificial Intelligence（黑森人工智能中心）；LMU Munich（慕尼黑大学）；German Center for Artificial Intelligence（德国人工智能研究中心）；Munich Center for Machine Learning（慕尼黑机器学习中心）
Code-MUE: Measuring Code LLMs' Uncertainty through Execution-based Semantic Interaction Graphs
Xi’an Jiaotong University（西安交通大学）；Institute of AI for Industries, Chinese Academy of Sciences（中国科学院人工智能产业研究院）；The University of Tokyo（东京大学）；University of Alberta（阿尔伯塔大学）
LIBRA: Measuring Bias of Large Language Model from a Local Context
School of Computer Science, University of Auckland（奥克兰大学计算机科学学院）；The Liggins Institute, University of Auckland（奥克兰大学利金斯研究所）；Research Centre for Māori Health and Development, Massey University（梅西大学毛利人健康与发展研究中心）
