Word embeddings typically represent different meanings of a word in a single conflated vector. Empirical analysis of embeddings of ambiguous words is currently limited by the small size of manually annotated resources and by the fact that word senses are treated as unrelated individual concepts. We present a large dataset based on manual Wikipedia annotations and word senses, where word senses from different words are related by semantic classes. This is the basis for novel diagnostic tests for an embedding's content: we probe word embeddings for semantic classes and analyze the embedding space by classifying embeddings into semantic classes. Our main findings are: (i) Information about a sense is generally represented well in a single-vector embedding - if the sense is frequent. (ii) A classifier can accurately predict whether a word is single-sense or multi-sense, based only on its embedding. (iii) Although rare senses are not well represented in single-vector embeddings, this does not have negative impact on an NLP application whose performance depends on frequent senses.
MUStReason: A Benchmark for Diagnosing Pragmatic Reasoning in Video-LMs for Multimodal Sarcasm Detection
Max Planck Institute for Informatics, Saarland Informatics Campus（马克斯·普朗克研究所信息学研究所，萨尔兰州信息学校园）；Saarland University（萨尔兰州大学）；The University of Edinburgh（爱丁堡大学）；Samsung AI Center, Cambridge（三星人工智能中心，剑桥）
Seeing Symbols, Missing Cultures: Probing Vision-Language Models' Reasoning on Fire Imagery and Cultural Meaning
Duncan of Jordanstone College of Art & Design (DJCAD), University of Dundee（邓迪大学邓迪艺术与设计学院（DJCAD））；Guangzhou Institute of Science and Technology (GZIST)（广州科学技术研究院）；Faculty of Arts, Xiamen University（厦门大学艺术学院）；University of Birmingham（伯明翰大学）
Beyond Tokens: Semantic-Aware Speculative Decoding for Efficient Inference by Probing Internal States
Centre for Software Excellence, Huawei, Canada（华为加拿大软件 excellence 中心）；Department of Computer Science, University of Manitoba, Canada（曼尼托巴大学计算机科学系）；School of Computing, Queen’s University, Canada（皇后大学计算学院）
Probing the statistical properties of enriched co-occurrence networks
Institute of Mathematics and Computer Science, University of São Paulo（圣保罗大学数学与计算机科学研究所）；Escola Politécnica da Universidade de São Paulo (EPUSP)（圣保罗大学工程学院）
University of Lisbon（里斯本大学）；Macquarie University（麦考瑞大学）
Lexical semantics theories differ in advocating that the meaning of words is represented as an inference graph, a feature mapping or a vector space, thus raising the question: is it the case that one of these approaches is superior to the others in representing lexical semantics appropriately? Or in its non antagonistic counterpart: could there be a unified account of lexical semantics where these approaches seamlessly emerge as (partial) renderings of (different) aspects of a core semantic knowledge base? In this paper, we contribute to these research questions with a number of experiments th…
University of Helsinki（赫尔辛基大学）；Université Paris-Saclay（巴黎萨克雷大学）；CNRS（法国国家科学研究中心）
Large scale language models encode rich commonsense knowledge acquired through exposure to massive data during pre-training, but their understanding of entities and their semantic properties is unclear. We probe BERT (Devlin et al., 2019) for the properties of English nouns as expressed by adjectives that do not restrict the reference scope of the noun they modify (as in "red car"), but instead emphasise some inherent aspect ("red strawberry"). We base our study on psycholinguistics datasets that capture the association strength between nouns and their semantic features. We probe BERT using cl…
Whole word masking (WWM), which masks all subwords corresponding to a word at once, makes a better English BERT model. For the Chinese language, however, there is no subword because each token is an atomic character. The meaning of a word in Chinese is different in that a word is a compositional unit consisting of multiple characters. Such difference motivates us to investigate whether WWM leads to better context understanding ability for Chinese BERT. To achieve this, we introduce two probing tasks related to grammatical error correction and ask pretrained models to revise or insert tokens in…
University of Cambridge（剑桥大学）；Mila/McGill University（米拉/麦吉尔大学）
Capturing word meaning in context and distinguishing between correspondences and variations across languages is key to building successful multilingual and cross-lingual text representation models. However, existing multilingual evaluation datasets that evaluate lexical semantics "in-context" have various limitations. In particular, 1) their language coverage is restricted to high-resource languages and skewed in favor of only a few language families and areas, 2) a design that makes the task solvable via superficial cues, which results in artificially inflated (and sometimes super-human) perf…
LMU Munich（慕尼黑大学）；Munich Center of Machine Learning（慕尼黑机器学习中心）；University of Oxford（牛津大学）
Construction Grammar (CxG) is a paradigm from cognitive linguistics emphasising the connection between syntax and semantics. Rather than rules that operate on lexical items, it posits constructions as the central building blocks of language, i.e., linguistic units of different granularity that combine syntax and semantics. As a first step towards assessing the compatibility of CxG with the syntactic and semantic knowledge demonstrated by state-of-the-art pretrained language models (PLMs), we present an investigation of their capability to classify and understand one of the most commonly studie…
University of Washington（华盛顿大学）；Microsoft Research（微软研究院）；Allen Institute for Artificial Intelligence（艾伦人工智能研究所）
Despite recent success, large neural models often generate factually incorrect text. Compounding this is the lack of a standard automatic evaluation for factuality--it cannot be meaningfully improved if it cannot be measured. Grounded generation promises a path to solving both of these problems: models draw on a reliable external document (grounding) for factual information, simplifying the challenge of factuality. Measuring factuality is also simplified--to factual consistency, testing whether the generation agrees with the grounding, rather than all facts. Yet, without a standard automatic m…
How does the input segmentation of pretrained language models (PLMs) affect their interpretations of complex words? We present the first study investigating this question, taking BERT as the example PLM and focusing on its semantic representations of English derivatives. We show that PLMs can be interpreted as serial dual-route models, i.e., the meanings of complex words are either stored or else need to be computed from the subwords, which implies that maximally meaningful input tokens should allow for the best generalization on new words. This hypothesis is confirmed by a series of semantic …
Institute of Physics of São Carlos, University of São Paulo（圣保罗大学圣保罗卡洛斯物理研究所）；Max Planck Institute for the Physics of Complex Systems (MPIPKS)（马克斯·普朗克复杂系统物理研究所）；Potsdam Institute for Climate Impact Research (PIK)（波茨坦气候影响研究所）
While the use of statistical physics methods to analyze large corpora has been useful to unveil many patterns in texts, no comprehensive investigation has been performed investigating the properties of statistical measurements across different languages and texts. In this study we propose a framework that aims at determining if a text is compatible with a natural language and which languages are closest to it, without any knowledge of the meaning of the words. The approach is based on three types of statistical measurements, i.e. obtained from first-order statistics of word properties in a tex…
