We consider two graph models of semantic change. The first is a time-series model that relates embedding vectors from one time period to embedding vectors of previous time periods. In the second, we construct one graph for each word: nodes in this graph correspond to time points and edge weights to the similarity of the word's meaning across two time points. We apply our two models to corpora across three different languages. We find that semantic change is linear in two senses. Firstly, today's embedding vectors (= meaning) of words can be derived as linear combinations of embedding vectors of their neighbors in previous time periods. Secondly, self-similarity of words decays linearly in time. We consider both findings as new laws/hypotheses of semantic change.
University of Oxford（牛津大学）；Google DeepMind（谷歌DeepMind）
This paper aims to explore the effect of prior disambiguation on neural network- based compositional models, with the hope that better semantic representations for text compounds can be produced. We disambiguate the input word vectors before they are fed into a compositional deep net. A series of evaluations shows the positive effect of prior disambiguation for such deep models.
When Meanings Meet: Investigating the Emergence and Quality of Shared Concept Spaces during Multilingual Language Model Training
MaiNLP, Center for Information and Language Processing, LMU Munich（MaiNLP，信息与语言处理中心，慕尼黑大学）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；University of Tokyo（东京大学）；IT University of Copenhagen（哥本哈根IT大学）；Language Technology Lab, University of Cambridge（语言技术实验室，剑桥大学）
National Laboratory of Pattern Recognition, CASIA（中国科学院自动化研究所模式识别国家重点实验室）；University of Chinese Academy of Sciences（中国科学院大学）；CAS Key Laboratory of Behavioural Science, Institute of Psychology（中国科学院心理研究所行为科学重点实验室）；Department of Psychology, University of Chinese Academy of Sciences（中国科学院大学心理学系）；CAS Center for Excellence in Brain Science and Intelligence Technology（中国科学院脑科学与智能技术卓越创新中心）
Multimodal models have been proven to outperform text-based approaches on learning semantic representations. However, it still remains unclear what properties are encoded in multimodal representations, in what aspects do they outperform the single-modality representations, and what happened in the process of semantic compositionality in different input modalities. Considering that multimodal models are originally motivated by human concept representations, we assume that correlating multimodal representations with brain-based semantics would interpret their inner properties to answer the above…
University of Padua（帕多瓦大学）；Beijing Institute of Technology（北京理工大学）
Capturing the meaning of sentences has long been a challenging task. Current models tend to apply linear combinations of word features to conduct semantic composition for bigger-granularity units e.g. phrases, sentences, and documents. However, the semantic linearity does not always hold in human language. For instance, the meaning of the phrase `ivory tower' can not be deduced by linearly combining the meanings of `ivory' and `tower'. To address this issue, we propose a new framework that models different levels of semantic units (e.g. sememe, word, sentence, and semantic abstraction) on a si…
Orange Innovation（Orange创新研究院）；Aix-Marseille Univ.（艾克斯-马赛大学）；CNRS（法国国家科学研究中心）；LIS（计算机与系统实验室）；AMU（艾克斯-马赛大学）；Huawei Edinburgh Research Centre（华为爱丁堡研究中心）
Text generation from Abstract Meaning Representation (AMR) has substantially benefited from the popularized Pretrained Language Models (PLMs). Myriad approaches have linearized the input graph as a sequence of tokens to fit the PLM tokenization requirements. Nevertheless, this transformation jeopardizes the structural integrity of the graph and is therefore detrimental to its resulting representation. To overcome this issue, Ribeiro et al. have recently proposed StructAdapt, a structure-aware adapter which injects the input graph connectivity within PLMs using Graph Neural Networks (GNNs). In …
Center for Information and Language Processing (CIS), LMU Munich（慕尼黑大学信息与语言处理中心（CIS））；Research Training Group AIPHES（AIPHES研究培训小组）
Graph-to-text generation aims to generate fluent texts from graph-based data. In this paper, we investigate two recently proposed pretrained language models (PLMs) and analyze the impact of different task-adaptive pretraining strategies for PLMs in graph-to-text generation. We present a study across three graph domains: meaning representations, Wikipedia knowledge graphs (KGs) and scientific KGs. We show that the PLMs BART and T5 achieve new state-of-the-art results and that task-adaptive pretraining strategies improve their performance even further. In particular, we report new state-of-the-a…
LMs stand their Ground: Investigating the Effect of Embodiment in Figurative Language Interpretation by Language Models
Ludwig-Maximilians-University (LMU)（路德维希-马克西米利安大学（慕尼黑大学））；Institute for Information and Language Processing (CIS)（信息与语言处理研究所）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）
Investigating Idiomaticity in Word Representations
University of Sheffield（谢菲尔德大学）；Federal University of Rio Grande do Sul（南里奥格兰德联邦大学）；University of Santiago de Compostela（圣地亚哥-德孔波斯特拉大学）；University of Exeter（埃克塞特大学）；The Alan Turing Institute（图灵研究所）
Embodied Agents Meet Personalization: Investigating Challenges and Solutions Through the Lens of Memory Utilization
Department of Artificial Intelligence, Yonsei University（人工智能系，延世大学）；Department of Computer Science and Engineering, Texas A&M University（计算机科学与工程系，德克萨斯大学）
How does the input segmentation of pretrained language models (PLMs) affect their interpretations of complex words? We present the first study investigating this question, taking BERT as the example PLM and focusing on its semantic representations of English derivatives. We show that PLMs can be interpreted as serial dual-route models, i.e., the meanings of complex words are either stored or else need to be computed from the subwords, which implies that maximally meaningful input tokens should allow for the best generalization on new words. This hypothesis is confirmed by a series of semantic …
Most words have several senses and connotations which evolve in time due to semantic shift, so that closely related words may gain different or even opposite meanings over the years. This evolution is very relevant to the study of language and of cultural changes, but the tools currently available for diachronic semantic analysis have significant, inherent limitations and are not suitable for real-time analysis. In this article, we demonstrate how the linearity of random vectors techniques enables building time series of congruent word embeddings (or semantic spaces) which can then be compared…
A key initial step in several natural language processing (NLP) tasks involves embedding phrases of text to vectors of real numbers that preserve semantic meaning. To that end, several methods have been recently proposed with impressive results on semantic similarity tasks. However, all of these approaches assume that perfect transcripts are available when generating the embeddings. While this is a reasonable assumption for analysis of written text, it is limiting for analysis of transcribed text. In this paper we investigate the effects of word substitution errors, such as those coming from a…
