We provide an overview of the hybrid compositional distributional model of meaning, developed in Coecke et al. (arXiv:1003.4394v1 [cs.CL]), which is based on the categorical methods also applied to the analysis of information flow in quantum protocols. The mathematical setting stipulates that the meaning of a sentence is a linear function of the tensor products of the meanings of its words. We provide concrete constructions for this definition and present techniques to build vector spaces for meaning vectors of words, as well as that of sentences. The applicability of these methods is demonstrated via a toy vector space as well as real data from the British National Corpus and two disambiguation experiments.
The categorical compositional distributional model of meaning gives the composition of words into phrases and sentences pride of place. However, it has so far lacked a model of logical negation. This paper gives some steps towards providing this operator, modelling it as a version of projection onto the subspace orthogonal to a word. We give a small demonstration of the operators performance in a sentence entailment task.
Bridging Compositional and Distributional Semantics: A Survey on Latent Semantic Geometry via AutoEncoder
Department of Computer Science, University of Manchester（曼彻斯特大学计算机科学系）；Idiap Research Institute（Idiap研究 institute）；Cancer Biomarker Centre, CRUK Manchester Institute（CRUK曼彻斯特癌症生物标志物中心）
The aim of this paper is twofold: first we will use vector space distributional compositional categorical models of meaning to compare the meaning of sentences in Irish and in English (and thus ascertain when a sentence is the translation of another sentence) using the cosine similarity score. Then we shall outline a procedure which translates nouns by understanding their context, using a conceptual space model of cognition. We shall use metrics on the category ConvexRel to determine the distance between concepts (and determine when a noun is the translation of another noun). This paper will f…
We propose a framework to model an operational conversational negation by applying worldly context (prior knowledge) to logical negation in compositional distributional semantics. Given a word, our framework can create its negation that is similar to how humans perceive negation. The framework corrects logical negation to weight meanings closer in the entailment hierarchy more than meanings further apart. The proposed framework is flexible to accommodate different choices of logical negations, compositions, and worldly context generation. In particular, we propose and motivate a new logical ne…
Czech Technical University（捷克技术大学）；University College London（伦敦大学学院）；Queen Mary University of London（伦敦玛丽女王大学）
Recent work on compositional distributional models shows that bialgebras over finite dimensional vector spaces can be applied to treat generalised quantifiers for natural language. That technique requires one to construct the vector space over powersets, and therefore is computationally costly. In this paper, we overcome this problem by considering fuzzy versions of quantifiers along the lines of Zadeh, within the category of many valued relations. We show that this category is a concrete instantiation of the compositional distributional model. We show that the semantics obtained in this model…
Indian Institute of Technology Kharagpur（印度理工学院卡拉格普尔分校）；National Research University Higher School of Economics（国立研究型大学高等经济学院）；Skolkovo Institute of Science and Technology（斯科尔科沃科学技术研究院）；Universität Hamburg（汉堡大学）
The compositionality degree of multiword expressions indicates to what extent the meaning of a phrase can be derived from the meaning of its constituents and their grammatical relations. Prediction of (non)-compositionality is a task that has been frequently addressed with distributional semantic models. We introduce a novel technique to blend hierarchical information with distributional information for predicting compositionality. In particular, we use hypernymy information of the multiword and its constituents encoded in the form of the recently introduced Poincaré embeddings in addition to …
Categorical compositional distributional semantics provide a method to derive the meaning of a sentence from the meaning of its individual words: the grammatical reduction of a sentence automatically induces a linear map for composing the word vectors obtained from distributional semantics. In this paper, we extend this passage from word-to-sentence to sentence-to-discourse composition. To achieve this we introduce a notion of basic anaphoric discourses as a mid-level representation between natural language discourse formalised in terms of basic discourse representation structures (DRS); and k…
Institute for Theoretical Physics（理论物理研究所）；ILLC, University of Amsterdam（阿姆斯特丹大学逻辑、语言与计算研究所）；Department of Mathematics（数学系）
To give vector-based representations of meaning more structure, one approach is to use positive semidefinite (psd) matrices. These allow us to model similarity of words as well as the hyponymy or is-a relationship. Psd matrices can be learnt relatively easily in a given vector space $M\otimes M^*$, but to compose words to form phrases and sentences, we need representations in larger spaces. In this paper, we introduce a generic way of composing the psd matrices corresponding to words. We propose that psd matrices for verbs, adjectives, and other functional words be lifted to completely positiv…
Most compositional distributional semantic models represent sentence meaning with a single vector. In this paper, we propose a Structured Distributional Model (SDM) that combines word embeddings with formal semantics and is based on the assumption that sentences represent events and situations. The semantic representation of a sentence is a formal structure derived from Discourse Representation Theory and containing distributional vectors. This structure is dynamically and incrementally built by integrating knowledge about events and their typical participants, as they are activated by lexical…
University of Amsterdam（阿姆斯特丹大学）；University of Bristol（布里斯托尔大学）；ILLC, University of Amsterdam（阿姆斯特丹大学语言、逻辑与计算研究所(ILLC)）
Words can have multiple senses. Compositional distributional models of meaning have been argued to deal well with finer shades of meaning variation known as polysemy, but are not so well equipped to handle word senses that are etymologically unrelated, or homonymy. Moving from vectors to density matrices allows us to encode a probability distribution over different senses of a word, and can also be accommodated within a compositional distributional model of meaning. In this paper we present three new neural models for learning density matrices from a corpus, and test their ability to discrimin…
Boston University（波士顿大学）；New York University（纽约大学）；Johns Hopkins University（约翰斯·霍普金斯大学）；Microsoft Research（微软研究院）
Human linguistic capacity is often characterized by compositionality and the generalization it enables -- human learners can produce and comprehend novel complex expressions by composing known parts. Several benchmarks exploit distributional control across training and test to gauge compositional generalization, where certain lexical items only occur in limited contexts during training. While recent work using these benchmarks suggests that pretrained models achieve impressive generalization performance, we argue that exposure to pretraining data may break the aforementioned distributional con…
We develop a vector space semantics for verb phrase ellipsis with anaphora using type-driven compositional distributional semantics based on the Lambek calculus with limited contraction (LCC) of Jäger (2006). Distributional semantics has a lot to say about the statistical collocation-based meanings of content words, but provides little guidance on how to treat function words. Formal semantics on the other hand, has powerful mechanisms for dealing with relative pronouns, coordinators, and the like. Type-driven compositional distributional semantics brings these two models together. We review pr…
Saarland University（萨尔大学）；University of Groningen（格罗宁根大学）
Natural language semantics has recently sought to combine the complementary strengths of formal and distributional approaches to meaning. More specifically, proposals have been put forward to augment formal semantic machinery with distributional meaning representations, thereby introducing the notion of semantic similarity into formal semantics, or to define distributional systems that aim to incorporate formal notions such as entailment and compositionality. However, given the fundamentally different 'representational currency' underlying formal and distributional approaches - models of the w…
形式化和分布式语义模型在建模意义方面提供了互补的优势。Coecke et al.（arXiv：1003.4394v1 [cs.CL]）的范畴成分分布（DisCoCat）意义模型结合了两者的方面，提供了一个一般框架，在这个框架中，单词的意义，分布获得，使用逻辑设置的方法组成句子意义。这种一般抽象背景的具体后果和对经验数据的应用正在积极研究中（Grefenstette等人，arxiv：1101.0309; Grefenstette and Sadrzadeh，arXiv：1106.4058v1 [cs.CL]）。.在本文中，我们扩展了这项研究，通过检查及物动词，表示为矩阵的DiscoCat。我们讨论了三种构建这种矩阵的方法，并在Grefenstette和Sadrzadeh开发的消歧任务中评估了每种方法（arXiv：1106.4058v1 [cs.CL]）。
Composing or Not Composing? Towards Distributional Construction Grammars
Laboratoire Parole et Langage, CNRS, France（法国语言与语言实验室，CNRS）；Department of Languages, Literatures and Modern Cultures, University of Bologna, Italy（博洛尼亚大学语言、文学与现代文化系）；Department of Chinese and Bilingual Studies, The Hong Kong Polytechnic University, Hong Kong（香港理工大学中文与双语研究系）；Department of Philology, Literature, and Linguistics, University of Pisa（比萨大学语言学、文学与语言学系）；ILCB - Institute of Language, Communication and the Brain, France（法国语言、交流与大脑研究所）
Lexical and Derivational Meaning in Vector-Based Models of Relativisation
Utrecht University（乌得勒支大学）；Queen Mary University of London（伦敦玛丽女王大学）
Sadrzadeh et al（2013）根据有限维向量空间的Frobenius代数结构提出了英语关系从句的成分分布分析。分析依赖于不同的类型分配和词汇配方的主体与客体的相对化。荷兰语的情况则不同：由于荷兰语的动词词尾性质，关系从句在主语与宾语的相对化阅读之间是模糊的。使用一个扩展版本的Lambek演算，我们提出了一个组成的分布框架，占这种派生的歧义，并允许我们给一个单一的意义配方的相对代词调和Frobenius语义与荷兰派生语法的要求。
Distributional Sentence Entailment Using Density Matrices
Queen Mary University of London（伦敦玛丽女王大学）；University of Oxford（牛津大学）
Coecke et al.（2010）的范畴组合分布模型提出了一种将形式的类型逻辑模型的语法组合与基于语料库的分布语义的经验词表示相结合的方法。本文通过扩展模型来捕获蕴涵关系，从而为该项目做出了贡献。这是通过将词的表示从意义空间中的点扩展到密度算子来实现的，密度算子是空间子空间上的概率分布。定义了相似性的对称度量和蕴涵的非对称度量，其中词汇蕴涵使用冯·诺依曼熵（Kullback-Leibler发散的量子变体）来测量。词汇蕴涵与词表征的组合图相结合，提供了一种在句子层次上获取蕴涵关系的方法。真值理论和语料库为基础的例子。
Building meaningful representations of noun compounds is not trivial since many of them scarcely appear in the corpus. To that end, composition functions approximate the distributional representation of a noun compound by combining its constituent distributional vectors. In the more general case, phrase embeddings have been trained by minimizing the distance between the vectors representing paraphrases. We compare various types of noun compound representations, including distributional, compositional, and paraphrase-based representations, through a series of tasks and analyses, and with an ext…
University of California, Santa Barbara（加州大学圣巴巴拉分校）；Beijing University of Posts and Telecommunications（北京邮电大学）
Deep reinforcement learning (RL) has been a commonly-used strategy for the abstractive summarization task to address both the exposure bias and non-differentiable task issues. However, the conventional reward Rouge-L simply looks for exact n-grams matches between candidates and annotated references, which inevitably makes the generated sentences repetitive and incoherent. In this paper, instead of Rouge-L, we explore the practicability of utilizing the distributional semantics to measure the matching degrees. With distributional semantics, sentence-level evaluation can be obtained, and semanti…
