A longstanding debate in semiotics centers on the relationship between linguistic signs and their corresponding semantics: is there an arbitrary relationship between a word form and its meaning, or does some systematic phenomenon pervade? For instance, does the character bigram \textit{gl} have any systematic relationship to the meaning of words like \textit{glisten}, \textit{gleam} and \textit{glow}? In this work, we offer a holistic quantification of the systematicity of the sign using mutual information and recurrent neural networks. We employ these in a data-driven and massively multilingual approach to the question, examining 106 languages. We find a statistically significant reduction in entropy when modeling a word form conditioned on its semantic representation. Encouragingly, we also recover well-attested English examples of systematic affixes. We conclude with the meta-point: Our approximate effect size (measured in bits) is quite small---despite some amount of systematicity between form and meaning, an arbitrary relationship and its resulting benefits dominate human language.
Measuring Meaning Composition in the Human Brain with Composition Scores from Large Language Models
National Key Laboratory for Novel Software Technology, Nanjing University(南京大学计算机软件新技术国家重点实验室);Department of Linguistics and Translation, City University of Hong Kong(香港城市大学语言及翻译学系)
Facebook AI Research(脸书人工智能研究院);University of Cambridge(剑桥大学);Johns Hopkins University(约翰斯·霍普金斯大学);New York University(纽约大学);University of York(约克大学);ETH Zürich(苏黎世联邦理工学院)
Theoretical foundations and limits of word embeddings: what types of meaning can they capture?
University of California, Los Angeles(加利福尼亚大学洛杉矶分校)
意义的测量是文化社会学的一个核心问题,而词嵌入可以提供强大的新工具来做到这一点。但就像任何工具一样,它们建立在理论假设之上并发挥作用。在本文中,我理论化的方式,其中词嵌入模型的三个核心前提的结构语言学理论的意义:意义是关系,连贯的,可以作为一个静态系统进行分析。在某些方面,词嵌入方法容易受到这些前提的相同的、持久的批评。在其他方面,他们为这些批评提供了新的解决方案。更广泛地说,用词嵌入形式化意义的研究为澄清文化社会学中的核心概念和争论提供了理论机会,例如意义的连贯性。正如网络分析明确了曾经模糊的社会关系概念(Borgatti et al. 2009),用嵌入方法形式化意义可以推动我们明确和重新想象意义本身。