词嵌入常因捕捉到性别刻板印象等不良词关联而受到批评，但衡量和消除此类偏见的方法仍未被充分理解。我们证明，对于任何隐式进行矩阵分解的嵌入模型，使用子空间投影（Bolukbasi等人，2016）事后消除偏见，在某些条件下等价于在无偏见语料库上进行训练。我们还证明，词嵌入最常用的关联测试WEAT会系统地高估偏见。鉴于子空间投影方法已被证明有效，我们用它推导了一种名为关系内积关联（relational inner product association，RIPA）的新关联度量。用RIPA进行的实验表明，平均而言，带负采样的跳元模型（skipgram with negative sampling，SGNS）不会让大多数词比训练语料库中的词更具性别特征；但对于性别刻板印象词，SGNS实际上会放大语料库中的性别关联。
英文摘要：
Word embeddings are often criticized for capturing undesirable word associations such as gender stereotypes. However, methods for measuring and removing such biases remain poorly understood. We show that for any embedding model that implicitly does matrix factorization, debiasing vectors post hoc using subspace projection (Bolukbasi et al., 2016) is, under certain conditions, equivalent to training on an unbiased corpus. We also prove that WEAT, the most common association test for word embeddings, systematically overestimates bias. Given that the subspace projection method is provably effective, we use it to derive a new measure of association called the $\textit{relational inner product association}$ (RIPA). Experiments with RIPA reveal that, on average, skipgram with negative sampling (SGNS) does not make most words any more gendered than they are in the training corpus. However, for gender-stereotyped words, SGNS actually amplifies the gender association in the corpus.
Static word embeddings encode word associations, extensively utilized in downstream NLP tasks. Although prior studies have discussed the nature of such word associations in terms of biases and lexical regularities captured, the variation in word associations based on the embedding training procedure remains in obscurity. This work aims to address this gap by assessing attributive word associations across five different static word embedding architectures, analyzing the impact of the choice of the model architecture, context learning flavor and training corpora. Our approach utilizes a semi-sup…
ICC (UBA-CONICET)（ICC（布宜诺斯艾利斯大学-国家科学技术研究委员会））；Maestría en Data Mining (UBA)（数据挖掘硕士项目（布宜诺斯艾利斯大学））；Escuela IDAES (UNSAM)（IDAES学院（圣马丁将军大学））
Numerous works use word embedding-based metrics to quantify societal biases and stereotypes in texts. Recent studies have found that word embeddings can capture semantic similarity but may be affected by word frequency. In this work we study the effect of frequency when measuring female vs. male gender bias with word embedding-based bias quantification methods. We find that Skip-gram with negative sampling and GloVe tend to detect male bias in high frequency words, while GloVe tends to return female bias in low frequency words. We show these behaviors still exist when words are randomly shuffl…
Recent research has demonstrated that vector space models of semantics can reflect undesirable biases in human culture. Our investigation of crosslinguistic word embeddings reveals that topical gender bias interacts with, and is surpassed in magnitude by, the effect of grammatical gender associations, and both may be attenuated by corpus lemmatization. This finding has implications for downstream applications such as machine translation.
This is a preprint of the article "Identifying Phrasemes via Interlingual Association Measures" that was presented in February 2016 at the LeKo (Lexical combinations and typified speech in a multilingual context) conference in Innsbruck.
Laboratoire d’analyse statistique des textes - LAST（文本统计分析实验室）；Institut de recherche en sciences psychologiques（心理科学研究所）；Université catholique de Louvain（鲁汶天主教大学）
This paper describes the system developed by the Laboratoire d'analyse statistique des textes (LAST) for the Lexical Complexity Prediction shared task at SemEval-2021. The proposed system is made up of a LightGBM model fed with features obtained from many word frequency lists, published lexical norms and psychometric data. For tackling the specificity of the multi-word task, it uses bigram association measures. Despite that the only contextual feature used was sentence length, the system achieved an honorable performance in the multi-word task, but poorer in the single word task. The bigram as…
Institute for Information Technology（信息技术研究所）；National Research Council Canada（加拿大国家研究委员会）
Since many real-world concepts are associated with colour, for example danger with red, linguistic information is often complimented with the use of appropriate colours in information visualization and product marketing. Yet, there is no comprehensive resource that captures concept-colour associations. We present a method to create a large word-colour association lexicon by crowdsourcing. We focus especially on abstract concepts and emotions to show that even though they cannot be physically visualized, they too tend to have strong colour associations. Finally, we show how word-colour associat…
Rutgers University（罗格斯大学）；Hasso Plattner Institute/University of Potsdam（哈索·普拉特纳研究所/波茨坦大学）
What do word vector representations reveal about the emotions associated with words? In this study, we consider the task of estimating word-level emotion intensity scores for specific emotions, exploring unsupervised, supervised, and finally a self-supervised method of extracting emotional associations from word vector representations. Overall, we find that word vectors carry substantial potential for inducing fine-grained emotion intensity scores, showing a far higher correlation with human ground truth ratings than achieved by state-of-the-art emotion lexicons.
Comparing Moral Values in Western English-speaking societies and LLMs with Word Associations
School of Computing and Information Systems, The University of Melbourne（墨尔本大学计算与信息系统学院）；Complex Human Data Hub, The University of Melbourne（墨尔本大学复杂人类数据中心）
为避免直接提示的局限性，本研究提出利用词汇联想作为底层表征，通过在人类与LLM联想图中传播Moral Foundation Theory的种子词，系统比较了西方英语社会与LLM在道德价值观上的差异。
Colour is a key component in the successful dissemination of information. Since many real-world concepts are associated with colour, for example danger with red, linguistic information is often complemented with the use of appropriate colours in information visualization and product marketing. Yet, there is no comprehensive resource that captures concept-colour associations. We present a method to create a large word-colour association lexicon by crowdsourcing. A word-choice question was used to obtain sense-level annotations and to ensure data quality. We focus especially on abstract concepts…
In this paper we show that if we want to obtain human evidence about conventionalization of some phrases, we should ask native speakers about associations they have to a given phrase and its component words. We have shown that if component words of a phrase have each other as frequent associations, then this phrase can be considered as conventionalized. Another type of conventionalized phrases can be revealed using two factors: low entropy of phrase associations and low intersection of component word and phrase associations. The association experiments were performed for the Russian language.
This paper formulates and evaluates a series of multi-unit measures of directional association, building on the pairwise ΔP measure, that are able to quantify association in sequences of varying length and type of representation. Multi-unit measures face an additional segmentation problem: once the implicit length constraint of pairwise measures is abandoned, association measures must also identify the borders of meaningful sequences. This paper takes a vector-based approach to the segmentation problem by using 18 unique measures to describe different aspects of multi-unit association. An exam…
Humans use countless basic, shared facts about the world to efficiently navigate in their environment. This commonsense knowledge is rarely communicated explicitly, however, understanding how commonsense knowledge is represented in different paradigms is important for both deeper understanding of human cognition and for augmenting automatic reasoning systems. This paper presents an in-depth comparison of two large-scale resources of general knowledge: ConcpetNet, an engineered relational database, and SWOW a knowledge graph derived from crowd-sourced word associations. We examine the structure…
Access to word-sentiment associations is useful for many applications, including sentiment analysis, stance detection, and linguistic analysis. However, manually assigning fine-grained sentiment association scores to words has many challenges with respect to keeping annotations consistent. We apply the annotation technique of Best-Worst Scaling to obtain real-valued sentiment association scores for words and phrases in three different domains: general English, English Twitter, and Arabic Twitter. We show that on all three domains the ranking of words by sentiment remains remarkably consistent …
Even though considerable attention has been given to the polarity of words (positive and negative) and the creation of large polarity lexicons, research in emotion analysis has had to rely on limited and small emotion lexicons. In this paper we show how the combined strength and wisdom of the crowds can be used to generate a large, high-quality, word-emotion and word-polarity association lexicon quickly and inexpensively. We enumerate the challenges in emotion annotation in a crowdsourcing scenario and propose solutions to address them. Most notably, in addition to questions about emotions ass…
The "Small World of Words" German Free-Association Norms
Center for Cognitive and Decision Sciences, Faculty of Psychology, University of Basel（巴塞尔大学心理学系认知与决策科学中心）；Center for Adaptive Rationality, Max Planck Institute for Human Development, Berlin（柏林洪堡人类发展研究所适应性理性中心）；Institute of Estonian and General Linguistics, University of Tartu（塔尔图大学爱沙尼亚与一般语言学研究所）；Melbourne School of Psychological Sciences, University of Melbourne（墨尔本大学心理科学学院）
The University of Melbourne（墨尔本大学）；Moscow Polytech（莫斯科理工大学）；Institute of the Science of Language（语言科学研究所）
We present a quantitative analysis of human word association pairs and study the types of relations presented in the associations. We put our main focus on the correlation between response types and respondent characteristics such as occupation and gender by contrasting syntagmatic and paradigmatic associations. Finally, we propose a personalised distributed word association model and show the importance of incorporating demographic factors into the models commonly used in natural language processing.
Two well-known databases of semantic relationships between pairs of words used in psycholinguistics, feature-based and association-based, are studied as complex networks. We propose an algorithm to disentangle feature based relationships from free association semantic networks. The algorithm uses the rich topology of the free association semantic network to produce a new set of relationships between words similar to those observed in feature production norms.
University of Chinese Academy of Sciences（中国科学院大学）；Chinese Academy of Sciences（中国科学院）；Shanghai Advanced Research Institute（上海高等研究院）；Shanghai Carbon Data Research Center（上海碳数据研究中心）；Centre for Excellence in Brain Science and Intelligence Technology（脑科学与智能技术卓越创新中心）
Recent work on segmentation-free word embedding(sembei) developed a new pipeline of word embedding for unsegmentated language while avoiding segmentation as a preprocessing step. However, too many noisy n-grams existing in the embedding vocabulary that do not have strong association strength between characters would limit the quality of learned word embedding. To deal with this problem, a new version of segmentation-free word embedding model is proposed by collecting n-grams vocabulary via a novel unsupervised association measure called pointwise association with times information(PATI). Compa…
São Carlos Institute of Physics, University of São Paulo（圣保罗大学圣卡洛斯物理研究所）
To a good extent, words can be understood as corresponding to patterns or categories that appeared in order to represent concepts and structures that are particularly important or useful in a given time and space. Words are characterized by not being completely general nor specific, in the sense that the same word can be instantiated or related to several different contexts, depending on specific situations. Indeed, the way in which words are instantiated and associated represents a particularly interesting aspect that can substantially help to better understand the context in which they are e…
Investigating the Frequency Distortion of Word Embeddings and Its Impact on Bias Metrics
Instituto de Investigación en Ciencias de la Computación, CONICET-UBA（CONICET-布宜诺斯艾利斯大学计算机科学研究所）；Universidad de Buenos Aires (UBA)（布宜诺斯艾利斯大学（UBA））；Departamento de Computación, FCEyN, UBA（布宜诺斯艾利斯大学理学院计算机系）
Word embeddings have gained significant attention as learnable representations of semantic relations between words, and have been shown to improve upon the results of traditional word representations. However, little effort has been devoted to using embeddings for the retrieval of entity associations beyond pairwise relations. In this paper, we use popular embedding methods to train vector representations of an entity-annotated news corpus, and evaluate their performance for the task of predicting entity participation in news events versus a traditional word cooccurrence network as a baseline.…
This paper was was first drafted in 2001 as a formalization of the system described in U.S. patent U.S. 7,392,174. It describes a system for implementing a parser based on a kind of cross-product over vectors of contextually similar words. It is being published now in response to nascent interest in vector combination models of syntax and semantics. The method used aggressive substitution of contextually similar words and word groups to enable product vectors to stay in the same space as their operands and make entire sentences comparable syntactically, and potentially semantically. The vector…
This article proposes a biologically inspired neurocomputational architecture which learns associations between words and referents in different contexts, considering evidence collected from the literature of Psycholinguistics and Neurolinguistics. The multi-layered architecture takes as input raw images of objects (referents) and streams of word's phonemes (labels), builds an adequate representation, recognizes the current context, and associates label with referents incrementally, by employing a Self-Organizing Map which creates new association nodes (prototypes) as required, adjusts the exi…
University of Chicago（芝加哥大学）；University of Chicago, Booth School of Business（芝加哥大学布斯商学院）；Amazon（亚马逊公司）
We demonstrate the utility of a new methodological tool, neural-network word embedding models, for large-scale text analysis, revealing how these models produce richer insights into cultural associations and categories than possible with prior methods. Word embeddings represent semantic relations between words as geometric relationships between vectors in a high-dimensional space, operationalizing a relational model of meaning consistent with contemporary theories of identity and culture. We show that dimensions induced by word differences (e.g. man - woman, rich - poor, black - white, liberal…
Peking University（北京大学）；Beijing Language and Culture University（北京语言大学）；Tencent（腾讯）；Institute for Clarity in Documentation（文档清晰度研究所）；Inria Paris-Rocquencourt（法国国家信息与自动化研究所巴黎-罗克库尔分部）；Rajiv Gandhi University（拉吉夫·甘地大学）；Tsinghua University（清华大学）；Palmer Research Laboratories（帕尔默研究实验室）
Online advertising is an important revenue source for many IT companies. In the search advertising scenario, advertisement text that meets the need of the search query would be more attractive to the user. However, the manual creation of query-variant advertisement texts for massive items is expensive. Traditional text generation methods tend to focus on the general searching needs with high frequency while ignoring the diverse personalized searching needs with low frequency. In this paper, we propose the query-variant advertisement text generation task that aims to generate candidate advertis…
Playing with Words, Improving with Rewards: Training Language Models for Creative Association
University of Massachusetts Lowell（马萨诸塞大学洛市分校）；Dartmouth College（达特茅斯学院）；University of Amsterdam（阿姆斯特丹大学）；Pennsylvania State University（宾夕法尼亚州立大学）；Amazon AGI（亚马逊人工智能研究院）
We design a new co-occurrence based word association measure by incorporating the concept of significant cooccurrence in the popular word association measure Pointwise Mutual Information (PMI). By extensive experiments with a large number of publicly available datasets we show that the newly introduced measure performs better than other co-occurrence based measures and despite being resource-light, compares well with the best known resource-heavy distributional similarity and knowledge based word association measures. We investigate the source of this performance improvement and find that of t…
We train word-emoji embeddings on large scale messaging data obtained from the Jodel online social network. Our data set contains more than 40 million sentences, of which 11 million sentences are annotated with a subset of the Unicode 13.0 standard Emoji list. We explore semantic emoji associations contained in this embedding by analyzing associations between emojis, between emojis and text, and between text and emojis. Our investigations demonstrate anecdotally that word-emoji embeddings trained on large scale messaging data can reflect real-world semantic associations. To enable further rese…
IIT Bombay（印度理工学院孟买分校）；Microsoft Research India（微软印度研究院）
Lexical co-occurrence is an important cue for detecting word associations. We present a theoretical framework for discovering statistically significant lexical co-occurrences from a given corpus. In contrast with the prevalent practice of giving weightage to unigram frequencies, we focus only on the documents containing both the terms (of a candidate bigram). We detect biases in span distributions of associated words, while being agnostic to variations in global unigram frequencies. Our framework has the fidelity to distinguish different classes of lexical co-occurrences, based on strengths of…
University of Westminster（威斯敏斯特大学）；Oakland University（奥克兰大学）
Given the recent advances and progress in Natural Language Processing (NLP), extraction of semantic relationships has been at the top of the research agenda in the last few years. This work has been mainly motivated by the fact that building knowledge graphs (KG) and bases (KB), as a key ingredient of intelligent applications, is a never-ending challenge, since new knowledge needs to be harvested while old knowledge needs to be revised. Currently, approaches towards relation extraction from text are dominated by neural models practicing some sort of distant (weak) supervision in machine learni…
We present new methods for pruning and enhancing item- sets for text classification via association rule mining. Pruning methods are based on dependency syntax and enhancing methods are based on replacing words by their hyperonyms of various orders. We discuss the impact of these methods, compared to pruning based on tfidf rank of words.
The evaluative character of a word is called its semantic orientation. Positive semantic orientation indicates praise (e.g., "honest", "intrepid") and negative semantic orientation indicates criticism (e.g., "disturbing", "superfluous"). Semantic orientation varies in both direction (positive or negative) and degree (mild to strong). An automated system for measuring semantic orientation would have application in text classification, text filtering, tracking opinions in online discussions, analysis of survey responses, and automated chat systems (chatbots). This paper introduces a method for i…
ALIGN: Word Association Learning for Cultural Alignment in Large Language Models
School of Computing and Information Systems, The University of Melbourne（计算与信息系统学院，墨尔本大学）；Melbourne Data Analytics Platform, The University of Melbourne（墨尔本数据分析平台，墨尔本大学）；Faculty of Information Technology, Monash University（信息技术学院，墨尔本大学）
Bias in word embeddings such as Word2Vec has been widely investigated, and many efforts made to remove such bias. We show how to use conceptors debiasing to post-process both traditional and contextualized word embeddings. Our conceptor debiasing can simultaneously remove racial and gender biases and, unlike standard debiasing methods, can make effect use of heterogeneous lists of biased words. We show that conceptor debiasing diminishes racial and gender bias of word representations as measured using the Word Embedding Association Test (WEAT) of Caliskan et al. (2017).
Institut de recherche en sciences psychologiques（心理科学研究所）；Université catholique de Louvain（鲁汶天主教大学）
To determine whether some often-used lexical association measures assign high scores to n-grams that chance could have produced as frequently as observed, we used an extension of Fisher's exact test to sequences longer than two words to analyse a corpus of four million words. The results, based on the precision-recall curve and a new index called chance-corrected average precision, show that, as expected, simple-ll is extremely effective. They also show, however, that MI3 is more efficient than the other hypothesis tests-based measures and even reaches a performance level almost equal to simpl…
This paper proposes an Approximate n-gram Markov Model for bag generation. Directed word association pairs with distances are used to approximate (n-1)-gram and n-gram training tables. This model has parameters of word association model, and merits of both word association model and Markov Model. The training knowledge for bag generation can be also applied to lexical selection in machine translation design.
University of Notre Dame（圣母大学）；Hong Kong University of Science and Technology（香港科技大学）
Math word problem (MWP) solving is an important task in question answering which requires human-like reasoning ability. Analogical reasoning has long been used in mathematical education, as it enables students to apply common relational structures of mathematical situations to solve new problems. In this paper, we propose to build a novel MWP solver by leveraging analogical MWPs, which advance the solver's generalization ability across different kinds of MWPs. The key idea, named analogy identification, is to associate the analogical MWP pairs in a latent space, i.e., encoding an MWP close to …
Recent research in Natural Language Processing has revealed that word embeddings can encode social biases present in the training data which can affect minorities in real world applications. This paper explores the gender bias implicit in Dutch embeddings while investigating whether English language based approaches can also be used in Dutch. We implement the Word Embeddings Association Test (WEAT), Clustering and Sentence Embeddings Association Test (SEAT) methods to quantify the gender bias in Dutch word embeddings, then we proceed to reduce the bias with Hard-Debias and Sent-Debias mitigati…
Automatic Multi-Word Term (MWT) extraction is a very important issue to many applications, such as information retrieval, question answering, and text categorization. Although many methods have been used for MWT extraction in English and other European languages, few studies have been applied to Arabic. In this paper, we propose a novel, hybrid method which combines linguistic and statistical approaches for Arabic Multi-Word Term extraction. The main contribution of our method is to consider contextual information and both termhood and unithood for association measures at the statistical filte…
A word association network methodology for evaluating implicit biases in LLMs compared to humans
University of Pisa, Department of Computer Science（比萨大学计算机科学系）；National Research Council of Italy, Institute of Information Science and Technologies（意大利国家研究委员会信息科学与技术研究所）；University of Trento, Department of Psychology and Cognitive Science（特伦托大学心理学与认知科学系）
Soochow University（苏州大学）；University of Lisbon（里斯本大学）；Alibaba Group（阿里巴巴集团）
In neural machine translation, a source sequence of words is encoded into a vector from which a target sequence is generated in the decoding phase. Differently from statistical machine translation, the associations between source words and their possible target counterparts are not explicitly stored. Source and target words are at the two ends of a long information processing procedure, mediated by hidden states at both the source encoding and the target decoding phases. This makes it possible that a source word is incorrectly translated into a target word that is not any of its admissible equ…
University of Cambridge（剑桥大学）；The ALTA Institute（ALTA研究所）；University of Amsterdam（阿姆斯特丹大学）
The advent of social media in recent years has fed into some highly undesirable phenomena such as proliferation of offensive language, hate speech, sexist remarks, etc. on the Internet. In light of this, there have been several efforts to automate the detection and moderation of such abusive content. However, deliberate obfuscation of words by users to evade detection poses a serious challenge to the effectiveness of these efforts. The current state of the art approaches to abusive language detection, based on recurrent neural networks, do not explicitly address this problem and resort to a ge…
Kungliga Tekniska Högskolan (KTH)（瑞典皇家理工学院）；Universidad de Zaragoza（萨拉戈萨大学）；Instituto de Sistemas e Robótica, Instituto Superior Técnico（系统与机器人研究所，里斯本高等理工学院）
We address the problem of bootstrapping language acquisition for an artificial system similarly to what is observed in experiments with human infants. Our method works by associating meanings to words in manipulation tasks, as a robot interacts with objects and listens to verbal descriptions of the interactions. The model is based on an affordance network, i.e., a mapping between robot actions, robot perceptions, and the perceived effects of these actions upon objects. We extend the affordance model to incorporate spoken words, which allows us to ground the verbal symbols to the execution of a…
Contrastive Analysis of Linguistic Representations in Large Language Model Outputs through Structured Synthetic Data Generation and Abstracted N-gram Associations
In this paper, we pose the question: do people talk about women and men in different ways? We introduce two datasets and a novel integration of approaches for automatically inferring gender associations from language, discovering coherent word clusters, and labeling the clusters for the semantic concepts they represent. The datasets allow us to compare how people write about women and men in two different settings - one set draws from celebrity news and the other from student reviews of computer science professors. We demonstrate that there are large-scale differences in the ways that people t…
