A stereotype is an over-generalized belief about a particular group of people, e.g., Asians are good at math or Asians are bad drivers. Such beliefs (biases) are known to hurt target groups. Since pretrained language models are trained on large real world data, they are known to capture stereotypical biases. In order to assess the adverse effects of these models, it is important to quantify the bias captured in them. Existing literature on quantifying bias evaluates pretrained language models on a small set of artificially constructed bias-assessing sentences. We present StereoSet, a large-scale natural dataset in English to measure stereotypical biases in four domains: gender, profession, race, and religion. We evaluate popular models like BERT, GPT-2, RoBERTa, and XLNet on our dataset and show that these models exhibit strong stereotypical biases. We also present a leaderboard with a hidden test set to track the bias of future language models at https://stereoset.mit.edu
In this paper, we investigate what types of stereotypical information are captured by pretrained language models. We present the first dataset comprising stereotypical attributes of a range of social groups and propose a method to elicit stereotypes encoded by pretrained language models in an unsupervised fashion. Moreover, we link the emergent stereotypes to their manifestation as basic emotions as a means to study their emotional effects in a more generalized manner. To demonstrate how our methods can be used to analyze emotion and stereotype shifts due to linguistic experience, we use fine-…
C2PO: Diagnosing and Disentangling Bias Shortcuts in LLMs
Jinan University（济南大学）；Nanyang Technological University（南洋理工大学）；Engineering Research Center of Trustworthy AI (Ministry of Education)（可信人工智能工程研究中心）；Guangxi Key Laboratory of Trusted Software（广西可信软件重点实验室）
MIT Computer Science and Artificial Intelligence Laboratory（麻省理工学院计算机科学与人工智能实验室）
Due to their similarity-based learning objectives, pretrained sentence encoders often internalize stereotypical assumptions that reflect the social biases that exist within their training corpora. In this paper, we describe several kinds of stereotypes concerning different communities that are present in popular sentence representation models, including pretrained next sentence prediction and contrastive sentence representation models. We compare such models to textual entailment models that learn language logic for a variety of downstream language understanding tasks. By comparing strong pret…
University of Mannheim（曼海姆大学）；Bocconi University（博科尼大学）
Unfair stereotypical biases (e.g., gender, racial, or religious biases) encoded in modern pretrained language models (PLMs) have negative ethical implications for widespread adoption of state-of-the-art language technology. To remedy for this, a wide range of debiasing techniques have recently been introduced to remove such stereotypical biases from PLMs. Existing debiasing methods, however, directly modify all of the PLMs parameters, which -- besides being computationally expensive -- comes with the inherent risk of (catastrophic) forgetting of useful language knowledge acquired in pretrainin…
Open-DeBias: Toward Mitigating Open-Set Bias in Language Models
Mehta Family School of DS & AI, Indian Institute of Technology Roorkee（梅hta家庭数据科学与人工智能学院，印度理工学院罗里克分校）；Computer Science and Engineering, Indian Institute of Technology Bombay（计算机科学与工程，印度理工学院孟买分校）
The Devil is in the Neurons: Interpreting and Mitigating Social Biases in Pre-trained Language Models
Chinese University of Hong Kong（香港中文大学）；Peking University（北京大学）；National University of Singapore（新加坡国立大学）；Microsoft Research（微软研究院）；IBM Research（IBM研究院）
Rutgers University（罗格斯大学）；HPI / University of Potsdam（哈索·普拉特纳研究所/波茨坦大学）
Recent breakthroughs in self supervised training have led to a new class of pretrained vision language models. While there have been investigations of bias in multimodal models, they have mostly focused on gender and racial bias, giving much less attention to other relevant groups, such as minorities with regard to religion, nationality, sexual orientation, or disabilities. This is mainly due to lack of suitable benchmarks for such groups. We seek to address this gap by providing a visual and textual bias benchmark called MMBias, consisting of around 3,800 images and phrases covering 14 popula…
