The measurement of bias in machine learning often focuses on model performance across identity subgroups (such as man and woman) with respect to groundtruth labels. However, these methods do not directly measure the associations that a model may have learned, for example between labels and identity subgroups. Further, measuring a model's bias requires a fully annotated evaluation dataset which may not be easily available in practice. We present an elegant mathematical solution that tackles both issues simultaneously, using image classification as a working example. By treating a classification model's predictions for a given image as a set of labels analogous to a bag of words, we rank the biases that a model has learned with respect to different identity labels. We use (man, woman) as a concrete example of an identity label set (although this set need not be binary), and present rankings for the labels that are most biased towards one identity or the other. We demonstrate how the statistical properties of different association metrics can lead to different rankings of the most "gender biased" labels, and conclude that normalized pointwise mutual information (nPMI) is most useful in practice. Finally, we announce an open-sourced nPMI visualization tool using TensorBoard.
It is tempting to think that machines are less prone to unfairness and prejudice. However, machine learning approaches compute their outputs based on data. While biases can enter at any stage of the development pipeline, models are particularly receptive to mirror biases of the datasets they are trained on and therefore do not necessarily reflect truths about the world but, primarily, truths about the data. To raise awareness about the relationship between modern algorithms and the data that shape them, we use a conditional invertible neural network to disentangle the dataset-specific informat…
Machine learning models are known to perpetuate and even amplify the biases present in the data. However, these data biases frequently do not become apparent until after the models are deployed. Our work tackles this issue and enables the preemptive analysis of large-scale datasets. REVISE (REvealing VIsual biaSEs) is a tool that assists in the investigation of a visual dataset, surfacing potential biases along three dimensions: (1) object-based, (2) person-based, and (3) geography-based. Object-based biases relate to the size, context, or diversity of the depicted objects. Person-based metric…
University of Edinburgh（爱丁堡大学）；IMT School for Advanced Studies Lucca（卢卡高等研究院）；Canon Medical Research Europe Ltd.（佳能医疗欧洲研究有限公司）；The Alan Turing Institute（艾伦·图灵研究所）
A recent spate of state-of-the-art semi- and un-supervised solutions disentangle and encode image "content" into a spatial tensor and image appearance or "style" into a vector, to achieve good performance in spatially equivariant tasks (e.g. image-to-image translation). To achieve this, they employ different model design, learning objective, and data biases. While considerable effort has been made to measure disentanglement in vector representations, and assess its impact on task performance, such analysis for (spatial) content - style disentanglement is lacking. In this paper, we conduct an e…
This work explores the biases in learning processes based on deep neural network architectures. We analyze how bias affects deep learning processes through a toy example using the MNIST database and a case study in gender detection from face images. We employ two gender detection models based on popular deep neural networks. We present a comprehensive analysis of bias effects when using an unbalanced training dataset on the features learned by the models. We show how bias impacts in the activations of gender detection models based on face images. We finally propose InsideBias, a novel method t…
VisBias: Measuring Explicit and Implicit Social Biases in Vision Language Models
University of Southern California（南加州大学）；University of Bristol（布里斯托大学）；Centre National de la Recherche Scientifique（法国国家科学研究中心）；University of California, Los Angeles（加州大学洛杉矶分校）
Ask Me Again Differently: GRAS for Measuring Bias in Vision Language Models on Gender, Race, Age, and Skin Tone
Guru Gobind Singh Indraprastha University（古鲁·戈宾德·辛格印度教普拉斯塔大学）；AI Institute, University of South Carolina（南卡罗来纳大学人工智能研究所）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Indian Institute of Technology Patna, India（印度理工学院帕纳布分校）
McGill University（麦吉尔大学）；Mila - Quebec AI Institute（魁北克人工智能研究所Mila）
Reanalysis datasets combining numerical physics models and limited observations to generate a synthesised estimate of variables in an Earth system, are prone to biases against ground truth. Biases identified with the NASA Modern-Era Retrospective Analysis for Research and Applications, Version 2 (MERRA-2) aerosol optical depth (AOD) dataset, against the Aerosol Robotic Network (AERONET) ground measurements in previous studies, motivated the development of a deep learning based AOD prediction model globally. This study combines a convolutional neural network (CNN) with MERRA-2, tested against a…
Do We Really Need Dice? The Hidden Region-Size Biases of Segmentation Losses
ÉTS Montréal（蒙特利尔工程学院）；MILA（蒙特利尔学习算法研究所）；CNRS（法国国家科学研究中心）；Université Paris-Saclay（巴黎萨克雷大学）；CentraleSupélec（中央理工-高等电力学院）；Centre de Recherche du Centre Hospitalier de l’Université de Montréal (CRCHUM)（蒙特利尔大学医院研究中心）；Universitat Pompeu Fabra（庞培法布拉大学）；Diagnos Inc.（诊断公司）
Unsupervised bias discovery in medical image segmentation
Research Institute for Signals, Systems and Computational Intelligence, sinc(i)（信号、系统与计算智能研究所sinc(i)）；CONICET（国家科学技术研究委员会）；Universidad Nacional del Litoral（国立滨海大学）
University of Warwick（华威大学）；NYU（纽约大学）；The Alan Turing Institute（艾伦·图灵研究所）
We propose a generalizable framework for the population estimation of dense, informal settlements in low-income urban areas--so called 'slums'--using high-resolution satellite imagery. Precise population estimates are a crucial factor for efficient resource allocations by government authorities and NGO's, for instance in medical emergencies. We utilize equitable ground-truth data, which is gathered in collaboration with local communities: Through training and community mapping, the local population contributes their unique domain knowledge, while also maintaining agency over their data. This p…
