Fairness in machine learning has attracted increasing attention in recent years. The fairness methods improving algorithmic fairness for in-distribution data may not perform well under distribution shifts. In this paper, we first theoretically demonstrate the inherent connection between distribution shift, data perturbation, and model weight perturbation. Subsequently, we analyze the sufficient conditions to guarantee fairness (i.e., low demographic parity) for the target dataset, including fairness for the source dataset, and low prediction difference between the source and target datasets for each sensitive attribute group. Motivated by these sufficient conditions, we propose robust fairness regularization (RFR) by considering the worst case within the model weight perturbation ball for each sensitive attribute group. We evaluate the effectiveness of our proposed RFR algorithm on synthetic and real distribution shifts across various datasets. Experimental results demonstrate that RFR achieves better fairness-accuracy trade-off performance compared with several baselines. The source code is available at \url{https://github.com/zhimengj0326/RFR_NeurIPS23}.
University of Maryland, College Park（马里兰大学帕克分校）；Boston University（波士顿大学）
The increasing reliance on ML models in high-stakes tasks has raised a major concern on fairness violations. Although there has been a surge of work that improves algorithmic fairness, most of them are under the assumption of an identical training and test distribution. In many real-world applications, however, such an assumption is often violated as previously trained fair models are often deployed in a different environment, and the fairness of such models has been observed to collapse. In this paper, we study how to transfer model fairness under distribution shifts, a widespread issue in pr…
Given an algorithmic predictor that is "fair" on some source distribution, will it still be fair on an unknown target distribution that differs from the source within some bound? In this paper, we study the transferability of statistical group fairness for machine learning predictors (i.e., classifiers or regressors) subject to bounded distribution shifts. Such shifts may be introduced by initial training data uncertainties, user adaptation to a deployed predictor, dynamic environments, or the use of pre-trained models in new settings. Herein, we develop a bound that characterizes such transfe…
University of Texas at Austin（德克萨斯大学奥斯汀分校）；Michigan State University（密歇根州立大学）
Increasing concerns have been raised on deep learning fairness in recent years. Existing fairness-aware machine learning methods mainly focus on the fairness of in-distribution data. However, in real-world applications, it is common to have distribution shift between the training and test data. In this paper, we first show that the fairness achieved by existing methods can be easily broken by slight distribution shifts. To solve this problem, we propose a novel fairness learning method termed CUrvature MAtching (CUMA), which can achieve robust fairness generalizable to unseen domains with unkn…
From Fragile to Certified: Wasserstein Audits of Group Fairness Under Distribution Shift
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Tübingen AI Center（图宾根人工智能中心）；Mila Québec AI Institute（魁北克人工智能研究所）；McGill University（麦吉尔大学）
Supervised Algorithmic Fairness in Distribution Shifts: A Survey
School of New Media and Communication, Tianjin University（天津大学新媒体与传播学院）；College of Intelligence and Computing, Tianjin University（天津大学智能与计算学部）；Department of Computer Science, Baylor University（贝勒大学计算机系）；Electrical Engineering and Computer Science Department, University of Arkansas（阿肯色大学电气与计算机科学系）
L3S Research Center（L3S研究中心）；Leibniz Universität Hannover（汉诺威莱布尼茨大学）；Freie Universität Berlin（柏林自由大学）；The Open University（开放大学）
With the ever growing involvement of data-driven AI-based decision making technologies in our daily social lives, the fairness of these systems is becoming a crucial phenomenon. However, an important and often challenging aspect in utilizing such systems is to distinguish validity for the range of their application especially under distribution shifts, i.e., when a model is deployed on data with different distribution than the training set. In this paper, we present a case study on the newly released American Census datasets, a reconstruction of the popular Adult dataset, to illustrate the imp…
University of Illinois at Chicago（伊利诺伊大学芝加哥分校）；California Institute of Technology（加州理工学院）
Making predictions that are fair with regard to protected group membership (race, gender, age, etc.) has become an important requirement for classification algorithms. Existing techniques derive a fair model from sampled labeled data relying on the assumption that training and testing data are identically and independently drawn (iid) from the same distribution. In practice, distribution shift can and does occur between training and testing datasets as the characteristics of individuals interacting with the machine learning system change. We investigate fairness under covariate shift, a relaxa…
IBM Research – T. J. Watson Research Center（IBM研究院–托马斯·J·沃森研究中心）；DataArt（数据艺术公司）；IBM Research – Africa（IBM研究院–非洲分院）
To ensure trust in AI models, it is becoming increasingly apparent that evaluation of models must be extended beyond traditional performance metrics, like accuracy, to other dimensions, such as fairness, explainability, adversarial robustness, and distribution shift. We describe an empirical study to evaluate multiple model types on various metrics along these dimensions on several datasets. Our results show that no particular model type performs well on all dimensions, and demonstrate the kinds of trade-offs involved in selecting models evaluated along multiple dimensions.
Secure and Confidential Certificates of Online Fairness
University of Toronto（多伦多大学）；Vector Institute（向量研究院）；Brave Software（Brave软件公司）；University of Massachusetts, Amherst（马萨诸塞大学阿默斯特分校）；Imperial College London（伦敦帝国理工学院）
Face4FairShifts: A Large Image Benchmark for Fairness and Robust Learning across Visual Domains
School of New Media and Communication, Tianjin University（天津大学新媒体与传播学院）；Department of Computer Science, Baylor University（贝勒大学计算机科学系）；Electrical Engineering and Computer Science Department, University of Arkansas（阿肯色大学电子工程与计算机科学系）；NEC Laboratories America（NEC美国实验室）；School of Computing, Southern Illinois University（南伊利诺伊大学计算机学院）
Many popular algorithmic fairness measures depend on the joint distribution of predictions, outcomes, and a sensitive feature like race or gender. These measures are sensitive to distribution shift: a predictor which is trained to satisfy one of these fairness definitions may become unfair if the distribution changes. In performative prediction settings, however, predictors are precisely intended to induce distribution shift. For example, in many applications in criminal justice, healthcare, and consumer finance, the purpose of building a predictor is to reduce the rate of adverse outcomes suc…
Basque Center for Applied Mathematics (BCAM)（巴斯克应用数学中心（BCAM））；University of Sussex（萨塞克斯大学）；Institute of Mathematics of the University of Valladolid (IMUVa)（巴利亚多利德大学数学研究所（IMUVa））；University of the Basque Country UPV/EHU（巴斯克大学（UPV/EHU））；Monash University（莫纳什大学）
Human lives are increasingly being affected by the outcomes of automated decision-making systems and it is essential for the latter to be, not only accurate, but also fair. The literature of algorithmic fairness has grown considerably over the last decade, where most of the approaches are evaluated under the strong assumption that the train and test samples are independently and identically drawn from the same underlying distribution. However, in practice, dissimilarity between the training and deployment environments exists, which compromises the performance of the decision-making algorithm a…
Measuring Fairness Under Unawareness of Sensitive Attributes: A Quantification-Based Approach
Max Planck Institute for Security and Privacy（马克斯·普朗克安全与隐私研究所）；University of Padova（帕多瓦大学）；Istituto di Scienza e Tecnologie dell’Informazione（信息科学与技术研究所）；Consiglio Nazionale delle Ricerche（意大利国家研究委员会）
Understanding challenges to the interpretation of disaggregated evaluations of algorithmic fairness
Google Research, Mountain View, CA, USA（谷歌研究, 山景城, 加州, 美国）；Google DeepMind, Mountain View, CA, USA（谷歌深Mind, 山景城, 加州, 美国）；New York University, New York, NY, USA（纽约大学, 纽约, 纽约, 美国）；Massachusetts Institute of Technology, Cambridge, MA, USA（麻省理工学院, 柏林, 马萨诸塞, 美国）；The Hopsital for Sick Children, Toronto, ON, Canada（儿童医院, 温哥华, 加拿大）；Stanford University, Stanford, CA, USA（斯坦福大学, 斯坦福, 加州, 美国）
University of Toronto（多伦多大学）；Vector Institute（向量研究所）
Machine learning models in health care are often deployed in settings where it is important to protect patient privacy. In such settings, methods for differentially private (DP) learning provide a general-purpose approach to learn models with privacy guarantees. Modern methods for DP learning ensure privacy through mechanisms that censor information judged as too unique. The resulting privacy-preserving models, therefore, neglect information from the tails of a data distribution, resulting in a loss of accuracy that can disproportionately affect small groups. In this paper, we study the effect…
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；University of Minnesota, Twin Cities（明尼苏达大学双城分校）
The Invariant Risk Minimization (IRM) framework aims to learn invariant features from a set of environments for solving the out-of-distribution (OOD) generalization problem. The underlying assumption is that the causal components of the data generating distributions remain constant across the environments or alternately, the data "overlaps" across environments to find meaningful invariant features. Consequently, when the "overlap" assumption does not hold, the set of truly invariant features may not be sufficient for optimal prediction performance. Such cases arise naturally in networked setti…
On the Fairness ROAD: Robust Optimization for Adversarial Debiasing
AXA Group Operations（安盛集团运营部）；Stanford University（斯坦福大学）；TRAIL, Sorbonne Université（索邦大学TRAIL）；LERIA, Université d’Angers（昂热大学LERIA）；Polish Academy of Science（波兰科学院）
University of Waterloo（滑铁卢大学）；Vector Institute（向量研究所）
With the growing awareness to fairness in machine learning and the realization of the central role that data representation has in data processing tasks, there is an obvious interest in notions of fair data representations. The goal of such representations is that a model trained on data under the representation (e.g., a classifier) will be guaranteed to respect some fairness constraints. Such representations are useful when they can be fixed for training models on various different tasks and also when they serve as data filtering between the raw data (known to the representation designer) and…
UIUC（伊利诺伊大学厄巴纳-香槟分校）；ETH Zurich（苏黎世联邦理工学院）；UC Santa Cruz（加州大学圣克鲁兹分校）
Extensive efforts have been made to understand and improve the fairness of machine learning models based on observational metrics, especially in high-stakes domains such as medical insurance, education, and hiring decisions. However, there is a lack of certified fairness considering the end-to-end performance of an ML model. In this paper, we first formulate the certified fairness of an ML model trained on a given data distribution as an optimization problem based on the model performance loss bound on a fairness constrained distribution, which is within bounded distributional distance with th…
Fix Representation (Optimally) Before Fairness: Finite-Sample Shrinkage Population Correction and the True Price of Fairness Under Subpopulation Shift
Department of Biostatistics, Vanderbilt University Medical Center, Nashville, TN 37232, USA（生物统计学系，范德比尔特大学医学中心）；Department of Informatics, King's College London, WC2R 2LS London, UK（信息学系，国王学院伦敦）
Indian Institute of Science（印度科学学院）；Microsoft Research（微软研究院）
In this paper, we study the problem of fair classification in the presence of prior probability shifts, where the training set distribution differs from the test set. This phenomenon can be observed in the yearly records of several real-world datasets, such as recidivism records and medical expenditure surveys. If unaccounted for, such shifts can cause the predictions of a classifier to become unfair towards specific population subgroups. While the fairness notion called Proportional Equality (PE) accounts for such shifts, a procedure to ensure PE-fairness was unknown. In this work, we propose…
University of Michigan（密歇根大学）；University of Konstanz（康斯坦茨大学）；IBM Research（IBM研究院）；Princeton University（普林斯顿大学）；Stanford University（斯坦福大学）
Many instances of algorithmic bias are caused by distributional shifts. For example, machine learning (ML) models often perform worse on demographic groups that are underrepresented in the training data. In this paper, we leverage this connection between algorithmic fairness and distribution shifts to show that algorithmic fairness interventions can help ML models overcome distribution shifts, and that domain adaptation methods (for overcoming distribution shifts) can mitigate algorithmic biases. In particular, we show that (i) enforcing suitable notions of individual fairness (IF) can improve…
We study the problem of learning fair prediction models for unseen test sets distributed differently from the train set. Stability against changes in data distribution is an important mandate for responsible deployment of models. The domain adaptation literature addresses this concern, albeit with the notion of stability limited to that of prediction accuracy. We identify sufficient conditions under which stable models, both in terms of prediction accuracy and fairness, can be learned. Using the causal graph describing the data and the anticipated shifts, we specify an approach based on featur…
Existing algorithms for ensuring fairness in AI use a single-shot training strategy, where an AI model is trained on an annotated training dataset with sensitive attributes and then fielded for utilization. This training strategy is effective in problems with stationary distributions, where both training and testing data are drawn from the same distribution. However, it is vulnerable with respect to distributional shifts in the input space that may occur after the initial training phase. As a result, the time-dependent nature of data can introduce biases into the model predictions. Model retra…
University of Michigan（密歇根大学）；IBM Research（IBM研究院）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）
Many instances of algorithmic bias are caused by subpopulation shifts. For example, ML models often perform worse on demographic groups that are underrepresented in the training data. In this paper, we study whether enforcing algorithmic fairness during training improves the performance of the trained model in the \emph{target domain}. On one hand, we conceive scenarios in which enforcing fairness does not improve performance in the target domain. In fact, it may even harm performance. On the other hand, we derive necessary and sufficient conditions under which enforcing algorithmic fairness l…
University of Arkansas（阿肯色大学）；University of Illinois（伊利诺伊大学）
Federated learning is an emerging framework that builds centralized machine learning models with training data distributed across multiple devices. Most of the previous works about federated learning focus on the privacy protection and communication cost reduction. However, how to achieve fairness in federated learning is under-explored and challenging especially when testing data distribution is different from training distribution or even unknown. Introducing simple fairness constraints on the centralized model cannot achieve model fairness on unknown testing data. In this paper, we develop …
KAIST（韩国科学技术院）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）
Model fairness is an essential element for Trustworthy AI. While many techniques for model fairness have been proposed, most of them assume that the training and deployment data distributions are identical, which is often not true in practice. In particular, when the bias between labels and sensitive groups changes, the fairness of the trained model is directly influenced and can worsen. We make two contributions for solving this problem. First, we analytically show that existing in-processing fair algorithms have fundamental limits in accuracy and group fairness. We introduce the notion of co…
Towards Collaborative Fairness in Federated Learning Under Imbalanced Covariate Shift
The Pennsylvania State University（宾夕法尼亚州立大学）；State University of New York at Albany（纽约州立大学阿尔巴尼分校）；University of Minnesota Twin Cities（明尼苏达大学双城分校）；Dalian University of Technology（大连理工大学）
FADE: Towards Fairness-aware Generation for Domain Generalization via Classifier-Guided Score-based Diffusion Models
School of New Media and Communication, Tianjin University（天津大学新媒体与传播学院）；School of Informatics, Xiamen University（厦门大学信息学院）；Department of Computer Science, Baylor University（贝勒大学计算机科学系）；Departments of Biostatistics and Epidemiology, Harvard University（哈佛大学生物统计学与流行病学系）
A New Perspective to Boost Performance Fairness for Medical Federated Learning
Australian National University（澳大利亚国立大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Guangxi Medical University（广西医科大学）；Institute of High Performance Computing (IHPC), Agency for Science, Technology and Research (A*STAR)（新加坡科技研究局高性能计算研究所）；Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)（穆罕默德·本·扎耶德人工智能大学）；The Hong Kong University of Science and Technology（香港科技大学）
As learning machines increase their influence on decisions concerning human lives, analyzing their fairness properties becomes a subject of central importance. Yet, our best tools for measuring the fairness of learning systems are rigid fairness metrics encapsulated as mathematical one-liners, offer limited power to the stakeholders involved in the prediction task, and are easy to manipulate when we exhort excessive pressure to optimize them. To advance these issues, we propose to shift focus from shaping fairness metrics to curating the distributions of examples under which these are computed…
Causal Pre-training Under the Fairness Lens: An Empirical Study of TabPFN
University of Bergen Centre for the Science of Learning \& Technology (SLATE) Bergen Norway；Department of Computer Science University of Oxford；Alan Turing Institute Oxford United Kingdom；University of Bergen；University of Oxford；Alan Turing Institute
Fair Federated Medical Image Classification Against Quality Shift via Inter-Client Progressive State Matching
School of Electronic Information and Communications, Huazhong University of Science and Technology（华中科技大学电子信息与通信学院）；Lassonde School of Engineering, York University（约克大学拉萨onde工程学院）
