Data sharing is crucial for open science and reproducible research, but the legal sharing of clinical data requires the removal of protected health information from electronic health records. This process, known as de-identification, is often achieved through the use of machine learning algorithms by many commercial and open-source systems. While these systems have shown compelling results on average, the variation in their performance across different demographic groups has not been thoroughly examined. In this work, we investigate the bias of de-identification systems on names in clinical notes via a large-scale empirical analysis. To achieve this, we create 16 name sets that vary along four demographic dimensions: gender, race, name popularity, and the decade of popularity. We insert these names into 100 manually curated clinical templates and evaluate the performance of nine public and private de-identification methods. Our findings reveal that there are statistically significant performance gaps along a majority of the demographic dimensions in most methods. We further illustrate that de-identification quality is affected by polysemy in names, gender context, and clinical note characteristics. To mitigate the identified gaps, we propose a simple and method-agnostic solution by fine-tuning de-identification methods with clinical context and diverse names. Overall, it is imperative to address the bias in existing methods immediately so that downstream stakeholders can build high-quality systems to serve all demographic parties fairly.
Eindhoven University of Technology（埃因霍温理工大学）；Microsoft（微软）；Google（谷歌）
Fairlearn is an open source project to help practitioners assess and improve fairness of artificial intelligence (AI) systems. The associated Python library, also named fairlearn, supports evaluation of a model's output across affected populations and includes several algorithms for mitigating fairness issues. Grounded in the understanding that fairness is a sociotechnical challenge, the project integrates learning resources that aid practitioners in considering a system's broader societal context.
KenSci Inc（KenSci公司）；University of Washington - Tacoma（华盛顿大学塔科马分校）；University of Washington - Bothell（华盛顿大学博塞尔分校）
Fairness in AI and machine learning systems has become a fundamental problem in the accountability of AI systems. While the need for accountability of AI models is near ubiquitous, healthcare in particular is a challenging field where accountability of such systems takes upon additional importance, as decisions in healthcare can have life altering consequences. In this paper we present preliminary results on fairness in the context of classification parity in healthcare. We also present some exploratory methods to improve fairness and choosing appropriate classification algorithms in the conte…
The University of Melbourne（墨尔本大学）；Huawei Technologies Co., Ltd.（华为技术有限公司）；MBZUAI（穆罕默德·本·扎耶德人工智能大学）
This paper presents fairlib, an open-source framework for assessing and improving classification fairness. It provides a systematic framework for quickly reproducing existing baseline models, developing new methods, evaluating models with different metrics, and visualizing their results. Its modularity and extensibility enable the framework to be used for diverse types of inputs, including natural language, images, and audio. In detail, we implement 14 debiasing methods, including pre-processing, at-training-time, and post-processing approaches. The built-in metrics cover the most commonly use…
Department of Computer Science（计算机科学系）；Department of Cognitive Sciences University of California, Irvine（加州大学尔湾分校认知科学系）
We investigate the problem of reliably assessing group fairness when labeled examples are few but unlabeled examples are plentiful. We propose a general Bayesian framework that can augment labeled data with unlabeled data to produce more accurate and lower-variance estimates compared to methods based on labeled data alone. Our approach estimates calibrated scores for unlabeled examples in each group using a hierarchical latent variable model conditioned on labeled examples. This in turn allows for inference of posterior distributions with associated notions of uncertainty for a variety of grou…
Recently pre-trained multimodal models, such as CLIP, have shown exceptional capabilities towards connecting images and natural language. The textual representations in English can be desirably transferred to multilingualism and support downstream multimodal tasks for different languages. Nevertheless, the principle of multilingual fairness is rarely scrutinized: do multilingual multimodal models treat languages equally? Are their performances biased towards particular languages? To answer these questions, we view language as the fairness recipient and introduce two new fairness notions, multi…
University of South Australia（南澳大学）；Simon Fraser University（西蒙菲莎大学）
The increasing application of machine learning techniques in everyday decision-making processes has brought concerns about the fairness of algorithmic decision-making. This paper concerns the problem of collider bias which produces spurious associations in fairness assessment and develops theorems to guide fairness assessment avoiding the collider bias. We consider a real-world application of auditing a trained classifier by an audit agency. We propose an unbiased assessment algorithm by utilising the developed theorems to reduce collider biases in the assessment. Experiments and simulations s…
Technical University of Denmark（丹麦技术大学）；Pioneer Centre for AI（人工智能先锋中心）；University of Copenhagen（哥本哈根大学）；Rigshospitalet（哥本哈根大学国立医院）
Recent work on algorithmic fairness has largely focused on the fairness of discrete decisions, or classifications. While such decisions are often based on risk score models, the fairness of the risk models themselves has received considerably less attention. Risk models are of interest for a number of reasons, including the fact that they communicate uncertainty about the potential outcomes to users, thus representing a way to enable meaningful human oversight. Here, we address fairness desiderata for risk score models. We identify the provision of similar epistemic value to different groups a…
Missing data are prevalent and present daunting challenges in real data analysis. While there is a growing body of literature on fairness in analysis of fully observed data, there has been little theoretical work on investigating fairness in analysis of incomplete data. In practice, a popular analytical approach for dealing with missing data is to use only the set of complete cases, i.e., observations with all features fully observed to train a prediction algorithm. However, depending on the missing data mechanism, the distribution of complete cases and the distribution of the complete data ma…
Disease identification is a core, routine activity in observational health research. Cohorts impact downstream analyses, such as how a condition is characterized, how patient risk is defined, and what treatments are studied. It is thus critical to ensure that selected cohorts are representative of all patients, independently of their demographics or social determinants of health. While there are multiple potential sources of bias when constructing phenotype definitions which may affect their fairness, it is not standard in the field of phenotyping to consider the impact of different definition…
Trilateral Research（三边研究公司）；Data Justice Lab, Cardiff University（卡迪夫大学数据正义实验室）；University of Granada（格拉纳达大学）
Fair machine learning works have been focusing on the development of equitable algorithms that address discrimination of certain groups. Yet, many of these fairness-aware approaches aim to obtain a unique solution to the problem, which leads to a poor understanding of the statistical limits of bias mitigation interventions. We present the first methodology that allows to explore those limits within a multi-objective framework that seeks to optimize any measure of accuracy and fairness and provides a Pareto front with the best feasible solutions. In this work, we focus our study on decision tre…
The increasing impact of algorithmic decisions on people's lives compels us to scrutinize their fairness and, in particular, the disparate impacts that ostensibly-color-blind algorithms can have on different groups. Examples include credit decisioning, hiring, advertising, criminal justice, personalized medicine, and targeted policymaking, where in some cases legislative or regulatory frameworks for fairness exist and define specific protected classes. In this paper we study a fundamental challenge to assessing disparate impacts in practice: protected class membership is often not observed in …
Assessing Dialect Fairness and Robustness of Large Language Models in Reasoning Tasks
University of Oxford（牛津大学）；Microsoft Research（微软研究院）；The Alan Turing Institute（艾伦·图灵研究所）；Allen Institute for AI（人工智能研究所）；University of Washington（华盛顿大学）；Microsoft Corporation（微软公司）；University of York（约克大学）
University of Stavanger（斯塔万格大学）；Stavanger University Hospital（斯塔万格大学医院）；Universidad de Antioquia（安蒂奥基亚大学）
As the number of automatic tools based on machine learning (ML) and resting-state electroencephalography (rs-EEG) for Parkinson's disease (PD) detection keeps growing, the assessment of possible exacerbation of health disparities by means of fairness and bias analysis becomes more relevant. Protected attributes, such as gender, play an important role in PD diagnosis development. However, analysis of sub-group populations stemming from different genders is seldom taken into consideration in ML models' development or the performance assessment for PD detection. In this work, we perform a systema…
Named Entity Recognition (NER) is often the first step towards automated Knowledge Base (KB) generation from raw text. In this work, we assess the bias in various Named Entity Recognition (NER) systems for English across different demographic groups with synthetically generated corpora. Our analysis reveals that models perform better at identifying names from specific demographic groups across two datasets. We also identify that debiased embeddings do not help in resolving this issue. Finally, we observe that character-based contextualized word representation models such as ELMo results in the…
Demographic classification is essential in fairness assessment in recommender systems or in measuring unintended bias in online networks and voting systems. Important fields like education and politics, which often lay a foundation for the future of equality in society, need scrutiny to design policies that can better foster equality in resource distribution constrained by the unbalanced demographic distribution of people in the country. We collect three publicly available datasets to train state-of-the-art classifiers in the domain of gender and caste classification. We train the models in th…
The Johns Hopkins University Applied Physics Laboratory（约翰斯·霍普金斯大学应用物理实验室）；Queens University（女王大学）；Johns Hopkins University（约翰斯·霍普金斯大学）
We propose a novel method for enforcing AI fairness with respect to protected or sensitive factors. This method uses a dual strategy performing training and representation alteration (TARA) for the mitigation of prominent causes of AI bias by including: a) the use of representation learning alteration via adversarial independence to suppress the bias-inducing dependence of the data representation from protected factors; and b) training set alteration via intelligent augmentation to address bias-causing data imbalance, by using generative models that allow the fine control of sensitive factors …
Mila – Quebec AI Institute（米拉-魁北克人工智能研究所）；University of Washington Tacoma（华盛顿大学塔科马分校）；Ghent University（根特大学）
Machine learning (ML) has become prominent in applications that directly affect people's quality of life, including in healthcare, justice, and finance. ML models have been found to exhibit discrimination based on sensitive attributes such as gender, race, or disability. Assessing if an ML model is free of bias remains challenging to date, and by definition has to be done with sensitive user characteristics that are subject of anti-discrimination and data protection law. Existing libraries for fairness auditing of ML models offer no mechanism to protect the privacy of the audit data. We presen…
University of Maryland（马里兰大学）；University of St. Thomas（圣托马斯大学）；Montgomery Blair High School（蒙哥马利布莱尔高中）；Microsoft Research（微软研究院）
We propose new tools for policy-makers to use when assessing and correcting fairness and bias in AI algorithms. The three tools are: - A new definition of fairness called "controlled fairness" with respect to choices of protected features and filters. The definition provides a simple test of fairness of an algorithm with respect to a dataset. This notion of fairness is suitable in cases where fairness is prioritized over accuracy, such as in cases where there is no "ground truth" data, only data labeled with past decisions (which may have been biased). - Algorithms for retraining a given class…
Algorithms are now routinely used to make consequential decisions that affect human lives. Examples include college admissions, medical interventions or law enforcement. While algorithms empower us to harness all information hidden in vast amounts of data, they may inadvertently amplify existing biases in the available datasets. This concern has sparked increasing interest in fair machine learning, which aims to quantify and mitigate algorithmic discrimination. Indeed, machine learning models should undergo intensive tests to detect algorithmic biases before being deployed at scale. In this pa…
Cross-Institutional Transfer Learning for Educational Models: Implications for Model Performance, Fairness, and Equity
University of Washington（华盛顿大学）；Columbia University（哥伦比亚大学）；University of British Columbia（不列颠哥伦比亚大学）；University of Michigan（密歇根大学）；Cornell University（康奈尔大学）
Towards Harmless Rawlsian Fairness Regardless of Demographic Prior
School of Computer Science and Engineering, Beihang University（北京航空航天大学计算机科学与工程学院）；Institute of High-Performance Computing, Agency for Science, Technology and Research（新加坡科技研究局高性能计算研究所）；Centre for Frontier AI Research, Agency for Science, Technology and Research（新加坡科技研究局前沿人工智能研究中心）；College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算与数据科学学院）
Understanding challenges to the interpretation of disaggregated evaluations of algorithmic fairness
Google Research, Mountain View, CA, USA（谷歌研究, 山景城, 加州, 美国）；Google DeepMind, Mountain View, CA, USA（谷歌深Mind, 山景城, 加州, 美国）；New York University, New York, NY, USA（纽约大学, 纽约, 纽约, 美国）；Massachusetts Institute of Technology, Cambridge, MA, USA（麻省理工学院, 柏林, 马萨诸塞, 美国）；The Hopsital for Sick Children, Toronto, ON, Canada（儿童医院, 温哥华, 加拿大）；Stanford University, Stanford, CA, USA（斯坦福大学, 斯坦福, 加州, 美国）
Causal Pre-training Under the Fairness Lens: An Empirical Study of TabPFN
University of Bergen Centre for the Science of Learning \& Technology (SLATE) Bergen Norway；Department of Computer Science University of Oxford；Alan Turing Institute Oxford United Kingdom；University of Bergen；University of Oxford；Alan Turing Institute
Social bias in machine learning has drawn significant attention, with work ranging from demonstrations of bias in a multitude of applications, curating definitions of fairness for different contexts, to developing algorithms to mitigate bias. In natural language processing, gender bias has been shown to exist in context-free word embeddings. Recently, contextual word representations have outperformed word embeddings in several downstream NLP tasks. These word representations are conditioned on their context within a sentence, and can also be used to encode the entire sentence. In this paper, w…
On the Origins of Sampling Bias: Implications on Fairness Measurement and Mitigation
University of Doha for Science and Technology (UDST)（多哈科技大学）；CISPA Helmholtz Center for Information Security（CISPA亥姆霍兹信息安全中心）；Université Claude Bernard Lyon 1（里昂第一大学）；Université de Sousse（苏塞大学）
CNRS（法国国家科学研究中心）；AXA（安盛集团）；Polish Academy of Science（波兰科学院）
In recent years, most fairness strategies in machine learning models focus on mitigating unwanted biases by assuming that the sensitive information is observed. However this is not always possible in practice. Due to privacy purposes and var-ious regulations such as RGPD in EU, many personal sensitive attributes are frequently not collected. We notice a lack of approaches for mitigating bias in such difficult settings, in particular for achieving classical fairness objectives such as Demographic Parity and Equalized Odds. By leveraging recent developments for approximate inference, we propose …
