Machine learning (ML) has become prominent in applications that directly affect people's quality of life, including in healthcare, justice, and finance. ML models have been found to exhibit discrimination based on sensitive attributes such as gender, race, or disability. Assessing if an ML model is free of bias remains challenging to date, and by definition has to be done with sensitive user characteristics that are subject of anti-discrimination and data protection law. Existing libraries for fairness auditing of ML models offer no mechanism to protect the privacy of the audit data. We present PrivFair, a library for privacy-preserving fairness audits of ML models. Through the use of Secure Multiparty Computation (MPC), PrivFair protects the confidentiality of the model under audit and the sensitive data used for the audit, hence it supports scenarios in which a proprietary classifier owned by a company is audited using sensitive audit data from an external investigator. We demonstrate the use of PrivFair for group fairness auditing with tabular data or image data, without requiring the investigator to disclose their data to anyone in an unencrypted manner, or the model owner to reveal their model parameters to anyone in plaintext.
Mila – Quebec AI Institute（Mila – 魁北克人工智能研究所）；HEC Montréal（蒙特利尔高等商学院）；University of Washington Tacoma（华盛顿大学塔科马分校）；Ghent University（根特大学）
Group fairness ensures that the outcome of machine learning (ML) based decision making systems are not biased towards a certain group of people defined by a sensitive attribute such as gender or ethnicity. Achieving group fairness in Federated Learning (FL) is challenging because mitigating bias inherently requires using the sensitive attribute values of all clients, while FL is aimed precisely at protecting privacy by not giving access to the clients' data. As we show in this paper, this conflict between fairness and privacy in FL can be resolved by combining FL with Secure Multiparty Computa…
Machine learning (ML) models have been deployed for high-stakes applications. Due to class imbalance in the sensitive attribute observed in the datasets, ML models are unfair on minority subgroups identified by a sensitive attribute, such as race and sex. In-processing fairness algorithms ensure model predictions are independent of sensitive attribute. Furthermore, ML models are vulnerable to attribute inference attacks where an adversary can identify the values of sensitive attribute by exploiting their distinguishable model predictions. Despite privacy and fairness being important pillars of…
University of Tsukuba（筑波大学）；RIKEN AIP（理化学研究所人工智能研究中心）；Osaka University（大阪大学）
Auditing fairness of decision-makers is now in high demand. To respond to this social demand, several fairness auditing tools have been developed. The focus of this study is to raise an awareness of the risk of malicious decision-makers who fake fairness by abusing the auditing tools and thereby deceiving the social communities. The question is whether such a fraud of the decision-maker is detectable so that the society can avoid the risk of fake fairness. In this study, we answer this question negatively. We specifically put our focus on a situation where the decision-maker publishes a benchm…
University of California, Davis（加州大学戴维斯分校）；University of Virginia（弗吉尼亚大学）；University at Albany – State University of New York（纽约州立大学奥尔巴尼分校）
Existing work on fairness typically focuses on making known machine learning algorithms fairer. Fair variants of classification, clustering, outlier detection and other styles of algorithms exist. However, an understudied area is the topic of auditing an algorithm's output to determine fairness. Existing work has explored the two group classification problem for binary protected status variables using standard definitions of statistical parity. Here we build upon the area of auditing by exploring the multi-group setting under more complex definitions of fairness.
Carnegie Mellon University（卡内基梅隆大学）；University of Arizona（亚利桑那大学）
The fast spreading adoption of machine learning (ML) by companies across industries poses significant regulatory challenges. One such challenge is scalability: how can regulatory bodies efficiently audit these ML models, ensuring that they are fair? In this paper, we initiate the study of query-based auditing algorithms that can estimate the demographic parity of ML models in a query-efficient manner. We propose an optimal deterministic algorithm, as well as a practical randomized, oracle-efficient algorithm with comparable guarantees. Furthermore, we make inroads into understanding the optima…
University of California, Los Angeles（加利福尼亚大学洛杉矶分校）
Auditing machine learning-based (ML) healthcare tools for bias is critical to preventing patient harm, especially in communities that disproportionately face health inequities. General frameworks are becoming increasingly available to measure ML fairness gaps between groups. However, ML for health (ML4H) auditing principles call for a contextual, patient-centered approach to model assessment. Therefore, ML auditing tools must be (1) better aligned with ML4H auditing principles and (2) able to illuminate and characterize communities vulnerable to the most harm. To address this gap, we propose s…
QuantumBlack, a McKinsey company（QuantumBlack，麦肯锡旗下公司）；University of Warwick（华威大学）
Machine learning algorithms are extensively used to make increasingly more consequential decisions about people, so achieving optimal predictive performance can no longer be the only focus. A particularly important consideration is fairness with respect to race, gender, or any other sensitive attribute. This paper studies intersectional fairness, where intersections of multiple sensitive attributes are considered. Prior research has mainly focused on fairness with respect to a single sensitive attribute, with intersectional fairness being comparatively less studied despite its critical importa…
Missouri University of Science and Technology（密苏里科技大学）
Bias evaluation in machine-learning based services (MLS) based on traditional algorithmic fairness notions that rely on comparative principles is practically difficult, making it necessary to rely on human auditor feedback. However, in spite of taking rigorous training on various comparative fairness notions, human auditors are known to disagree on various aspects of fairness notions in practice, making it difficult to collect reliable feedback. This paper offers a paradigm shift to the domain of algorithmic fairness via proposing a new fairness notion based on the principle of non-comparative…
University of Pennsylvania（宾夕法尼亚大学）；Microsoft Research-New York City（微软研究院（纽约市））
The most prevalent notions of fairness in machine learning are statistical definitions: they fix a small collection of pre-defined groups, and then ask for parity of some statistic of the classifier across these groups. Constraints of this form are susceptible to intentional or inadvertent "fairness gerrymandering", in which a classifier appears to be fair on each individual group, but badly violates the fairness constraint on one or more structured subgroups defined over the protected attributes. We propose instead to demand statistical notions of fairness across exponentially (or infinitely)…
Université Libre de Bruxelles（布鲁塞尔自由大学）；Athena Research Center（雅典娜研究中心）；Tampere University（坦佩雷大学）
This paper studies algorithmic fairness when the protected attribute is location. To handle protected attributes that are continuous, such as age or income, the standard approach is to discretize the domain into predefined groups, and compare algorithmic outcomes across groups. However, applying this idea to location raises concerns of gerrymandering and may introduce statistical bias. Prior work addresses these concerns but only for regularly spaced locations, while raising other issues, most notably its inability to discern regions that are likely to exhibit spatial unfairness. Similar to es…
In recent years, discussions about fairness in machine learning, AI ethics and algorithm audits have increased. Many entities have developed framework guidance to establish a baseline rubric for fairness and accountability. However, in spite of increased discussions and multiple frameworks, algorithm and data auditing still remain difficult to execute in practice. Many open-source auditing tools are available, but users aren't always aware of the tools, what they are useful for, or how to access them. Model auditing and evaluation are not frequently emphasized skills in machine learning. There…
German Aerospace Center (DLR)（德国航空航天中心）；Technical University of Munich (TUM)（慕尼黑工业大学）
Humanitarian mapping from space with machine learning helps policy-makers to timely and accurately identify people in need. However, recent concerns around fairness and transparency of algorithmic decision-making are a significant obstacle for applying these methods in practice. In this paper, we study if humanitarian mapping approaches from space are prone to bias in their predictions. We map village-level poverty and electricity rates in India based on nighttime lights (NTLs) with linear regression and random forest and analyze if the predictions systematically show prejudice against schedul…
Exposing the Illusion of Fairness: Auditing Vulnerabilities to Distributional Manipulation Attacks
Institut de Mathématiques de Toulouse (IMT)（图卢兹数学研究所）；Institut national de recherche en sciences et technologies du numérique (INRIA)（国家数字科学与技术研究院）；Artificial and Natural Intelligence Toulouse Institute 2 (ANITI 2)（图卢兹人工智能与自然智能研究所2）；Institut de Recherche en Informatique de Toulouse (IRIT)（图卢兹计算机研究所）；Independent Researcher（独立研究员）；CNRS IRL CROSSING（国家科学研究中心IRL CROSSING）
From Fragile to Certified: Wasserstein Audits of Group Fairness Under Distribution Shift
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Tübingen AI Center（图宾根人工智能中心）；Mila Québec AI Institute（魁北克人工智能研究所）；McGill University（麦吉尔大学）
University of Illinois at Chicago（伊利诺伊大学芝加哥分校）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）
Colleges and universities use predictive analytics in a variety of ways to increase student success rates. Despite the potential for predictive analytics, two major barriers exist to their adoption in higher education: (a) the lack of democratization in deployment, and (b) the potential to exacerbate inequalities. Education researchers and policymakers encounter numerous challenges in deploying predictive modeling in practice. These challenges present in different steps of modeling including data preparation, model development, and evaluation. Nevertheless, each of these steps can introduce ad…
As AI-based systems increasingly impact many areas of our lives, auditing these systems for fairness is an increasingly high-stakes problem. Traditional group fairness metrics can miss discrimination against individuals and are difficult to apply after deployment. Counterfactual fairness describes an individualized notion of fairness but is even more challenging to evaluate after deployment. We present prediction sensitivity, an approach for continual audit of counterfactual fairness in deployed classifiers. Prediction sensitivity helps answer the question: would this prediction have been diff…
Certified in Theory, Broken in Practice: Assumption Gaps in Cryptographic Model Certification
University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）；Vector Institute & University of Toronto（向量研究所及多伦多大学）；Carnegie Mellon University（卡内基梅隆大学）；J.P.Morgan AI Research & AlgoCRYPT CoE（摩根大通人工智能研究及AlgoCRYPT中心）；University of Toronto（多伦多大学）
Recommender systems are facing scrutiny because of their growing impact on the opportunities we have access to. Current audits for fairness are limited to coarse-grained parity assessments at the level of sensitive groups. We propose to audit for envy-freeness, a more granular criterion aligned with individual preferences: every user should prefer their recommendations to those of other users. Since auditing for envy requires to estimate the preferences of users beyond their existing recommendations, we cast the audit as a new pure exploration problem in multi-armed bandits. We propose a sampl…
Center for Data Science and Public Policy（数据科学与公共政策中心）；University of Chicago（芝加哥大学）
Recent work has raised concerns on the risk of unintended bias in AI systems being used nowadays that can affect individuals unfairly based on race, gender or religion, among other possible characteristics. While a lot of bias metrics and fairness definitions have been proposed in recent years, there is no consensus on which metric/definition should be used and there are very few available resources to operationalize them. Therefore, despite recent awareness, auditing for bias and fairness when developing and deploying AI systems is not yet a standard practice. We present Aequitas, an open sou…
School of Social Sciences, University of Mannheim（曼海姆大学社会科学学院）；dmTECH；LMU Munich（慕尼黑大学）；Joint Program in Survey Methodology, University of Maryland（马里兰大学联合调查方法学项目）
Algorithmic profiling is increasingly used in the public sector as a means to allocate limited public resources effectively and objectively. One example is the prediction-based statistical profiling of job seekers to guide the allocation of support measures by public employment services. However, empirical evaluations of potential side-effects such as unintended discrimination and fairness concerns are rare. In this study, we compare and evaluate statistical models for predicting job seekers' risk of becoming long-term unemployed with respect to prediction performance, fairness metrics, and vu…
