Open challenges have become the de facto standard for comparative ranking of medical AI methods. Despite their importance, medical AI leaderboards exhibit three persistent limitations: (1) score gaps are rarely tested for statistical significance, so rank stability is unknown; (2) single averaged metrics are applied to every organ, hiding clinically important boundary errors; (3) performance across intersecting demographics is seldom reported, masking fairness and equity gaps. We introduce RankInsight, an open-source toolkit that seeks to address these limitations. RankInsight (1) computes pair-wise significance maps that show the nnU-Net family outperforms Vision-Language and MONAI submissions with high statistical certainty; (2) recomputes leaderboards with organ-appropriate metrics, reversing the order of the top four models when Dice is replaced by NSD for tubular structures; and (3) audits intersectional fairness, revealing that more than half of the MONAI-based entries have the largest gender-race discrepancy on our proprietary Johns Hopkins Hospital dataset. The RankInsight toolkit is publicly released and can be directly applied to past, ongoing, and future challenges. It enables organizers and participants to publish rankings that are statistically sound, clinically meaningful, and demographically fair.
QuantumBlack, a McKinsey company（QuantumBlack，麦肯锡旗下公司）；University of Warwick（华威大学）
Machine learning algorithms are extensively used to make increasingly more consequential decisions about people, so achieving optimal predictive performance can no longer be the only focus. A particularly important consideration is fairness with respect to race, gender, or any other sensitive attribute. This paper studies intersectional fairness, where intersections of multiple sensitive attributes are considered. Prior research has mainly focused on fairness with respect to a single sensitive attribute, with intersectional fairness being comparatively less studied despite its critical importa…
University of Tsukuba（筑波大学）；RIKEN AIP（理化学研究所人工智能研究中心）；Osaka University（大阪大学）
Auditing fairness of decision-makers is now in high demand. To respond to this social demand, several fairness auditing tools have been developed. The focus of this study is to raise an awareness of the risk of malicious decision-makers who fake fairness by abusing the auditing tools and thereby deceiving the social communities. The question is whether such a fraud of the decision-maker is detectable so that the society can avoid the risk of fake fairness. In this study, we answer this question negatively. We specifically put our focus on a situation where the decision-maker publishes a benchm…
University of Pennsylvania（宾夕法尼亚大学）；Microsoft Research-New York City（微软研究院（纽约市））
The most prevalent notions of fairness in machine learning are statistical definitions: they fix a small collection of pre-defined groups, and then ask for parity of some statistic of the classifier across these groups. Constraints of this form are susceptible to intentional or inadvertent "fairness gerrymandering", in which a classifier appears to be fair on each individual group, but badly violates the fairness constraint on one or more structured subgroups defined over the protected attributes. We propose instead to demand statistical notions of fairness across exponentially (or infinitely)…
The ImageNet dataset ushered in a flood of academic and industry interest in deep learning for computer vision applications. Despite its significant impact, there has not been a comprehensive investigation into the demographic attributes of images contained within the dataset. Such a study could lead to new insights on inherent biases within ImageNet, particularly important given it is frequently used to pretrain models for a wide variety of computer vision tasks. In this work, we introduce a model-driven framework for the automatic annotation of apparent age and gender attributes in large-sca…
FairFedMed: Benchmarking Group Fairness in Federated Medical Imaging with FairLoRA
Harvard AI and Robotics Lab（哈佛人工智能与机器人实验室）；Harvard University（哈佛大学）；Harvard Ophthalmology AI lab（哈佛眼科人工智能实验室）；University of Central Florida（佛罗里达中央大学）；University of Louisiana at Lafayette（路易斯安那州立大学拉斐特分校）；New York University（纽约大学）；NYUAD Center for Artificial Intelligence and Robotics（纽约大学阿布扎克人工智能与机器人中心）
Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；Google Research（谷歌研究院）
Much of the previous machine learning (ML) fairness literature assumes that protected features such as race and sex are present in the dataset, and relies upon them to mitigate fairness concerns. However, in practice factors like privacy and regulation often preclude the collection of protected features, or their use for training or inference, severely limiting the applicability of traditional fairness research. Therefore we ask: How can we train an ML model to improve fairness when we do not even know the protected group memberships? In this work we address this problem by proposing Adversari…
Person-Centric Annotations of LAION-400M: Auditing Bias and Its Transfer to Models
Technical University of Munich, Munich Center for Machine Learning (MCML), MDSI（慕尼黑工业大学，慕尼黑机器学习中心（MCML），MDSI）；LTCI, Télécom Paris, Institut Polytechnique de Paris, France（法国巴黎理工学院，巴黎电信学院，LTCI）；Helmholtz Munich（亥姆霍兹慕尼黑中心）；University of California, Berkeley（加州大学伯克利分校）
Auditing Demographic Bias in Facial Landmark Detection for Fair Human-Robot Interaction
Departamento de Inteligencia Artificial, Universidad Politécnica de Madrid（智能人工智能部门，马德里理工大学）；Departamento de Informática y Estadística, Universidad Rey Juan Carlos（信息与统计学部门，皇家胡安·卡洛斯大学）；ELLIS Unit Madrid（马德里ELLIS单位）
Colorimeter-Supervised Skin Tone Estimation from Dermatoscopic Images for Fairness Auditing
Faculty of Electrical Engineering, Computer Science and Information Technology Osijek, Josip Juraj Strossmayer University of Osijek（奥西杰克大学电子工程、计算机科学与信息科技学院）
FairForensics: Seeing Expressions and Parsing Demographics via Vision-Language Modeling for Generalizable Fair Deepfake Detection
Shenzhen University（深圳大学）；Shandong Artificial Intelligence Institute（山东省人工智能研究院）；Qilu University of Technology (Shandong Academy of Sciences)（齐鲁工业大学（山东省科学院））；Tianjin University of Technology（天津理工大学）
There has been increasing awareness of ethical issues in machine learning, and fairness has become an important research topic. Most fairness efforts in computer vision have been focused on human sensing applications and preventing discrimination by people's physical attributes such as race, skin color or age by increasing visual representation for particular demographic groups. We argue that ML fairness efforts should extend to object recognition as well. Buildings, artwork, food and clothing are examples of the objects that define human culture. Representing these objects fairly in machine l…
Mitigating Group-Level Fairness Disparities in Federated Visual Language Models
Great Bay University（大湾大学）；Tsinghua Shenzhen International Graduate School（清华大学深圳国际研究生院）；Nanyang Technological University（南洋理工大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；Shenzhen University（深圳大学）；Pengcheng Laboratory（鹏城实验室）；Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区）
Hochschule Darmstadt（达姆施塔特应用技术大学）；Fraunhofer Institute for Computer Graphics Research（弗劳恩霍夫计算机图形学研究所）；TU Darmstadt（达姆施塔特工业大学）
Algorithmic decision systems have frequently been labelled as "biased", "racist", "sexist", or "unfair" by numerous media outlets, organisations, and researchers. There is an ongoing debate whether such assessments are justified and whether citizens and policymakers should be concerned. These and other related matters have recently become a hot topic in the context of biometric technologies, which are ubiquitous in personal, commercial, and governmental applications. Biometrics represent an essential component of many surveillance, access control, and operational identity management systems, t…
T2I-BiasBench: A Multi-Metric Framework for Auditing Demographic and Cultural Bias in Text-to-Image Models
Department of Information Technology, Rajkiya Engineering College Banda（拉贾基亚工程学院班达信息科技系）；School of AI and Data Science, Indian Institute of Technology Jodhpur（印度理工学院乔普拉人工智能与数据科学学院）
Recently, different researchers have found that the gallery composition of a face database can induce performance differentials to facial identification systems in which a probe image is compared against up to all stored reference images to reach a biometric decision. This negative effect is referred to as "watchlist imbalance effect". In this work, we present a method to theoretically estimate said effect for a biometric identification system given its verification performance across demographic groups and the composition of the used gallery. Further, we report results for identification expe…
University of Wisconsin–Madison（威斯康星大学麦迪逊分校）；University of Michigan（密歇根大学）
The proliferation of automated face recognition in the commercial and government sectors has caused significant privacy concerns for individuals. One approach to address these privacy concerns is to employ evasion attacks against the metric embedding networks powering face recognition systems: Face obfuscation systems generate imperceptibly perturbed images that cause face recognition systems to misidentify the user. Perturbed faces are generated on metric embedding networks, which are known to be unfair in the context of face recognition. A question of demographic fairness naturally follows: …
Fairness in Multi-modal Medical Diagnosis with Demonstration Selection
Arizona State University（亚利桑那州立大学）；University of Rochester（罗切斯特大学）；University of Virginia（弗吉尼亚大学）；UCL（伦敦大学学院）；University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；University of Central Florida（佛罗里达中央大学）
Decoupling Bias, Aligning Distributions: Synergistic Fairness Optimization for Deepfake Detection
Nanchang University（南昌大学）；Guangdong Provincial Key Laboratory of Intelligent Information Processing（广东省智能信息处理重点实验室）；Shenzhen Key Laboratory of Media Security（深圳媒体安全重点实验室）；SZU–AFS Joint Innovation Center for AI Technology（深圳大学-AFS人工智能技术联合创新中心）；Shenzhen University（深圳大学）；Purdue University（普渡大学）
Face biometrics are playing a key role in making modern smart city applications more secure and usable. Commonly, the recognition threshold of a face recognition system is adjusted based on the degree of security for the considered use case. The likelihood of a match can be for instance decreased by setting a high threshold in case of a payment transaction verification. Prior work in face recognition has unfortunately showed that error rates are usually higher for certain demographic groups. These disparities have hence brought into question the fairness of systems empowered with face biometri…
Microsoft AI R&D Center, Taiwan（台湾微软人工智能研发中心）；National Tsing Hua University（国立清华大学）
Although significant progress has been made in face recognition, demographic bias still exists in face recognition systems. For instance, it usually happens that the face recognition performance for a certain demographic group is lower than the others. In this paper, we propose MixFairFace framework to improve the fairness in face recognition models. First of all, we argue that the commonly used attribute-based fairness metric is not appropriate for face recognition. A face recognition system can only be considered fair while every person has a close performance. Hence, we propose a new evalua…
On the Illusion of Gender Bias in Face Recognition: Explaining the Fairness Issue Through Non-demographic Attributes
Paderborn University（帕德博恩大学）；Technical University of Darmstadt（达姆施塔特技术大学）；University of Notre Dame（诺特丹大学）；Johannes Gutenberg University Mainz（美因茨约翰· Gutenberg大学）
