Are generative models fair? A study of racial bias in dermatological image generation
Instituto Universitario de Investigación en Tecnología Centrada en el Ser Humano, Universitat Politècnica de València（瓦伦西亚理工大学 以人为本技术大学研究所）；Technical University of Denmark（丹麦技术大学）
Racial bias in medicine, such as in dermatology, presents significant ethical and clinical challenges. This is likely to happen because there is a significant underrepresentation of darker skin tones in training datasets for machine learning models. While efforts to address bias in dermatology have focused on improving dataset diversity and mitigating disparities in discriminative models, the impact of racial bias on generative models remains underexplored. Generative models, such as Variational Autoencoders (VAEs), are increasingly used in healthcare applications, yet their fairness across diverse skin tones is currently not well understood. In this study, we evaluate the fairness of generative models in clinical dermatology with respect to racial bias. For this purpose, we first train a VAE with a perceptual loss to generate and reconstruct high-quality skin images across different skin tones. We utilize the Fitzpatrick17k dataset to examine how racial bias influences the representation and performance of these models. Our findings indicate that VAE performance is, as expected, influenced by representation, i.e. increased skin tone representation comes with increased performance on the given skin tone. However, we also observe, even independently of representation, that the VAE performs better for lighter skin tones. Additionally, the uncertainty estimates produced by the VAE are ineffective in assessing the model's fairness. These results highlight the need for more representative dermatological datasets, but also a need for better understanding the sources of bias in such model, as well as improved uncertainty quantification mechanisms to detect and address racial bias in generative models for trustworthy healthcare technologies.
University of Notre Dame（圣母大学）；University of Pittsburgh（匹兹堡大学）
Dermatological diseases pose a major threat to the global health, affecting almost one-third of the world's population. Various studies have demonstrated that early diagnosis and intervention are often critical to prognosis and outcome. To this end, the past decade has witnessed the rapid evolvement of deep learning based smartphone apps, which allow users to conveniently and timely identify issues that have emerged around their skins. In order to collect sufficient data needed by deep learning and at the same time protect patient privacy, federated learning is often used, where individual cli…
FairSkin: Fair Diffusion for Skin Disease Image Generation
Arizona State University（亚利桑那州立大学）；University of Pittsburgh（匹兹堡大学）；University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；Michigan State University（密歇根州立大学）
Trustworthy and Fair SkinGPT-R1 for Democratizing Dermatological Reasoning across Diverse Ethnicities
School of Data Science, The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen)（数据科学学院，香港中文大学（深圳））；Faculty of Information Technology, Monash University（信息技术学院，莫纳什大学）；Department of Dermatology, Tianjin Institute of Integrative Dermatology, Tianjin Academy of Traditional Chinese Medicine Affiliated Hospital（皮肤科，天津整合皮肤科研究所，天津中医研究院附属医院）；Department of Dermatology, The First Affiliated Hospital, Shantou University Medical College（皮肤科，汕头大学医学院第一附属医院）；Department of Dermatology, Beijing AnZhen Hospital, Capital Medical University（皮肤科，北京安贞医院，首都医科大学）；School of Computing and Data Science, The University of Hong Kong（计算与数据科学学院，香港大学）；Institute of Automation, Chinese Academy of Sciences（自动化研究所，中国科学院）；Department of Dermatology, Beijing Aerospace General Hospital（皮肤科，北京航天总医院）
Guangdong Provincial People’s Hospital（广东省人民医院）；University of Notre Dame（圣母大学）；University of Pittsburgh（匹兹堡大学）
Many works have shown that deep learning-based medical image classification models can exhibit bias toward certain demographic attributes like race, gender, and age. Existing bias mitigation methods primarily focus on learning debiased models, which may not necessarily guarantee all sensitive information can be removed and usually comes with considerable accuracy degradation on both privileged and unprivileged groups. To tackle this issue, we propose a method, FairPrune, that achieves fairness by pruning. Conventionally, pruning is used to reduce the model size for efficient inference. However…
Bayesian generative models can flag performance loss, bias, and out-of-distribution image content
Instituto Universitario de Investigación en Tecnología Centrada en el Ser Humano, Universitat Politècnica de València（瓦伦西亚理工大学人类中心技术研究大学学院）；Technical University of Denmark（丹麦技术大学）
Computer vision is widely deployed, has highly visible, society altering applications, and documented problems with bias and representation. Datasets are critical for benchmarking progress in fair computer vision, and often employ broad racial categories as population groups for measuring group fairness. Similarly, diversity is often measured in computer vision datasets by ascribing and counting categorical race labels. However, racial categories are ill-defined, unstable temporally and geographically, and have a problematic history of scientific use. Although the racial categories used across…
King's College London（伦敦国王学院）；Guy’s and St Thomas' Hospital（盖伊和圣托马斯医院）；University of Oxford（牛津大学）；Queen Mary University London（伦敦玛丽女王大学）；St Bartholomew’s Hospital（圣巴塞洛缪医院）；Barts Health NHS Trust（巴茨健康国民保健服务信托基金）；Health Data Research UK（英国健康数据研究所）；Alan Turing Institute（艾伦·图灵研究所）
The subject of "fairness" in artificial intelligence (AI) refers to assessing AI algorithms for potential bias based on demographic characteristics such as race and gender, and the development of algorithms to address this bias. Most applications to date have been in computer vision, although some work in healthcare has started to emerge. The use of deep learning (DL) in cardiac MR segmentation has led to impressive results in recent years, and such techniques are starting to be translated into clinical practice. However, no work has yet investigated the fairness of such models. In this work, …
Nanjing University of Science and Technology（南京理工大学）；University of Chinese Academy of Sciences（中国科学院大学）；CAS Center for Excellence in Brain Science and Intelligence Technology（中国科学院脑科学与智能技术卓越创新中心）
Face recognition (FR) has made extraordinary progress owing to the advancement of deep convolutional neural networks. However, demographic bias among different racial cohorts still challenges the practical face recognition system. The race factor has been proven to be a dilemma for fair FR (FFR) as the subject-related specific attributes induce the classification bias whilst carrying some useful cues for FR. To mitigate racial bias and meantime preserve robust FR, we abstract face identity-related representation as a signal denoising problem and propose a progressive cross transformer (PCT) me…
Beijing University of Posts and Telecommunications（北京邮电大学）
Racial equality is an important theme of international human rights law, but it has been largely obscured when the overall face recognition accuracy is pursued blindly. More facts indicate racial bias indeed degrades the fairness of recognition system and the error rates on non-Caucasians are usually much higher than Caucasians. To encourage fairness, we introduce the idea of adaptive margin to learn balanced performance for different races based on large margin losses. A reinforcement learning based race balance network (RL-RBN) is proposed. We formulate the process of finding the optimal mar…
University of Maryland（马里兰大学）；University of Chicago（芝加哥大学）
Many existing works have made great strides towards reducing racial bias in face recognition. However, most of these methods attempt to rectify bias that manifests in models during training instead of directly addressing a major source of the bias, the dataset itself. Exceptions to this are BUPT-Balancedface/RFW and Fairface, but these works assume that primarily training on a single race or not racially balancing the dataset are inherently disadvantageous. We demonstrate that these assumptions are not necessarily valid. In our experiments, training on only African faces induced less bias than…
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）
Virtual facial avatars will play an increasingly important role in immersive communication, games and the metaverse, and it is therefore critical that they be inclusive. This requires accurate recovery of the appearance, represented by albedo, regardless of age, sex, or ethnicity. While significant progress has been made on estimating 3D facial geometry, albedo estimation has received less attention. The task is fundamentally ambiguous because the observed color is a function of albedo and lighting, both of which are unknown. We find that current methods are biased towards light skin tones due…
Large scale image classification models trained on top of popular datasets such as Imagenet have shown to have a distributional skew which leads to disparities in prediction accuracies across different subsections of population demographics. A lot of approaches have been made to solve for this distributional skew using methods that alter the model pre, post and during training. We investigate one such approach - which uses a multi-label softmax loss with cross-entropy as the loss function instead of a binary cross-entropy on a multi-label classification problem on the Inclusive Images dataset …
Harvard Glaucoma Fairness: A Retinal Nerve Disease Dataset for Fairness Learning and Fair Identity Normalization
Harvard Medical School（哈佛医学院）；Massachusetts Eye and Ear（麻省眼耳医院）；Schepens Eye Research Institute（谢彭斯眼科研究所）；Eye and Vision Research Institute（眼科与视觉研究所）；Icahn School of Medicine at Mount Sinai（西奈山伊坎医学院）
Face recognition performance has seen a tremendous gain in recent years, mostly due to the availability of large-scale face images dataset that can be exploited by deep neural networks to learn powerful face representations. However, recent research has shown differences in face recognition performance across different ethnic groups mostly due to the racial imbalance in the training datasets where Caucasian identities largely dominate other ethnicities. This is actually symptomatic of the under-representation of non-Caucasian ethnic groups in the celebdom from which face datasets are usually g…
While deep learning based approaches have demonstrated expert-level performance in dermatological diagnosis tasks, they have also been shown to exhibit biases toward certain demographic attributes, particularly skin types (e.g., light versus dark), a fairness concern that must be addressed. We propose CIRCLe, a skin color invariant deep representation learning method for improving fairness in skin lesion classification. CIRCLe is trained to classify images by utilizing a regularization loss that encourages images with the same diagnosis but different skin types to have similar latent represent…
The presence of decision-making algorithms in society is rapidly increasing nowadays, while concerns about their transparency and the possibility of these algorithms becoming new sources of discrimination are arising. In fact, many relevant automated systems have been shown to make decisions based on sensitive information or discriminate certain social groups (e.g. certain biometric systems for person recognition). With the aim of studying how current multimodal algorithms based on heterogeneous sources of information are affected by sensitive elements and inner biases in the data, we propose …
Face recognition algorithms, when used in the real world, can be very useful, but they can also be dangerous when biased toward certain demographics. So, it is essential to understand how these algorithms are trained and what factors affect their accuracy and fairness to build better ones. In this study, we shed some light on the effect of racial distribution in the training data on the performance of face recognition models. We conduct 16 different experiments with varying racial distributions of faces in the training data. We analyze these trained models using accuracy metrics, clustering me…
From Bias to Balance: Detecting Facial Expression Recognition Biases in Large Multimodal Foundation Models
Lake Wales High School（莱克威尔士高中）；Thomas Jefferson Sr. High School（托马斯·杰斐逊高中）；Bellarmine College Preparatory（贝拉明学院预科学校）；Algoverse AI Research（Algoverse AI研究机构）
Fairness Without Labels: Pseudo-Balancing for Bias Mitigation in Face Gender Classification
National Institute of Informatics, Japan（日本信息机构国家研究所）；INESC-ID, Instituto Superior Técnico, University of Lisbon, Portugal（里斯本大学技术高等学院、INESC-ID，葡萄牙）；University of Amsterdam, The Netherlands（荷兰阿姆斯特丹大学）；LIMSI, CNRS / Université Paris-Saclay, France（法国CNRS/巴黎萨克雷大学LIMSI）
Institute of Smart Cities（智慧城市研究所）；Public University of Navarre（纳瓦拉公立大学）
The increasing amount of applications of Artificial Intelligence (AI) has led researchers to study the social impact of these technologies and evaluate their fairness. Unfortunately, current fairness metrics are hard to apply in multi-class multi-demographic classification problems, such as Facial Expression Recognition (FER). We propose a new set of metrics to approach these problems. Of the three metrics proposed, two focus on the representational and stereotypical bias of the dataset, and the third one on the residual bias of the trained model. These metrics combined can potentially be used…
