他们都是医生：合成多样化反事实以缓解关联偏差
They're All Doctors: Synthesizing Diverse Counterfactuals to Mitigate Associative Bias
AI总结：
本文提出用分割与修复模型生成多样化反事实图像并微调 CLIP，使人物外观与职业语境解耦，在保持下游性能的同时显著改善检索公平性。
摘要：
诸如 CLIP 之类的视觉语言模型（VLM）是强大的模型；然而，它们可能表现出不良偏差，这使其在直接部署于文生图、文生视频检索、反向搜索或分类任务等应用时安全性较低。在本工作中，我们提出了一个新颖框架，用于生成合成反事实图像，以构建一个多样化且平衡的数据集，可用于微调 CLIP。给定一组来自文生图模型的多样化合成基础图像，我们利用现成的分割和修复模型，将具有多样视觉外观的人物置入场景中。我们表明，在此类数据集上训练的 CLIP 学会了将人物外观与图像语境解耦，也就是说，使人成为医生的因素并不与人物的视觉外观（如肤色或体型）相关，而是与语境相关，例如背景、他们所穿的服装或他们手持的物体。我们证明，我们微调后的 CLIP 模型 $CF_α$ 在图像检索任务上将 MaxSkew、MinSkew 和 NDKL 等关键公平性指标改善了 40–66%，同时在下游任务中仍达到相近的性能水平。我们表明，通过设计，我们的模型与原始 CLIP 模型保持最大程度的兼容性，并且能够以便插即用的方式轻松控制，以支持不同的准确性与公平性权衡。
英文摘要：
Vision Language Models (VLMs) such as CLIP are powerful models; however they can exhibit unwanted biases, making them less safe when deployed directly in applications such as text-to-image, text-to-video retrievals, reverse search, or classification tasks. In this work, we propose a novel framework to generate synthetic counterfactual images to create a diverse and balanced dataset that can be used to fine-tune CLIP. Given a set of diverse synthetic base images from text-to-image models, we leverage off-the-shelf segmentation and inpainting models to place humans with diverse visual appearances in context. We show that CLIP trained on such datasets learns to disentangle the human appearance from the context of an image, i.e., what makes a doctor is not correlated to the person's visual appearance, like skin color or body type, but to the context, such as background, the attire they are wearing, or the objects they are holding. We demonstrate that our fine-tuned CLIP model, $CF_α$, improves key fairness metrics such as MaxSkew, MinSkew, and NDKL by 40-66\% for image retrieval tasks, while still achieving similar levels of performance in downstream tasks. We show that, by design, our model retains maximal compatibility with the original CLIP models, and can be easily controlled to support different accuracy versus fairness trade-offs in a plug-n-play fashion.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
