FairCLIP：基于属性原型学习和表示中和的社会偏见消除
FairCLIP: Social Bias Elimination based on Attribute Prototype Learning and Representation Neutralization
AI总结：
本文提出 FairCLIP 方法，通过属性原型学习（APL）提取去偏概念，并利用重表示矩阵（RRM）实现表示中和（RN），有效消除了基于 CLIP 的图像检索中的社会偏见，同时保持了检索性能，并可作为 CLIP 其他下游任务的通用去偏方案。
摘要：
近年来，像 CLIP 这样的视觉-语言预训练（VLP）模型越来越受欢迎。然而，许多研究发现，CLIP 中隐藏的社会偏见很容易在下游任务中显现，尤其是在图像检索中，这可能对社会产生有害影响。在本研究中，我们提出 FairCLIP 来消除基于 CLIP 的图像检索中的社会偏见，同时不损害检索性能，从而实现去偏效果与检索性能之间的兼容。FairCLIP 分为两个步骤：属性原型学习（APL）和表示中和（RN）。在第一步中，我们提取 CLIP 中去偏所需的概念。我们使用带有可学习词向量前缀的查询作为提取结构。在第二步中，我们首先将属性分为目标属性和偏见属性。通过分析，我们发现这两种属性都对偏见产生影响。因此，我们尝试使用重表示矩阵（RRM）来消除偏见，以实现表示的中和。我们将去偏效果和检索性能与其他方法进行比较，实验表明 FairCLIP 能够实现最佳的兼容性。尽管 FairCLIP 用于消除图像检索中的偏见，但它实现了对所有 CLIP 下游任务通用的表示中和。这意味着 FairCLIP 可以作为一种通用的去偏方法，应用于与 CLIP 相关的其他公平性问题。
英文摘要：
The Vision-Language Pre-training (VLP) models like CLIP have gained popularity in recent years. However, many works found that the social biases hidden in CLIP easily manifest in downstream tasks, especially in image retrieval, which can have harmful effects on human society. In this work, we propose FairCLIP to eliminate the social bias in CLIP-based image retrieval without damaging the retrieval performance achieving the compatibility between the debiasing effect and the retrieval performance. FairCLIP is divided into two steps: Attribute Prototype Learning (APL) and Representation Neutralization (RN). In the first step, we extract the concepts needed for debiasing in CLIP. We use the query with learnable word vector prefixes as the extraction structure. In the second step, we first divide the attributes into target and bias attributes. By analysis, we find that both attributes have an impact on the bias. Therefore, we try to eliminate the bias by using Re-Representation Matrix (RRM) to achieve the neutralization of the representation. We compare the debiasing effect and retrieval performance with other methods, and experiments demonstrate that FairCLIP can achieve the best compatibility. Although FairCLIP is used to eliminate bias in image retrieval, it achieves the neutralization of the representation which is common to all CLIP downstream tasks. This means that FairCLIP can be applied as a general debiasing method for other fairness issues related to CLIP.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
