GenderBias-VL：通过反事实探测基准评估视觉语言模型中的性别偏见
GenderBias-\emph{VL}: Benchmarking Gender Bias in Vision Language Models via Counterfactual Probing
AI总结：
本文首次提出GenderBias-VL基准，通过反事实视觉问答在个体公平性标准下评估视觉语言模型的职业性别偏见，覆盖177种职业的34,581个反事实配对，并对15个开源模型及GPT-4o等商业API进行了系统评测。
摘要：
大型视觉语言模型（LVLMs）已在各种应用中被广泛采用；然而，它们表现出显著的性别偏见。现有基准主要在人口群体层面评估性别偏见，忽视了强调相似个体应受平等对待的个体公平性。这一研究空白限制了歧视性行为的检测，因为个体公平性能够对群体公平性可能忽略的偏见进行更细粒度的考察。本文首次引入了GenderBias-VL基准，用于在个体公平性标准下通过反事实视觉问题评估LVLMs中与职业相关的性别偏见。为构建该基准，我们首先利用文本到图像扩散模型生成职业图像及其性别反事实图像。随后，我们通过识别在现实世界统计中具有高语义相似性但性别比例相反的职业刻板印象配对，生成相应的文本职业选项。该方法能够创建大规模视觉问答反事实样本，以暴露LVLMs中的偏见，并且通过修改特定模态中的性别属性，可同时适用于多模态和单模态情境。总体而言，我们的GenderBias-VL基准包含34,581个视觉问答反事实配对，覆盖177种职业。利用我们的基准，我们广泛评估了15个常用的开源LVLM（例如LLaVA）以及最先进的商业API，包括GPT-4o和Gemini-Pro。我们的研究结果揭示了现有LVLMs中普遍存在的性别偏见。我们的基准提供：（1）一个用于职业相关性别偏见评估的综合数据集；（2）一个关于LVLM偏见的最新排行榜；以及（3）对这些模型所呈现偏见的细致理解。数据集和代码可在https://genderbiasvl.github.io/网站上获取。
英文摘要：
Large Vision-Language Models (LVLMs) have been widely adopted in various applications; however, they exhibit significant gender biases. Existing benchmarks primarily evaluate gender bias at the demographic group level, neglecting individual fairness, which emphasizes equal treatment of similar individuals. This research gap limits the detection of discriminatory behaviors, as individual fairness offers a more granular examination of biases that group fairness may overlook. For the first time, this paper introduces the GenderBias-\emph{VL} benchmark to evaluate occupation-related gender bias in LVLMs using counterfactual visual questions under individual fairness criteria. To construct this benchmark, we first utilize text-to-image diffusion models to generate occupation images and their gender counterfactuals. Subsequently, we generate corresponding textual occupation options by identifying stereotyped occupation pairs with high semantic similarity but opposite gender proportions in real-world statistics. This method enables the creation of large-scale visual question counterfactuals to expose biases in LVLMs, applicable in both multimodal and unimodal contexts through modifying gender attributes in specific modalities. Overall, our GenderBias-\emph{VL} benchmark comprises 34,581 visual question counterfactual pairs, covering 177 occupations. Using our benchmark, we extensively evaluate 15 commonly used open-source LVLMs (\eg, LLaVA) and state-of-the-art commercial APIs, including GPT-4o and Gemini-Pro. Our findings reveal widespread gender biases in existing LVLMs. Our benchmark offers: (1) a comprehensive dataset for occupation-related gender bias evaluation; (2) an up-to-date leaderboard on LVLM biases; and (3) a nuanced understanding of the biases presented by these models. \footnote{The dataset and code are available at the \href{https://genderbiasvl.github.io/}{website}.}
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
