从我的视角看：语言如何影响图像理解中的文化偏见
See It from My Perspective: How Language Affects Cultural Bias in Image Understanding
AI总结：
本研究调查了VLMs在图像理解中的西方文化偏见，发现其源于语言模型构建缺乏多样性，并证明在纯文本预训练中充分代表特定语言能有效减少偏见，即使使用英语提示也是如此。
摘要：
视觉语言模型（VLMs）能够用多种语言回答关于图像的查询。然而，除了语言之外，文化也会影响我们看待事物的方式。例如，来自西方文化背景的个体更关注图像中的中心人物，而来自东亚文化背景的个体则更关注场景的背景。在本研究中，我们刻画了VLMs在图像理解中的西方偏见，并调查了语言在这种差异中所起的作用。我们使用具有文化多样性的图像和标注，在主观和客观视觉任务上对VLMs进行了评估。我们发现，在每项任务中，VLMs在西方数据集划分上的表现均优于东亚数据集划分。通过对照实验，我们将图像理解中这种偏见的一个源头追溯到语言模型构建过程中缺乏多样性。虽然使用更接近某种文化的语言进行推理可以减少偏见，但我们表明，当该语言在纯文本预训练期间得到充分代表时，这种效果要显著得多。有趣的是，即使使用英语进行提示，这也能带来偏见的减少。我们的工作强调了在构建公平的VLMs时，更丰富地代表所有语言的重要性。
英文摘要：
Vision-language models (VLMs) can respond to queries about images in many languages. However, beyond language, culture affects how we see things. For example, individuals from Western cultures focus more on the central figure in an image while individuals from East Asian cultures attend more to scene context. In this work, we characterize the Western bias of VLMs in image understanding and investigate the role that language plays in this disparity. We evaluate VLMs across subjective and objective visual tasks with culturally diverse images and annotations. We find that VLMs perform better on the Western split than on the East Asian split of each task. Through controlled experimentation, we trace one source of this bias in image understanding to the lack of diversity in language model construction. While inference in a language nearer to a culture can lead to reductions in bias, we show it is much more effective when that language was well-represented during text-only pre-training. Interestingly, this yields bias reductions even when prompting in English. Our work highlights the importance of richer representation of all languages in building equitable VLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
