CultureCLIP：通过合成图像与情境化描述赋予CLIP文化意识
CultureCLIP: Empowering CLIP with Cultural Awareness through Synthetic Images and Contextualized Captions
AI总结：
本文提出CultureCLIP，通过构建合成文化数据集CulTwin并采用定制对比学习微调CLIP，显著提升了模型对视觉相似但文化含义不同概念的细粒度识别能力，同时保持了原有泛化性能。
摘要：
诸如CLIP等预训练视觉语言模型（VLMs）在通用多模态理解方面表现出色，但往往难以捕捉细微的、依赖情境的视觉线索。这使得模型难以区分外观相似但可能具有不同文化含义的概念。这些缺陷主要源于高质量文化数据与情境信息的数量有限，且缺乏能突出细微差异的负样本。为缓解此问题，我们设计了一个数据筛选流程，利用开源VLMs和文本到图像模型构建了合成文化数据集CulTwin。该数据集由成对的概念-描述-图像三元组组成，其中各概念在视觉上相似但在文化上存在差异。随后，我们在CulTwin上微调CLIP以开发CultureCLIP，通过定制的对比学习将文化概念与情境增强的描述及合成图像进行对齐。在特定文化基准上的实验表明，CultureCLIP优于基础CLIP，在某些任务的细粒度概念识别上实现了高达5.49%的显著提升，同时保留了CLIP原有的泛化能力，验证了我们的数据合成与VLM骨干训练范式在捕捉细微文化差异方面的有效性。
英文摘要：
Pretrained vision-language models (VLMs) such as CLIP excel in general multimodal comprehension but often struggle to capture nuanced, context-dependent visual cues. This makes it difficult to distinguish between similar-looking concepts with potentially different cultural meanings. Such deficiencies are mainly due to a limited amount of high-quality cultural data, contextual information, and the lack of negative examples that highlight subtle differences. To mitigate this, we design a data curation pipeline leveraging open-sourced VLMs and text-to-image models to construct CulTwin, a synthetic cultural dataset. This dataset consists of paired concept-caption-image triplets, where concepts visually resemble each other but are culturally different. Then, we fine-tune CLIP on CulTwin to develop CultureCLIP, which aligns cultural concepts with contextually enhanced captions and synthetic images through tailored contrastive learning. Experiments on culture-specific benchmarks show that CultureCLIP outperforms the base CLIP, achieving up to a notable 5.49% improvement in fine-grained concept recognition on certain tasks while preserving CLIP's original generalization ability, validating the effectiveness of our data synthesis and VLM backbone training paradigm in capturing subtle cultural distinctions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
