arXiv 2505.144622026-02-14cs.CVcs.CL

RAVENEA：多模态检索增强视觉文化理解的基准

RAVENEA: A Benchmark for Multimodal Retrieval-Augmented Visual Culture Understanding

University of Copenhagen（哥本哈根大学）；ETH Zürich（苏黎世联邦理工学院）；University of Amsterdam（阿姆斯特丹大学）；University of Cambridge（剑桥大学）；Massachusetts Institute of Technology（麻省理工学院）；Singapore University of Technology and Design（新加坡科技设计大学）；Singapore Management University（新加坡管理大学）

Jiaang Li, Yifei Yuan, Wenyan Li, Mohammad Aliannejadi, Daniel Hershcovich, Anders Søgaard, Ivan Vulić, Wenxuan Zhang, Paul Pu Liang, Yang Deng, Serge Belongie

备注：: ICLR 2026; Project page: https://jiaangli.github.io/ravenea/

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

RAVENEA通过多模态检索增强方法提升视觉文化理解，验证了文化注释对多模态检索和下游任务的增强效果，并揭示了不同国家间性能差异。

摘要：

随着视觉语言模型（VLMs）日益融入日常生活，准确理解视觉文化变得至关重要。然而，这些模型在有效解释文化细微差别方面常常表现不足。先前的工作表明，检索增强生成（RAG）在纯文本环境下能有效提升文化理解，但在多模态场景中的应用仍显不足。为填补这一空白，我们引入RAVENEA（检索增强视觉文化理解），一个新的基准，旨在通过检索推进视觉文化理解，聚焦于两个任务：文化导向的视觉问答（cVQA）和文化引导的图像描述生成（cIC）。RAVENEA通过整合超过11,396个由人类标注员筛选和排序的唯一维基百科文档扩展了现有数据集。通过在七个多模态检索器和十五个VLMs上的广泛评估，RAVENEA揭示了一些未被发现的发现：（i）通常，文化定位注释可以增强多模态检索及其相应的下游任务。（ii）当VLMs与文化意识检索相结合时，通常会优于非增强版本（在cVQA上平均提升6%，在cIC上提升11%）。（iii）文化意识检索增强的表现因国家而异。这些发现突显了当前多模态检索器和VLMs的局限性，强调了在RAG系统中增强视觉文化理解的必要性。我们相信RAVENEA为推进检索增强视觉文化理解的研究提供了有价值的资源。

英文摘要：

As vision-language models (VLMs) become increasingly integrated into daily life, the need for accurate visual culture understanding is becoming critical. Yet, these models frequently fall short in interpreting cultural nuances effectively. Prior work has demonstrated the effectiveness of retrieval-augmented generation (RAG) in enhancing cultural understanding in text-only settings, while its application in multimodal scenarios remains underexplored. To bridge this gap, we introduce RAVENEA (Retrieval-Augmented Visual culturE uNdErstAnding), a new benchmark designed to advance visual culture understanding through retrieval, focusing on two tasks: culture-focused visual question answering (cVQA) and culture-informed image captioning (cIC). RAVENEA extends existing datasets by integrating over 11,396 unique Wikipedia documents curated and ranked by human annotators. Through the extensive evaluation on seven multimodal retrievers and fifteen VLMs, RAVENEA reveals some undiscovered findings: (i) In general, cultural grounding annotations can enhance multimodal retrieval and corresponding downstream tasks. (ii) VLMs, when augmented with culture-aware retrieval, generally outperform their non-augmented counterparts (by averaging +6% on cVQA and +11% on cIC). (iii) Performance of culture-aware retrieval augmented varies widely across countries. These findings highlight the limitations of current multimodal retrievers and VLMs, underscoring the need to enhance visual culture understanding within RAG systems. We believe RAVENEA offers a valuable resource for advancing research on retrieval-augmented visual culture understanding.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。