arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.06177cs.CVcs.AIcs.CLcs.CY
收藏

文化与包容性考量下的视觉-语言模型

Vision-Language Models under Cultural and Inclusive Considerations

University of Copenhagen(哥本哈根大学)

Antonia Karamolegkou, Phillip Rust, Yong Cao, Ruixiang Cui, Anders Søgaard, Daniel Hershcovich

AI总结:

针对视障用户图像描述的文化适配问题,本文通过偏好调查并筛选 VizWiz 构建以文化为中心的 VLM 评估基准,评测多种模型并揭示幻觉与指标偏差。

摘要:

大型视觉-语言模型(VLM)能够通过描述视障人士日常生活中的图像来为其提供帮助。当前的评估数据集可能无法反映多元文化用户背景或该使用场景的情境语境。为解决这一问题,我们开展了一项调查以确定图像描述偏好,并通过筛选 VizWiz(一个包含盲人拍摄图像的现有数据集)提出了一个以文化为中心的评估基准。随后,我们评估了多个 VLM,考察其在文化多元环境中作为视觉助手的可靠性。尽管最先进模型的结果令人鼓舞,但我们也发现了幻觉以及自动评估指标与人类判断不一致等挑战。我们公开了调查、数据、代码和模型输出。

英文摘要:

Large vision-language models (VLMs) can assist visually impaired people by describing images from their daily lives. Current evaluation datasets may not reflect diverse cultural user backgrounds or the situational context of this use case. To address this problem, we create a survey to determine caption preferences and propose a culture-centric evaluation benchmark by filtering VizWiz, an existing dataset with images taken by people who are blind. We then evaluate several VLMs, investigating their reliability as visual assistants in a culturally diverse setting. While our results for state-of-the-art models are promising, we identify challenges such as hallucination and misalignment of automatic evaluation metrics with human judgment. We make our survey, data, code, and model outputs publicly available.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑