arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2409.11007cs.CLcs.CV
收藏

CAST:面向视觉语言模型的跨模态对齐相似性测试

CAST: Cross-modal Alignment Similarity Test for Vision Language Models

University of Edinburgh(爱丁堡大学);University of Trento(特伦托大学)

Gautier Dagan, Olga Loginova, Anil Batra

AI总结:

针对视觉问答无法全面评估视觉语言模型模态偏差与幻觉的问题,提出跨模态对齐相似性测试CAST,通过多模态场景相似性识别任务检验模型输出的内部自洽性,指出能力合格的视觉语言模型必须具备自洽性。

摘要:

视觉语言模型(Vision Language Models, VLMs)通常通过视觉问答(Visual Question Answering, VQA)任务进行评估,这类任务衡量模型对场景的理解能力。VQA 表现优异常被视为模型能在更广泛的需同时输入视觉与语言的任务上表现良好的证据。然而,具备场景感知能力的 VQA 无法完全捕捉输入偏差,也无法评估由模态间不对齐引发的幻觉问题。为解决这一问题,我们提出跨模态对齐相似性测试(Cross-modal Alignment Similarity Test, CAST),用于探究 VLMs 在跨模态下的自一致性。该测试要求模型通过纯文本、纯图像或两者结合的方式识别两个场景间的相似性,随后评估其生成的相似性内容的真实性。由于不存在可对比的基准真值,该评估不关注客观准确率,而是聚焦于 VLMs 的输出是否具备内部一致性。我们认为,尽管并非所有自洽的模型都具备足够能力或准确率,但所有具备能力的 VLM 都必须是自洽的。

英文摘要:

Vision Language Models (VLMs) are typically evaluated with Visual Question Answering (VQA) tasks which assess a model's understanding of scenes. Good VQA performance is taken as evidence that the model will perform well on a broader range of tasks that require both visual and language inputs. However, scene-aware VQA does not fully capture input biases or assess hallucinations caused by a misalignment between modalities. To address this, we propose a Cross-modal Alignment Similarity Test (CAST) to probe VLMs for self-consistency across modalities. This test involves asking the models to identify similarities between two scenes through text-only, image-only, or both and then assess the truthfulness of the similarities they generate. Since there is no ground-truth to compare against, this evaluation does not focus on objective accuracy but rather on whether VLMs are internally consistent in their outputs. We argue that while not all self-consistent models are capable or accurate, all capable VLMs must be self-consistent.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑