面向图像-文本检索评估的基于视觉-语言模型的自动相关性判断研究
Toward Automatic Relevance Judgment using Vision--Language Models for Image--Text Retrieval Evaluation
AI总结：
本文评估了CLIP、LLaVA和GPT-4V等视觉-语言模型在零样本图像-文本检索评估中的相关性判断能力，发现LLM驱动的VLMs在多项指标上优于CLIPScore，展现出增强相关性判断的潜力。
摘要：
视觉-语言模型（VLMs）已在多种应用中展现出成功，但它们在辅助相关性判断方面的潜力仍不确定。本文评估了VLMs的相关性估计能力，包括CLIP、LLaVA和GPT-4V，在一个面向多媒体内容创作的大规模即席检索任务中，以零样本方式进行。初步实验揭示了以下结果：（1）LLaVA和GPT-4V，涵盖开源和闭源的视觉指令微调大语言模型（LLMs），在与人类相关性判断比较时达到了显著的Kendall's τ~0.4，超过了CLIPScore指标。（2）虽然CLIPScore被强烈偏好，但LLMs对基于CLIP的检索系统的偏见较小。（3）GPT-4V的分数分布比其他模型更接近人类判断，实现了约0.08的Cohen's κ值，优于CLIPScore约-0.096的值。这些发现强调了LLM驱动的VLMs在增强相关性判断方面的潜力。
英文摘要：
Vision--Language Models (VLMs) have demonstrated success across diverse applications, yet their potential to assist in relevance judgments remains uncertain. This paper assesses the relevance estimation capabilities of VLMs, including CLIP, LLaVA, and GPT-4V, within a large-scale \textit{ad hoc} retrieval task tailored for multimedia content creation in a zero-shot fashion. Preliminary experiments reveal the following: (1) Both LLaVA and GPT-4V, encompassing open-source and closed-source visual-instruction-tuned Large Language Models (LLMs), achieve notable Kendall's $τ\sim 0.4$ when compared to human relevance judgments, surpassing the CLIPScore metric. (2) While CLIPScore is strongly preferred, LLMs are less biased towards CLIP-based retrieval systems. (3) GPT-4V's score distribution aligns more closely with human judgments than other models, achieving a Cohen's $κ$ value of around 0.08, which outperforms CLIPScore at approximately -0.096. These findings underscore the potential of LLM-powered VLMs in enhancing relevance judgments.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
