CapArena：大语言模型时代详细图像描述的基准测试与分析
CapArena: Benchmarking and Analyzing Detailed Image Captioning in the LLM Era
AI总结：
本文构建含6000余场成对对决的CapArena基准，评估VLM详细图像描述与自动指标，并推出低成本、高相关性的CapArena-Auto。
摘要：
图像描述一直是视觉-语言研究中的一项长期挑战。随着LLM的兴起，现代视觉-语言模型（VLM）能够生成详细且全面的图像描述。然而，如何对此类描述的质量进行基准测试仍未解决。本文探讨两个关键问题：（1）当前VLM在图像描述上的实际表现如何，尤其是与人类相比？我们构建了CapArena平台，其中包含超过6000场成对描述对决以及高质量的人类偏好投票。我们的竞技场式评估标志着一个里程碑，结果显示GPT-4o等领先模型达到甚至超越人类表现，而大多数开源模型仍落后。（2）自动化指标能否可靠评估详细描述质量？利用来自CapArena的人工标注，我们评估了传统和近期的描述指标，以及VLM-as-a-Judge。分析表明，尽管METEOR等部分指标在描述层面与人类具有尚可的一致性，但其系统性偏差导致模型排名不一致。相比之下，VLM-as-a-Judge在描述和模型两个层面均表现出稳健的判别能力。基于这些洞见，我们发布CapArena-Auto，这是一个准确、高效的详细描述自动化基准，与人类排名的相关性达到94.3%，而每次测试仅需4美元。数据和资源将在https://caparena.github.io开源。
英文摘要：
Image captioning has been a longstanding challenge in vision-language research. With the rise of LLMs, modern Vision-Language Models (VLMs) generate detailed and comprehensive image descriptions. However, benchmarking the quality of such captions remains unresolved. This paper addresses two key questions: (1) How well do current VLMs actually perform on image captioning, particularly compared to humans? We built CapArena, a platform with over 6000 pairwise caption battles and high-quality human preference votes. Our arena-style evaluation marks a milestone, showing that leading models like GPT-4o achieve or even surpass human performance, while most open-source models lag behind. (2) Can automated metrics reliably assess detailed caption quality? Using human annotations from CapArena, we evaluate traditional and recent captioning metrics, as well as VLM-as-a-Judge. Our analysis reveals that while some metrics (e.g., METEOR) show decent caption-level agreement with humans, their systematic biases lead to inconsistencies in model ranking. In contrast, VLM-as-a-Judge demonstrates robust discernment at both the caption and model levels. Building on these insights, we release CapArena-Auto, an accurate and efficient automated benchmark for detailed captioning, achieving 94.3% correlation with human rankings at just $4 per test. Data and resources will be open-sourced at https://caparena.github.io.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
