CompareBench: 一个用于评估视觉比较推理能力的视觉-语言模型基准
CompareBench: A Benchmark for Visual Comparison Reasoning in Vision-Language Models
AI总结：
CompareBench是一个用于评估视觉-语言模型中视觉比较推理能力的基准，揭示了当前模型在时间排序、空间关系和基本计数上的局限性。
摘要：
我们介绍了CompareBench，这是一个用于评估视觉-语言模型（VLMs）中视觉比较推理能力的基准，这是一种基础但研究不足的技能。CompareBench包含1000个问题-答案对，涵盖四个任务：数量（600个）、时间（100个）、几何（200个）和空间（100个）。它来源于我们构建的两个辅助数据集：TallyBench（2000张计数图像和问题答案）和HistCaps（515张历史图像和双语描述）。我们评估了闭源API（OpenAI、Gemini、Claude）和开源模型（Qwen2.5-VL和Qwen3-VL系列）。结果表明，存在明显的扩展趋势，但也揭示了关键限制：即使最强的模型在时间排序和空间关系上也经常失败，并且在基本计数和几何比较上常犯错误，这些对于人类来说是微不足道的。这些发现表明，视觉比较仍然是当前VLMs的系统性盲点。通过提供受控、多样和诊断性的评估，CompareBench为推进更可靠的多模态推理奠定了基础。
英文摘要：
Visual comparison reasoning is a fundamental capability of vision-language models (VLMs), covering judgments of object quantity, geometric dimensions, spatial relations, and temporal order. Yet existing benchmarks rarely isolate comparison as a reasoning axis, leaving it unclear whether models can reliably perform comparative visual judgments. We introduce a benchmark suite organized around three top-level resources: TallyBench, a 2,000-image object counting benchmark; OmniCaps, a 716-image caption and tag resource; and CompareBench, a 1,200-QA visual comparison benchmark. CompareBench contains four sub-benchmarks spanning quantity, geometric, spatial, and temporal comparison, with the temporal component unifying historical scenes, landmarks, and public figures. Evaluating nine closed-source model routes from Anthropic, Google, and OpenAI on TallyBench and CompareBench reveals strong overall performance but persistent failures in counting, spatial reasoning, geometric comparison, and temporal ordering. These results show that visual comparison remains a systematic weakness of current VLMs and establish CompareBench as a focused benchmark for multimodal reasoning evaluation. All data, code, and prompts will be released at https://github.com/caijie0620/CompareBench.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
