发表机构
Meituan（美团）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
研究针对多模态大语言模型定量精度瓶颈及现有计数基准局限，引入HoloCount基准，从语义、分析计数和鲁棒性测试三方面评估模型，经对20多个模型详尽评估，揭示其性能差距，为多模态系统发展提供路线图。
AI 中文摘要
视觉计数是多模态智能的基本支柱，需要细粒度基础和空间推理的无缝集成。虽然多模态大语言模型（MLLMs）在定性场景理解方面取得了显著成功，但其定量精度仍是重大瓶颈，常存在持续的数值幻觉。现有计数基准主要关注简化情境下的基本感知，无法捕捉逻辑约束或对抗条件下出现的复杂失败模式。为解决这些局限，我们引入HoloCount，这是一个围绕三级分层分类构建的整体且诊断丰富的基准。它从语义计数、分析计数和鲁棒性测试三方面评估MLLMs。通过对20多个先进MLLMs的详尽评估，发现即使顶级模型在任务从感知过渡到复杂分析推理和不利场景时也会显著退化。我们的发现提供了当前MLLM计数能力的系统图景，并为开发更有基础和可靠的多模态系统提供了路线图。数据集可通过此https URL获取。
英文摘要
Visual counting is a fundamental pillar of multimodal intelligence, requiring a seamless integration of fine-grained grounding and spatial reasoning. While Multimodal Large Language Models (MLLMs) have achieved remarkable success in qualitative scene understanding, their quantitative precision remains a significant bottleneck, often characterized by persistent numerical hallucinations. Existing counting benchmarks primarily focus on basic perception in simplified contexts, failing to capture the complex failure modes that emerge under logical constraints or adversarial conditions. To address these limitations, we introduce HoloCount, a holistic and diagnostically rich benchmark structured around a three-level hierarchical taxonomy. HoloCount evaluates MLLMs across: (1) Semantic Counting, focusing on atomic and property-based enumeration; (2) Analytical Counting, assessing logical composition through spatial and set-based reasoning; and (3) Robustness Testing, probing model integrity against adverse scenarios and grounded counter-priors, such as high-density scenes and linguistic biases. Through an exhaustive evaluation of over 20 state-of-the-art MLLMs, we reveal a critical performance gap: even top-tier models degrade significantly as tasks transition from perception to complex analytical reasoning and adverse scenarios. Our findings provide a systematic landscape of current MLLM counting capabilities and offer a roadmap for developing more grounded and reliable multimodal systems. The dataset is available at https://mm-mvr.github.io/HoloCount/.
CommentsTechnical report
