VidNum-1.4K：一个全面的视频基于数值推理基准
VidNum: Diagnosing VLM Failure Modes in Video-Grounded Numerical Reasoning
AI总结：
本文提出VidNum-1.4K基准，通过1379个严格人工标注的视频问题对，评估视频基于的数值推理能力，揭示当前VLMs在多步骤逻辑推理上的不足。
摘要：
视频基于的数值推理为测试Vision-Language Models (VLMs)是否真正理解现实动态提供了重要平台，因为准确的数值推导需要对时间事件、物体永恒性和组合逻辑的深刻理解，而不仅仅是表面模式匹配。然而，现有基准往往局限于狭窄领域，如重复性运动，或简单计数仅视为表面回归任务，未能评估现实世界多媒体内容中的多步骤数值逻辑。我们引入VidNum-1.4K，一个包含1,379个严格人工标注的视频问题对的综合VideoQA基准，旨在评估在高度多样化的环境中真正的数值推理能力，涵盖物体、动作和事件的量化。VidNum-1.4K独特地分为三级层次，从直接视觉感知演变为基于视频的组合数值推理，要求模型进行算术运算、比较和基于时间证据的逻辑推断。我们的评估显示，尽管Gemini-3.1-pro仅达到60%的准确率门槛，代表性的开源模型在25%至45%范围内表现不佳。这些发现表明，当前VLMs仍缺乏稳定的'内部世界模型'，将VidNum-1.4K定位为下一代数值视频智能的严格诊断测试平台。
英文摘要：
Video-grounded numerical reasoning requires Vision-Language Models (VLMs) to identify, track, and combine quantitative evidence across frames, actions, and scene changes. Existing benchmarks provide fragmented coverage: general VideoQA includes counting among broader tasks, while dedicated benchmarks focus on repetition counting, ultra-long-video enumeration, or instructional mathematics. We introduce VidNum, a manually curated and independently verified benchmark containing 1,167 multiple-choice questions. Its three task groups distinguish Direct and Distinct Enumeration, Conditioned and Structured Enumeration, and Compositional Quantitative Reasoning. Question-level annotations further identify the evidence target, counting structure, and required reasoning operation. The best evaluated VLM reaches 59.8% accuracy, compared with 98.2% for human annotators, and no evaluated open-weight model exceeds 45%. Stratified analyses reveal that failures are not uniformly distributed: structured target construction and action-grounded compositional reasoning form recurring bottlenecks across models. Zero-shot chain-of-thought prompting is not a reliable remedy: it recovers some errors but breaks previously correct answers, with effects that vary across models and task structures. VidNum therefore supports diagnostic analysis beyond a single aggregate score.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
