OrdinalBench: 一种用于诊断视觉-语言模型在序数理解通用性限制的基准数据集
OrdinalBench: A Benchmark Dataset for Diagnosing Generalization Limits in Ordinal Number Understanding of Vision-Language Models
AI总结:
OrdinalBench通过评估视觉-语言模型在序数理解上的表现,揭示其在大序数和复杂路径条件下的泛化能力不足问题。
摘要:
视觉-语言模型(VLMs)在多模态基准上已取得显著进展,但仍在序数理解方面存在明显差距,即跟踪相对位置并推广到大索引的能力。我们提出了OrdinalBench,一个诊断基准,将序数理解标准化为VLMs的评估任务。核心任务是第N个对象识别,由起始参考和遍历规则定义。任务难度沿三个轴控制:(i)序数大小,从小数字到极端情况,最大达到300;(ii)排列复杂性,从单循环到迷宫式路径;(iii)对象数量。该基准提供了39,000个问题-答案对,每个都标注了真实推理轨迹,并在不同难度水平上平衡,以实现受控的大规模测试。除了仅答案评估外,我们的框架要求模型生成结构化的逐步计数轨迹,并提供一个开放的评估工具包,该工具包测量最终准确性和步骤级路径一致性。对GPT-5、Gemini 2.5 Flash Lite、Qwen2.5-VL、InternVL3.5和Molmo的零样本评估显示,在大序数和复杂路径条件下存在明显退化,突显了在标准多模态任务上得分强劲但泛化能力弱的缺点。通过将序数理解作为核心目标,OrdinalBench为开发具有更强顺序推理能力的VLMs提供了一种可重复的基准和诊断框架。所有数据和代码均在https://ordinalbench.github.io/上提供。
英文摘要:
Vision-Language Models (VLMs) have advanced across multimodal benchmarks but still show clear gaps in ordinal number understanding, i.e., the ability to track relative positions and generalize to large indices. We present OrdinalBench, a diagnostic benchmark that standardizes ordinal number understanding as an evaluation task for VLMs. The core task is N-th object identification, defined by a starting reference and traversal rule. Task difficulty is controlled along three axes: (i) ordinal magnitude, from small numbers to extreme cases up to 300; (ii) arrangement complexity, from single loops to maze-like paths; and (iii) object count. The benchmark provides 39,000 question-answer pairs, each annotated with a ground-truth reasoning trajectory and balanced across difficulty levels for controlled large-scale testing. Beyond answer-only evaluation, our framework requires models to generate structured stepwise traces of the counting process and provides an open evaluation toolkit that measures both final accuracy and step-level path consistency. Zero-shot evaluations of GPT-5, Gemini 2.5 Flash Lite, Qwen2.5-VL, InternVL3.5, and Molmo reveal sharp degradation under large-ordinal and complex-path conditions, highlighting weak generalization despite strong scores on standard multimodal tasks. By framing ordinal number understanding as a core target, OrdinalBench provides a reproducible benchmark and diagnostic framework for developing VLMs with stronger sequential reasoning. All data and code are available at https://ordinalbench.github.io/
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。