arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.00593cs.CVcs.LG
收藏

Pix2Fact: 当视觉不够时——基于网络验证的细粒度VQA基准测试

Pix2Fact: When Vision Is Not Enough -- Benchmarking Fine-Grained VQA with Web Verification on High-Resolution Real-World Scenes

GADE Union (Global AI Data Experts Union)(GADE联盟(全球人工智能数据专家联盟));Shanghai Jiao Tong University(上海交通大学);Nanyang Technological University(南洋理工大学);New York University(纽约大学);Cambridge University(剑桥大学);The University of Hong Kong(香港大学)

Yifan Jiang, Cong Zhang, Bofei Zhang, Qiaofeng Zheng, Yifan Yang, Bingzhang Wang, Yew-Soon Ong

AI总结:

本文提出Pix2Fact基准测试,通过高分辨率真实场景中的网络验证,评估细粒度视觉问答中的专家级视觉感知和知识搜索能力,发现现有模型在复杂任务中存在显著不足。

摘要:

尽管在通用任务上取得了进展,视觉-语言模型(VLMs)仍然在需要精细视觉定位和外部知识的挑战中面临困难,而现有基准测试未能综合评估这些能力。为填补这一空白,我们引入Pix2Fact,一个视觉问答基准测试,旨在评估专家级视觉感知和知识搜索能力。Pix2Fact包含1000张高分辨率(4K+)图像,覆盖八个场景。其问题和答案由来自全球顶尖大学的博士持有标注者精心设计。每个问题都需要详细的视觉定位和外部知识的整合。评估十种最先进的VLMs,包括专有模型如Gemini-3.1-Pro和GPT-5.4,发现Pix2Fact对模型提出了严峻挑战:最先进的模型(Gemini-3.1-Pro)在有视觉地面真实和搜索工具的情况下仅达到51.7%的平均准确率。我们的分析将低准确率归因于三个因素:即使有视觉地面真实,频繁的视觉定位错误,浅层搜索利用,以及VLM无法检索长尾、无结构的局部信息。这种显著的差距暴露了当前模型在帮助人类处理需要超负荷视觉理解的现实场景中的局限性。我们相信Pix2Fact将作为推动下一代语言-视觉代理的关键基准测试,这些代理能够无缝整合细粒度感知与稳健的知识搜索。

英文摘要:

Despite progress on general tasks, vision-language models (VLMs) still struggle with challenges that demand both fine-grained visual grounding and external knowledge, a synergy overlooked by existing benchmarks that evaluate these abilities in isolation. To fill this void, we introduce Pix2Fact, a visual question-answering benchmark designed to assess expert-level visual perception and knowledge search. Pix2Fact comprises 1,000 high-resolution (4K+) images spanning eight scenarios. Its questions and answers are meticulously crafted by PhD-holding annotators from top global universities across diverse disciplines. Each question requires detailed visual grounding and the integration of external knowledge. Evaluating ten state-of-the-art VLMs, including proprietary models such as Gemini-3.1-Pro and GPT-5.4, we find that Pix2Fact poses a formidable challenge: the most advanced model (Gemini-3.1-Pro) achieves only 51.7% average accuracy, even with access to visual ground truth and search tools. Our analysis attributes this low accuracy to three factors, frequent visual grounding errors even with visual ground truth, shallow search harnessing, and VLM's inability to retrieve long-tail, unstructured local information. This striking gap exposes the limitations of current models in assisting humans with real-world scenarios that demand overwhelming visual comprehension. We believe Pix2Fact will serve as a critical benchmark to drive the next generation of language-vision agents that seamlessly integrate fine-grained perception with robust knowledge search.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑