识别和解决基于知识的VQA基准测试中的陷阱：审计、修复与增强
Identifying and Resolving Pitfalls of Knowledge-Based VQA Benchmarks: Auditing, Repairing, and Augmenting
AI总结：
本文审计现有KB-VQA基准测试，发现答案缺失/矛盾、问题不明确和视觉场景简单等系统性问题，导致准确率指标误导模型排名；提出审计-修复协议和多重实体增强协议，重新评估后性能趋势显著变化。
摘要：
基于知识的视觉问答（KB-VQA）旨在评估视觉语言模型（VLM）能否检索、定位并推理视觉证据之外的外部结构化知识。实践中，答案准确率被广泛用作主要评估指标，隐含地将正确性视为知识驱动推理的代理。然而，对于现有的KB-VQA基准测试，这一代理依赖于一些关键假设，这些假设常被忽视且因基准测试问题而变得不可靠：标注答案必须能从相关知识库推导得出，问题必须表述良好且具有充分约束，视觉设置必须有意义地要求基于上下文的消歧。在这项工作中，我们表明这些假设在现有KB-VQA基准测试中系统性地被违反。我们的审计揭示了大量实例存在缺失或矛盾的答案以及表述不清的问题，使得准确率成为误导性指标。此外，我们发现现有数据集依赖视觉上琐碎的单实体场景，绕过了复杂的视觉到知识映射需求。我们证明，即使使用受控架构，这些缺陷也会导致模型排名扭曲和推理能力的高估。为解决此问题，我们引入了（1）一个原则性的审计-修复协议，恢复答案可推导性和问题清晰度，以及（2）一个受控的多实体增强协议，引入视觉歧义以挑战初始检索和基于上下文的推理。在修正和增强设置下的重新评估产生了显著不同的性能趋势。我们的发现呼吁重新思考评估协议，并设计更多交互感知的KB-VQA基准测试，优先考虑可验证的推理而非简单匹配。
英文摘要：
Knowledge-Based Visual Question Answering (KB-VQA) aims to evaluate whether Visual Language Models (VLMs) can retrieve, ground, and reason over external structured knowledge beyond visual evidence. In practice, answer accuracy is widely adopted as the primary evaluation metric, implicitly treating correctness as a proxy for knowledge-grounded reasoning. However, for existing KB-VQA benchmarks, this proxy relies on critical assumptions that are often overlooked and rendered unreliable by benchmark issues: annotated answer must be derivable from the associated knowledge base, question must be well-posed with sufficient constraints, and visual setting must meaningfully require grounded disambiguation. In this work, we show that these assumptions are systematically violated in existing KB-VQA benchmarks. Our audit reveals substantial instances with missing or contradicted answers and underspecified questions that render accuracy a misleading metric. Furthermore, we find that existing datasets rely on visually trivial, single-entity scenes that bypass the need for sophisticated visual-to-knowledge mapping. We demonstrate that even with controlled architectures, these flaws lead to distorted model rankings and overestimations of reasoning capabilities. To address this, we introduce (1) a principled audit-and-repair protocol that restores answer derivability and question clarity, and (2) a controlled multi-entity augmentation protocol that introduces visual ambiguity to challenge initial retrieval and grounded reasoning. Re-evaluation under corrected and augmented settings yields markedly different performance trends. Our findings call for rethinking evaluation protocols and designing more interaction-aware KB-VQA benchmarks that prioritize verifiable reasoning over simple matching.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
