VisualNeedle: 信息密集场景中的主动视觉搜索基准
VisualNeedle: Benchmarking Active Visual Search in Information-Dense Scenes
AI总结：
针对多模态大语言模型在细粒度感知基准中依赖捷径而非真实视觉证据的问题，提出VisualNeedle基准，通过反事实裁剪-黑化设置评估模型在信息密集场景中的主动视觉搜索能力，实验表明最佳模型准确率仅56.01%，落后人类63.00%。
摘要：
前沿多模态大语言模型（MLLMs）被报道在细粒度感知基准上达到超过90%的准确率。然而，这样的分数并不一定意味着对视觉证据的忠实使用。先前的研究已经识别出三种抬高基准性能的捷径。首先，问题中的语言先验和词汇线索使模型能够在未见图像的情况下推断出看似合理的答案。其次，来自视觉编码器的粗略全局语义可以绕过细粒度的局部细节。第三，在一些“用图像思考”的基准中，破坏视觉工具返回的中间图像几乎不影响最终答案。这些发现表明，仅靠更高的输入分辨率或更大的问题池并不能引发真正的主动视觉搜索。为了解决这个问题，我们引入了VisualNeedle，这是一个具有挑战性、信息密集且细粒度的基准，用于关键证据在空间上局限于微小区域且无法一眼看出的场景。我们进一步提出了一种反事实裁剪-黑化设置，将工具返回的裁剪区域替换为相同大小的黑色图像，以测试工具启用的性能是否真正依赖于中间视觉证据。我们在三种设置下评估了9个著名的MLLMs：无工具、标准工具启用和裁剪-黑化。无工具准确率保持在20%以下，最佳工具启用模型仅达到56.01%，仍落后于63.00%的人类多数投票准确率。这些结果揭示了细粒度视觉搜索中持续存在的局限性，而裁剪-黑化消融实验证实，VisualNeedle上的成功依赖于真正的中间视觉证据。
英文摘要：
Frontier multimodal large language models (MLLMs) have been reported to achieve over 90\% accuracy on fine-grained perception benchmarks. However, such scores do not necessarily imply faithful use of visual evidence. Prior studies have identified three shortcuts that inflate benchmark performance. First, linguistic priors and lexical cues in questions often enable models to infer plausible answers without seeing the image. Second, coarse global semantics from the visual encoder can bypass fine-grained local details. Third, in some ``think-with-images'' benchmarks, corrupting the intermediate images returned by visual tools barely affects the final answer. These findings suggest that higher input resolution or larger question pools alone do not elicit genuine active visual search. To address this, we introduce VisualNeedle, a challenging, information-dense, and fine-grained benchmark for scenes where critical evidence is spatially constrained to minute regions and not discernible at a glance. We further propose a counterfactual crop-black setting, which replaces crops returned by tools with black images of the same size, to test whether tool-enabled performance truly relies on intermediate visual evidence.We evaluate 9 prominent MLLMs across four settings: text-only, without tools, with tools, and crop-black. Text-only accuracy stays below 10\%, while accuracy without tools remains below 20\%. The best tool-enabled model reaches only 56.00\%, still trailing the 63.00\% human majority-vote accuracy. These results reveal persistent limitations in fine-grained visual search, while the crop-black ablation confirms that success on VisualNeedle hinges on genuine intermediate visual evidence.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
