arXivDaily arXiv每日学术速递 周一至周五更新
​通知:联邦节假日调休:哥伦布日 / 原住民日(Columbus Day / Indigenous Peoples' Day),今日(13日)无更新。
arXiv 2610.13790cs.CVcs.CL

OccludeBench:视觉语言模型能否发现缺失的内容?

OccludeBench: Can VLMs Find What's Missing?

Princeton University(普林斯顿大学)

Jasin Cekinmez, Addison J. Wu, Ryo Mitsuhashi, Linrong Cai, Xingyu Fu, Zhuang Liu

AI总结:

本研究构建OccludeBench基准测试VLMs对图像缺失内容的推理能力,发现其在直接与推理案例间存在20.0个百分点的准确率差距,提出OccludeTrace诊断瓶颈,“先修复再验证”干预可缩小该差距。

摘要:

视觉语言模型(VLMs)对图像中缺失内容的推理能力如何?我们通过OccludeBench研究该问题,OccludeBench是一个人工整理的基准,用于从剩余视觉证据中恢复完全被遮挡的物体。OccludeBench将直接案例与推理案例区分开来:直接案例中,隐藏物体必须从阴影或反射等特定物理线索中恢复;推理案例中,可从更广泛的场景上下文推断。在5个VLMs上,直接案例的准确率平均为42.7%,推理案例为62.7%,存在20.0个百分点的差距。人类在直接和推理图像上的准确率分别为63.4%和68.5%,无显著差距。为定位该失败原因,我们引入OccludeTrace,这是一个四阶段诊断方法,用于分离视觉推理过程的崩溃环节。瓶颈出现在模型需从全图像投影中恢复相关物体时:模型在失败案例中仅成功50%,而在正确回答的案例中成功率达91%。扩展推理、工具使用和少样本提示的改进效果不一致,无法可靠消除差距。相比之下,“先修复再验证”干预方法(提示模型先生成候选隐藏物体,再针对可见物理线索评估该假设),对于具备图像生成能力的模型,可将差距缩小至无统计学显著差异。综上,我们的结果建立了一个可操作、可控的诊断方法,用于分离VLMs对图像中缺失视觉内容的推理方式。

英文摘要:

How well can vision-language models (VLMs) reason about what is missing from an image? We study this question with OccludeBench, a manually curated benchmark for recovering fully occluded objects from the visual evidence that remains. OccludeBench distinguishes direct cases, where the hidden object must be recovered from a specific physical cue such as a shadow or reflection, from inferential cases, where it can be inferred from broader scene context. Across five VLMs, accuracy averages 42.7% on direct cases versus 62.7% on inferential cases, exhibiting a 20.0-point gap. Humans attain higher accuracies of 63.4% versus 68.5% for direct and inferential images, respectively, with no significant gap. To locate this failure, we introduce OccludeTrace, a four-stage diagnostic that isolates where the visual reasoning process breaks down. The bottleneck emerges when models must recover the relevant object from the projection within the full image: they succeed on only 50% of failed examples, compared with 91% of examples they answer correctly. Extended reasoning, tool use, and few-shot prompting yield inconsistent improvements and do not reliably eliminate the gap. In contrast, an inpaint-then-confront intervention, which prompts models to first generate a candidate hidden object and then evaluate that hypothesis against the visible physical cue, reduces the gap to a statistically non-significant difference for both models with image-generation capabilities. Together, our results establish a tractable, controlled diagnostic for isolating how VLMs reason about visual content that is missing from an image.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑