arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.39639cs.CVcs.AI
收藏

视觉语言模型在3D推理中会在何处失效?

Where Do Vision Language Models Fail at 3D Reasoning?

Khai Le-Duc, Nguyen Kim Hai Bui, An Ngo, Nam D. Hoang, Yueqi Song, Jiarui Liu, Thai Quoc Hoang, Fang Wu, Chris Ngo, Terry Jingchen Zhang, Mona T. Diab, Ran Xu, … 展开作者

Khai Le-Duc, Nguyen Kim Hai Bui, An Ngo, Nam D. Hoang, Yueqi Song, Jiarui Liu, Thai Quoc Hoang, Fang Wu, Chris Ngo, Terry Jingchen Zhang, Mona T. Diab, Ran Xu, Silvio Savarese, Xiao-Ming Wu

AI总结:

该研究引入BedBench基准,发现视觉语言模型在3D推理的按颜色计数任务中表现远差于人类,微调可提升准确率但存在一致性问题,探测Qwen3.5-27B揭示其计数恢复与答案输出的差距,提出需从多维度评估3D推理。

摘要:

视觉语言模型可以对单个判断做出正确预测,但难以恢复并一致使用3D物体的精确属性。为了描述这一差距,我们引入BedBench,这是一个包含13个程序生成任务的受控基准,涵盖单个判断和复合答案。在12个开放权重模型和4个前沿模型中,按颜色计数暴露出明显的弱点:人类准确率达93.3%,而最佳模型仅为7.2%。针对性微调使某模型的准确率从3.0%提升至68.6%,但也揭示了进一步的困难:其关于最常见颜色的答案与自身按颜色的计数不一致。将这些计数作为文本提供可改善其判断,表明恢复属性并不意味着一致使用它。推理、缩放和添加视觉线索均能改善单个判断,但都无法使精确答案可靠。为定位困难,我们探测Qwen3.5-27B:精确的按颜色计数在语言模型输入处仍可恢复,但模型的答案很少能恢复这些计数(探测准确率为65.0%,模型仅为3.0%)。移除图像的控制实验进一步表明,一些看似成功的结果不需要视觉证据。这些发现推动通过精确正确性、一致使用和图像依赖性来评估3D推理,而非仅用准确率。代码和数据集已公开:this https URL

英文摘要:

Vision-language models can get individual judgments right while struggling to recover and consistently use the exact properties of a 3D object. To characterize this gap, we introduce BedBench, a controlled benchmark of thirteen procedurally generated tasks spanning individual judgments and composite answers. Across twelve open-weight and four frontier models, per-color counting exposes a pronounced weakness: humans achieve 93.3% accuracy, compared with 7.2% for the best model. Targeted fine-tuning raises one model's accuracy from 3.0% to 68.6%, but reveals a further difficulty: its answers about the most common colors remain inconsistent with its own per-color counts. Supplying those counts as text improves its judgments, showing that recovering a property does not mean using it consistently. Reasoning, scaling, and added visual cues each improve individual judgments, but none makes exact answers reliable. To locate the difficulty, we probe Qwen3.5-27B: exact per-color counts remain recoverable at the language model's input, yet the model's answers rarely recover them (65.0% for the probe vs. 3.0% for the model). Image-removal controls further show that some apparent successes require no visual evidence. These findings motivate evaluating 3D reasoning by exact correctness, consistent use, and image dependence, rather than accuracy alone. Code and dataset are publicly available: https://github.com/leduckhai/BedBench

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑