arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2601.01547cs.CVcs.AIcs.LG
收藏

视觉-语言模型在物理动态和意图推理上仍逊于人类表现

Vision-language models lag human performance on physical dynamics and intent reasoning

East China Normal University(东华大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Shanghai Innovation Institute(上海创新研究院);University of Agder(阿格德大学)

Tianjun Gu, Jingyu Gong, Zhizhong Zhang, Yuan Xie, Lizhuang Ma, Xin Tan, Athanasios V

AI总结:

本文提出Teleo-Spatial Intelligence(TSI)以连接时空变化与目标导向结构,通过EscherVerse大规模开放世界资源评估,发现视觉-语言模型在开放世界中仍无法达到人类水平的意图推理能力。

摘要:

空间智能是具身认知的核心,但当前AI系统仍难以在开放世界中推理物理互动。尽管在受控基准上表现优异,视觉-语言模型往往无法联合建模物理动态、参考系和驱动空间变化的隐含人类意图。我们引入Teleo-Spatial Intelligence(TSI),一种将时空变化与目标导向结构联系起来的推理能力。为评估TSI,我们提出了EscherVerse,一个由11328个真实世界视频构建的大规模开放世界资源,包括8000个示例基准和35963个指令微调集。在27种最先进的视觉-语言模型和11名标注员的首次回应分析中,我们发现存在持续的teleo-空间推理差距:最强的专有模型总体准确率为57.26%,远低于首次回应的人类表现,范围从84.81%到95.14%,平均为90.62%。在真实世界、意图感知数据上微调缩小了开放权重模型的差距,但未完全解决。EscherVerse为目的感知的空间推理提供了诊断测试平台,并突显了模式识别与人类水平理解在具身AI中的关键差距。

英文摘要:

Spatial intelligence is central to embodied cognition, yet contemporary AI systems still struggle to reason about physical interactions in open-world human environments. Despite strong performance on controlled benchmarks, vision-language models often fail to jointly model physical dynamics, reference frames, and the latent human intentions that drive spatial change. We introduce Teleo-Spatial Intelligence (TSI), a reasoning capability that links spatiotemporal change to goal-directed structure. To evaluate TSI, we present EscherVerse, a large-scale open-world resource built from 11,328 real-world videos, including an 8,000-example benchmark and a 35,963-example instruction-tuning set. Across 27 state-of-the-art vision-language models and an independent analysis of first-pass human responses from 11 annotators, we identify a persistent teleo-spatial reasoning gap: the strongest proprietary model achieves 57.26\% overall accuracy, far below first-pass human performance, which ranges from 84.81\% to 95.14\% with a mean of 90.62\%. Fine-tuning on real-world, intent-aware data narrows this gap for open-weight models, but does not close it. EscherVerse provides a diagnostic testbed for purpose-aware spatial reasoning and highlights a critical gap between pattern recognition and human-level understanding in embodied AI.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑