arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.13294cs.CVcs.AI
收藏

VisPhyWorld: 通过代码驱动的视频重建探测物理推理

VisPhyWorld: Probing Physical Reasoning via Code-Driven Video Reconstruction

University of Waterloo(滑铁卢大学);Autodesk AI Lab(Autodesk人工智能实验室);Independent Researcher(独立研究者)

Jiarong Liang, Max Ku, Ka-Hei Hui, Ping Nie, Wenhu Chen

AI总结:

本文提出VisPhyWorld框架,通过要求模型从视觉观察生成可执行的模拟器代码来评估物理推理能力,引入VisPhyBench基准测试集,验证模型在重建外观和模拟物理运动方面的能力,发现最先进的MLLM在准确推断物理参数和模拟一致的物理动态方面存在困难。

摘要:

评估多模态大语言模型(MLLMs)是否真正理解物理动态仍然具有挑战性。现有的基准测试大多依赖于识别式协议,如视觉问答(VQA)和期望违反(VoE),这些协议通常可以在不承诺明确、可测试的物理假设的情况下回答。我们提出了VisPhyWorld,一个基于执行的框架,通过要求模型从视觉观察生成可执行的模拟器代码来评估物理推理能力。通过生成可运行的代码,推断的世界表示可以直接检查、编辑和验证。这将物理推理与渲染分开。基于此框架,我们引入了VisPhyBench,包含209个评估场景,这些场景源自108个物理模板和一个系统化的协议,用于评估模型在重建外观和模拟物理合理的运动方面的能力。我们的流水线在97.7%的基准运行中生成有效的重建视频之前会回退。实验表明,尽管最先进的MLLM在语义场景理解方面表现强劲,但在准确推断物理参数和模拟一致的物理动态方面存在困难。我们的代码可在https://github.com/TIGER-AI-Lab/VisPhyWorld上获得。

英文摘要:

Evaluating whether Multimodal Large Language Models (MLLMs) genuinely reason about physical dynamics remains challenging. Most existing benchmarks rely on recognition-style protocols such as Visual Question Answering (VQA) and Violation of Expectation (VoE), which can often be answered without committing to an explicit, testable physical hypothesis. We propose VisPhyWorld, an execution-based framework that evaluates physical reasoning by requiring models to generate executable simulator code from visual observations. By producing runnable code, the inferred world representation is directly inspectable, editable, and falsifiable. This separates physical reasoning from rendering. Building on this framework, we introduce VisPhyBench, comprising 209 evaluation scenes derived from 108 physical templates and a systematic protocol that evaluates how well models reconstruct appearance and reproduce physically plausible motion. Our pipeline produces valid reconstructed videos in 97.7% of benchmark runs before fallback. Experiments show that while state-of-the-art MLLMs achieve strong semantic scene understanding, they struggle to accurately infer physical parameters and to simulate consistent physical dynamics. Our code is available https://github.com/TIGER-AI-Lab/VisPhyWorld

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑