arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2605.05997cs.CV
收藏

4DThinker: 用4D图像进行动态空间理解的思考

4DThinker: Thinking with 4D Imagery for Dynamic Spatial Understanding

Tsinghua University, SIGS(清华大学 SIGS);Meituan(美团);The Chinese University of Hong Kong(香港中文大学);National University of Singapore(新加坡国立大学);LMMs-Lab(LMMs实验室);University of California, Los Angeles(加州大学洛杉矶分校)

Zhangquan Chen, Manyuan Zhang, Xinlei Yu, Xiang An, Bo Li, Xin Xie, ZiDong Wang, Mingze Sun, Shuang Chen, Hongyu Li, Xiaobin Hu, Ruqi Huang

AI总结:

提出4DThinker框架,通过动态潜在心理图像(在连续隐藏空间中模拟场景演化)增强视觉语言模型的动态空间推理能力,并引入无标注数据生成、动态图像微调及4D强化学习,在多个基准上超越强基线。

摘要:

从单目视频中进行动态空间推理对于连接视觉智能与物理世界至关重要,但对视觉语言模型(VLM)仍然具有挑战性。先前的方法要么将时空推理完全表述为文本,这对于复杂动态来说本质上是冗长且不精确的,要么依赖外部几何模块,这增加了推理复杂性而不培养内在模型能力。在本文中,我们提出了4DThinker,这是第一个使VLM能够通过动态潜在心理图像(即在连续隐藏空间内模拟场景如何演化)进行“4D思考”的框架。具体来说,我们首先引入了一个可扩展的、无需标注的数据生成流程,从原始视频中合成4D推理数据。然后我们提出了动态图像微调(DIFT),它联合监督文本令牌和4D潜在变量,将模型锚定在动态视觉语义中。在此基础上,4D强化学习(4DRL)通过基于结果的奖励进一步处理复杂推理任务,将策略梯度限制在文本令牌上以确保稳定优化。在多个动态空间推理基准上的大量实验表明,4DThinker始终优于强基线,并为VLM中的4D推理提供了新视角。我们的代码可在https://github.com/zhangquanchen/4DThinker获取。

英文摘要:

Dynamic spatial reasoning from monocular video is essential for bridging visual intelligence and the physical world, yet remains challenging for vision-language models (VLMs). Prior approaches either verbalize spatial-temporal reasoning entirely as text, which is inherently verbose and imprecise for complex dynamics, or rely on external geometric modules that increase inference complexity without fostering intrinsic model capability. In this paper, we present 4DThinker, the first framework that enables VLMs to "think with 4D" through dynamic latent mental imagery, i.e., internally simulating how scenes evolve within the continuous hidden space. Specifically, we first introduce a scalable, annotation-free data generation pipeline that synthesizes 4D reasoning data from raw videos. We then propose Dynamic-Imagery Fine-Tuning (DIFT), which jointly supervises textual tokens and 4D latents to ground the model in dynamic visual semantics. Building on this, 4D Reinforcement Learning (4DRL) further tackles complex reasoning tasks via outcome-based rewards, restricting policy gradients to text tokens to ensure stable optimization. Extensive experiments across multiple dynamic spatial reasoning benchmarks demonstrate that 4DThinker consistently outperforms strong baselines and offers a new perspective toward 4D reasoning in VLMs. Our code is available at https://github.com/zhangquanchen/4DThinker.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑