VLM4D：迈向视觉语言模型中的时空感知
VLM4D: Towards Spatiotemporal Awareness in Vision Language Models
AI总结：
本文提出VLM4D基准，评估视觉语言模型的时空推理能力，发现其与人类差距显著，并探索4D特征场重建与时空微调以提升动态理解。
摘要：
视觉语言模型（VLMs）在整合语言与视觉推理方面展现出卓越能力，但在理解动态时空交互方面仍存在根本性局限。人类能够轻松地追踪并推理物体的运动、旋转和视角变化——这些能力对于稳健的动态现实世界理解至关重要，然而当前VLM却明显缺乏。在本文中，我们引入VLM4D，这是首个专门用于评估VLM时空推理能力的基准。我们的基准包含多样化的真实世界与合成视频，并配有精心设计的问题-答案对，重点强调平移与旋转运动、视角感知及运动连续性。通过对最先进的开源和闭源VLM进行全面评估，我们发现其与人类基线相比存在显著性能差距，凸显了现有模型的根本缺陷。广泛分析表明，VLM在整合多种视觉线索和维持时间连贯性方面尤为困难。我们进一步探索了有前景的方向，如利用4D特征场重建和针对性的时空监督微调，证明了它们在增强时空理解方面的有效性。我们的工作旨在鼓励更深入地探索提升VLM的空间与时间基础，为动态环境中更强大、更可靠的视觉智能铺平道路。
英文摘要：
Vision language models (VLMs) have shown remarkable capabilities in integrating linguistic and visual reasoning but remain fundamentally limited in understanding dynamic spatiotemporal interactions. Humans effortlessly track and reason about object movements, rotations, and perspective shifts-abilities essential for robust dynamic real-world understanding yet notably lacking in current VLMs. In this paper, we introduce VLM4D, the first benchmark specifically designed to evaluate the spatiotemporal reasoning capabilities of VLMs. Our benchmark comprises diverse real-world and synthetic videos accompanied by carefully curated question-answer pairs emphasizing translational and rotational motions, perspective awareness, and motion continuity. Through comprehensive evaluations of state-of-the-art open and closed-source VLMs, we identify significant performance gaps compared to human baselines, highlighting fundamental deficiencies in existing models. Extensive analysis reveals that VLMs struggle particularly with integrating multiple visual cues and maintaining temporal coherence. We further explore promising directions, such as leveraging 4D feature field reconstruction and targeted spatiotemporal supervised fine-tuning, demonstrating their effectiveness in enhancing spatiotemporal comprehension. Our work aims to encourage deeper exploration into improving VLMs' spatial and temporal grounding, paving the way towards more capable and reliable visual intelligence for dynamic environments.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
