动态思维：多模态大语言模型如何感知、跟踪和推理物理4D世界的动态
Thinking in Dynamics: How Multimodal Large Language Models Perceive, Track, and Reason Dynamics in Physical 4D World
AI总结：
本文提出Dyn-Bench基准测试，评估多模态大语言模型在动态场景中的时空推理和动态物体感知能力，发现现有模型难以同时保持强性能，而结构化整合方法显著提升其动态感知能力。
摘要：
人类生活在具有几何结构和语义内容随时间演变的物理4D世界中，构成动态4D现实。尽管当前多模态大语言模型（MLLMs）在静态视觉理解上表现优异，它们能否具备'动态思维'能力，即感知、跟踪和推理动态场景中的时空动态？为系统评估其时空推理和局部动态感知能力，我们引入Dyn-Bench，一个基于多样化真实世界和合成视频数据集的大型基准测试，能够实现稳健且可扩展的时空理解评估。通过多阶段过滤来自海量2D和4D数据源，Dyn-Bench提供高质量的动态场景集合，包含1000个视频、7000个视觉问答（VQA）对和3000个动态物体定位对。我们探测通用、空间和区域级MLLMs，揭示它们如何在语言和视觉上思考动态，发现现有模型难以同时保持在时空推理和动态物体定位上的强性能，常产生运动和交互的不一致解释。值得注意的是，传统提示策略（如推理链或基于描述的提示）提供有限改进，而结构化整合方法，包括Mask-Guided Fusion和Spatio-Temporal Textual Cognitive Map（ST-TCM），显著增强MLLMs在物理4D世界中的动态感知和时空推理能力。代码和基准可在https://dyn-bench.github.io/获取。
英文摘要：
Humans inhabit a physical 4D world where geometric structure and semantic content evolve over time, constituting a dynamic 4D reality (spatial with temporal dimension). While current Multimodal Large Language Models (MLLMs) excel in static visual understanding, can they also be adept at "thinking in dynamics", i.e., perceive, track and reason about spatio-temporal dynamics in evolving scenes? To systematically assess their spatio-temporal reasoning and localized dynamics perception capabilities, we introduce Dyn-Bench, a large-scale benchmark built from diverse real-world and synthetic video datasets, enabling robust and scalable evaluation of spatio-temporal understanding. Through multi-stage filtering from massive 2D and 4D data sources, Dyn-Bench provides a high-quality collection of dynamic scenes, comprising 1k videos, 7k visual question answering (VQA) pairs, and 3k dynamic object grounding pairs. We probe general, spatial and region-level MLLMs to express how they think in dynamics both linguistically and visually, and find that existing models cannot simultaneously maintain strong performance in both spatio-temporal reasoning and dynamic object grounding, often producing inconsistent interpretations of motion and interaction. Notably, conventional prompting strategies (e.g., chain-of-thought or caption-based hints) provide limited improvement, whereas structured integration approaches, including Mask-Guided Fusion and Spatio-Temporal Textual Cognitive Map (ST-TCM), significantly enhance MLLMs' dynamics perception and spatio-temporal reasoning in the physical 4D world. Code and benchmark are available at https://dyn-bench.github.io/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
