SNOW：基于世界知识的时空场景理解用于开放世界具身推理
SNOW: Spatio-Temporal Scene Understanding with World Knowledge for Open-World Embodied Reasoning
AI总结：
SNOW通过整合视觉语言模型与点云几何，实现统一的4D场景理解，提升开放世界具身推理的性能。
摘要：
自主机器人系统需要对动态环境进行时空理解，以确保可靠的导航和交互。虽然视觉-语言模型（VLMs）提供了开放世界的语义先验，但它们缺乏对3D几何和时间动态的支撑。相反，几何感知捕捉结构和运动，但仍然语义稀疏。我们提出了SNOW（Scene Understanding with Open-World Knowledge），一种无需训练且不依赖骨干网络的框架，用于统一的4D场景理解，它整合了VLM衍生的语义与点云几何和时间一致性。SNOW处理同步的RGB图像和3D点云，使用HDBSCAN聚类生成物体级提案，这些提案指导SAM2分割。每个分割区域通过我们提出的时空令牌化补丁编码（STEP）进行编码，生成多模态令牌，捕捉局部语义、几何和时间属性。这些令牌逐步整合到4D场景图（4DSG）中，该图作为4D先验用于后续推理。一个轻量级的SLAM后端将所有STEP令牌在环境中锚定，提供全局参考对齐，并确保时间内的无歧义空间定位。最终的4DSG通过形成可查询的统一世界模型，使VLM能够直接解释空间场景结构和时间动态。在多样化的基准测试中，SNOW实现了精确的4D场景理解和空间支撑的推断，从而在多个设置中设定了新的最先进性能，突显了结构化4D先验对于具身推理和自主机器人的重要性。
英文摘要：
Autonomous robotic systems require spatio-temporal understanding of dynamic environments to ensure reliable navigation and interaction. While Vision-Language Models (VLMs) provide open-world semantic priors, they lack grounding in 3D geometry and temporal dynamics. Conversely, geometric perception captures structure and motion but remains semantically sparse. We propose SNOW (Scene Understanding with Open-World Knowledge), a training-free and backbone-agnostic framework for unified 4D scene understanding that integrates VLM-derived semantics with point cloud geometry and temporal consistency. SNOW processes synchronized RGB images and 3D point clouds, using HDBSCAN clustering to generate object-level proposals that guide SAM2-based segmentation. Each segmented region is encoded through our proposed Spatio-Temporal Tokenized Patch Encoding (STEP), producing multimodal tokens that capture localized semantic, geometric, and temporal attributes. These tokens are incrementally integrated into a 4D Scene Graph (4DSG), which serves as 4D prior for downstream reasoning. A lightweight SLAM backend anchors all STEP tokens spatially in the environment, providing the global reference alignment, and ensuring unambiguous spatial grounding across time. The resulting 4DSG forms a queryable, unified world model through which VLMs can directly interpret spatial scene structure and temporal dynamics. Experiments on a diverse set of benchmarks demonstrate that SNOW enables precise 4D scene understanding and spatially grounded inference, thereby setting new state-of-the-art performance in several settings, highlighting the importance of structured 4D priors for embodied reasoning and autonomous robotics.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
