Scene-R1：用于3D场景推理的视频基础大型语言模型，无需3D标注
Scene-R1: Video-Grounded Large Language Models for 3D Scene Reasoning without 3D Annotations
AI总结：
Scene-R1提出视频接地框架，结合强化学习推理与两阶段接地，无需3D标注即可实现3D场景推理，超越开放词汇基线并提供透明推理依据。
摘要：
目前，利用大型语言模型理解3D世界正变得流行。然而，现有的3D感知大型语言模型如同黑盒：它们输出边界框或文本答案，却不揭示这些决策是如何做出的，并且仍然依赖预训练的3D检测器来提供物体提议。我们引入了Scene-R1，一个视频基础框架，通过将强化学习驱动的推理与两阶段接地流程相结合，无需任何逐点3D实例监督即可学习推理3D场景。在时间接地阶段，我们显式地推理视频，并选择与开放式查询最相关的视频片段。在随后的图像接地阶段，我们分析图像并预测2D边界框。之后，我们使用SAM2跟踪物体，在RGB帧中生成像素级精确掩码，并将其投影回3D，从而消除了对基于3D检测器的提议的需求，同时捕获精细的几何和材质线索。Scene-R1还可以适应3D视觉问答任务，直接根据视频回答自由形式的问题。我们的训练流程仅需要任务级2D框或文本标签，无需密集的3D逐点标签。Scene-R1在多个数据集上超越了现有的开放词汇基线，同时提供了透明、逐步的推理依据。这些结果表明，基于强化学习的推理与RGB-D视频相结合，为可信赖的3D场景理解提供了一条实用、标注高效的途径。
英文摘要：
Currently, utilizing large language models to understand the 3D world is becoming popular. Yet existing 3D-aware LLMs act as black boxes: they output bounding boxes or textual answers without revealing how those decisions are made, and they still rely on pre-trained 3D detectors to supply object proposals. We introduce Scene-R1, a video-grounded framework that learns to reason about 3D scenes without any point-wise 3D instance supervision by pairing reinforcement-learning-driven reasoning with a two-stage grounding pipeline. In the temporal grounding stage, we explicitly reason about the video and select the video snippets most relevant to an open-ended query. In the subsequent image grounding stage, we analyze the image and predict the 2D bounding box. After that, we track the object using SAM2 to produce pixel-accurate masks in RGB frames, and project them back into 3D, thereby eliminating the need for 3D detector-based proposals while capturing fine geometry and material cues. Scene-R1 can also adapt to the 3D visual question answering task to answer free-form questions directly from video. Our training pipeline only needs task-level 2D boxes or textual labels without dense 3D point-wise labels. Scene-R1 surpasses existing open-vocabulary baselines on multiple datasets, while delivering transparent, step-by-step rationales. These results show that reinforcement-learning-based reasoning combined with RGB-D video alone offers a practical, annotation-efficient route to trustworthy 3D scene understanding.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
