基于空间代码的物理世界视频推理
Thinking with Spatial Code for Physical-World Video Reasoning
AI总结：
基于空间代码的物理世界视频推理方法，通过空间编码器生成3D表示并提升大语言模型的推理能力。
摘要：
我们引入了Thinking with Spatial Code框架，该框架将RGB视频转换为显式的、时间上连贯的3D表示，用于物理世界视觉问答。我们强调实证发现，所提出的空间编码器能够将视频解析为具有显式3D定向边界框和语义标签的结构化空间代码，使大型语言模型（LLMs）能够直接在显式空间变量上进行推理。具体而言，我们提出了一个空间编码器，通过统一6D物体解析和跟踪backbone与几何预测来编码图像和几何特征，并进一步通过使用空间rubric奖励对LLMs进行微调，采用强化学习鼓励视角感知、几何基础的推理。结果，我们的模型在VSI-Bench上优于专有视觉-语言模型，设定了新的state-of-the-art。代码可在https://github.com/Beckschen/spatialcode获取。
英文摘要：
We introduce Thinking with Spatial Code, a framework that transforms RGB video into explicit, temporally coherent 3D representations for physical-world visual question answering. We highlight the empirical finding that our proposed spatial encoder can parse videos into structured spatial code with explicit 3D oriented bounding boxes and semantic labels, enabling large language models (LLMs) to reason directly over explicit spatial variables. Specifically, we propose the spatial encoder that encodes image and geometric features by unifying 6D object parsing and tracking backbones with geometric prediction, and we further finetuning LLMs with reinforcement learning using a spatial rubric reward that encourages perspective-aware, geometrically grounded inference. As a result, our model outperforms proprietary vision-language models on VSI-Bench, setting a new state-of-the-art. Code is available at https://github.com/Beckschen/spatialcode.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
