3D-R1: 增强3D视觉语言模型的推理能力以实现统一场景理解
3D-R1: Enhancing Reasoning in 3D VLMs for Unified Scene Understanding
AI总结：
针对现有3D视觉语言模型推理与泛化能力不足的问题，研究提出3D-R1基础模型，通过构建带CoT的Scene-30K数据集、结合GRPO的RLHF训练与多奖励函数、动态视角选择策略，实现3D场景基准平均10%的性能提升。
摘要：
大型视觉语言模型（VLM）在2D视觉理解任务中已取得重大进展，引发了将这些能力扩展到3D场景理解的研究兴趣。然而，受限于高质量空间数据的不足以及视角假设的静态特性，当前的3D VLM往往难以实现稳健的推理和泛化。为应对这些挑战，我们提出了3D-R1，这是一种可增强3D VLM推理能力的基础模型。具体而言，我们首先利用现有3D-VL数据集和基于Gemini 2.5 Pro的数据引擎，构建了一个带有思维链（CoT）的高质量合成数据集Scene-30K，作为3D-R1的冷启动初始化数据。此外，我们在强化学习训练过程中采用GRPO等基于人类反馈的强化学习（RLHF）策略来增强推理能力，并引入三种奖励函数：感知奖励、语义相似度奖励和格式奖励，以维持检测精度和答案的语义准确性。我们还提出了一种动态视角选择策略，可自适应地选择信息最丰富的视角用于3D场景理解。大量实验表明，3D-R1在各类3D场景基准测试上平均性能提升10%，凸显了其在增强3D场景理解的推理和泛化能力方面的有效性。代码链接：https://github.com/AIGeeksGroup/3D-R1，项目主页：https://aigeeksgroup.github.io/3D-R1/。
英文摘要：
Large vision-language models (VLMs) have made significant strides in 2D visual understanding tasks, sparking interest in extending these capabilities to 3D scene understanding. However, current 3D VLMs often struggle with robust reasoning and generalization due to limitations in high-quality spatial data and the static nature of viewpoint assumptions. To address these challenges, we propose 3D-R1, a foundation model that enhances the reasoning capabilities of 3D VLMs. Specifically, we first construct a high-quality synthetic dataset with CoT, named Scene-30K, leveraging existing 3D-VL datasets and a data engine based on Gemini 2.5 Pro. It serves as cold-start initialization data for 3D-R1. Moreover, we leverage RLHF policy such as GRPO in the reinforcement learning training process to enhance reasoning capabilities and introduce three reward functions: a perception reward, a semantic similarity reward and a format reward to maintain detection accuracy and answer semantic precision. Furthermore, we introduce a dynamic view selection strategy that adaptively chooses the most informative perspectives for 3D scene understanding. Extensive experiments demonstrate that 3D-R1 delivers an average improvement of 10% across various 3D scene benchmarks, highlighting its effectiveness in enhancing reasoning and generalization in 3D scene understanding. Code: https://github.com/AIGeeksGroup/3D-R1. Website: https://aigeeksgroup.github.io/3D-R1.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
