RoboRefer: 向视觉语言模型在机器人中的空间指称推理迈进
RoboRefer: Towards Spatial Referring with Reasoning in Vision-Language Models for Robotics
AI总结：
RoboRefer通过整合深度编码器和强化微调方法，实现了视觉语言模型在机器人中的空间指称推理，提升了复杂场景下的交互能力。
摘要：
空间指称是实体机器人与3D物理世界交互的基本能力。然而，即使使用强大的预训练视觉语言模型（VLMs），最近的方法仍然无法准确理解复杂的3D场景并动态推理指示位置以进行交互。为此，我们提出了RoboRefer，一种具有3D意识的VLM，它首先通过集成一个解耦但专用的深度编码器，通过监督微调（SFT）实现精确的空间理解。此外，RoboRefer通过强化微调（RFT）实现了通用的多步空间推理，其度量敏感的进程奖励函数专门针对空间指称任务。为了支持SFT和RFT训练，我们引入了RefSpatial，一个包含2000万对问答对（2倍于先前）的大规模数据集，覆盖了31种空间关系（相比15种先前）并支持复杂的推理过程（最多5步）。此外，我们引入了RefSpatial-Bench，一个具有挑战性的基准，填补了在具有多步推理的空间指称评估中的空白。实验表明，经过SFT训练的RoboRefer在空间理解方面达到了最先进的水平，平均成功率为89.6%。经过RFT训练的RoboRefer进一步在所有其他基线之上取得了显著优势，甚至在RefSpatial-Bench上以17.4%的平均准确率超过了Gemini-2.5-Pro。值得注意的是，RoboRefer可以与各种控制策略集成，以在多样化的机器人（例如UR5、G1人形机器人）中执行长周期、动态任务，在拥挤的现实世界场景中。
英文摘要：
Spatial referring is a fundamental capability of embodied robots to interact with the 3D physical world. However, even with the powerful pretrained vision language models (VLMs), recent approaches are still not qualified to accurately understand the complex 3D scenes and dynamically reason about the instruction-indicated locations for interaction. To this end, we propose RoboRefer, a 3D-aware VLM that can first achieve precise spatial understanding by integrating a disentangled but dedicated depth encoder via supervised fine-tuning (SFT). Moreover, RoboRefer advances generalized multi-step spatial reasoning via reinforcement fine-tuning (RFT), with metric-sensitive process reward functions tailored for spatial referring tasks. To support SFT and RFT training, we introduce RefSpatial, a large-scale dataset of 20M QA pairs (2x prior), covering 31 spatial relations (vs. 15 prior) and supporting complex reasoning processes (up to 5 steps). In addition, we introduce RefSpatial-Bench, a challenging benchmark filling the gap in evaluating spatial referring with multi-step reasoning. Experiments show that SFT-trained RoboRefer achieves state-of-the-art spatial understanding, with an average success rate of 89.6%. RFT-trained RoboRefer further outperforms all other baselines by a large margin, even surpassing Gemini-2.5-Pro by 17.4% in average accuracy on RefSpatial-Bench. Notably, RoboRefer can be integrated with various control policies to execute long-horizon, dynamic tasks across diverse robots (e,g., UR5, G1 humanoid) in cluttered real-world scenes.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
