ZING-3D：基于视觉语言模型的零样本增量式3D场景图
ZING-3D: Zero-shot Incremental 3D Scene Graphs via Vision-Language Models
AI总结：
本文提出ZING-3D框架，利用预训练视觉语言模型实现零样本开放词汇3D场景图生成，支持增量更新和几何锚定，在Replica和HM3D数据集上验证了其无需任务特定训练即可有效捕捉空间与语义关系。
摘要：
理解和推理复杂的3D环境需要结构化的场景表示，这种表示不仅要捕捉物体，还要捕捉它们的语义和空间关系。虽然近期关于3D场景图生成的工作利用了预训练的视觉语言模型（VLM）而无需任务特定的微调，但这些工作大多局限于单视角设置，无法支持随着新观测到来而进行的增量更新，并且缺乏在3D空间中的显式几何锚定，所有这些对于具身场景都是必不可少的。在本文中，我们提出了ZING-3D，一个利用预训练基础模型的广泛知识来实现开放词汇识别并以零样本方式生成场景的丰富语义表示的框架，同时支持增量更新和3D空间中的几何锚定，使其适用于下游机器人应用。我们的方法利用VLM推理生成丰富的2D场景图，并利用深度信息将其锚定到3D中。节点表示具有特征、3D位置和语义上下文的开放词汇物体，而边则通过物体间距离捕捉空间和语义关系。我们在Replica和HM3D数据集上的实验表明，ZING-3D无需任务特定训练即可有效捕捉空间和关系知识。
英文摘要：
Understanding and reasoning about complex 3D environments requires structured scene representations that capture not only objects but also their semantic and spatial relationships. While recent works on 3D scene graph generation have leveraged pretrained VLMs without task-specific fine-tuning, they are largely confined to single-view settings, fail to support incremental updates as new observations arrive and lack explicit geometric grounding in 3D space, all of which are essential for embodied scenarios. In this paper, we propose, ZING-3D, a framework that leverages the vast knowledge of pretrained foundation models to enable open-vocabulary recognition and generate a rich semantic representation of the scene in a zero-shot manner while also enabling incremental updates and geometric grounding in 3D space, making it suitable for downstream robotics applications. Our approach leverages VLM reasoning to generate a rich 2D scene graph, which is grounded in 3D using depth information. Nodes represent open-vocabulary objects with features, 3D locations, and semantic context, while edges capture spatial and semantic relations with inter-object distances. Our experiments on scenes from the Replica and HM3D dataset show that ZING-3D is effective at capturing spatial and relational knowledge without the need of task-specific training.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
