We present an Open-Vocabulary 3D Scene Graph (OVSG), a formal framework for grounding a variety of entities, such as object instances, agents, and regions, with free-form text-based queries. Unlike conventional semantic-based object localization approaches, our system facilitates context-aware entity localization, allowing for queries such as ``pick up a cup on a kitchen table" or ``navigate to a sofa on which someone is sitting". In contrast to existing research on 3D scene graphs, OVSG supports free-form text input and open-vocabulary querying. Through a series of comparative experiments using the ScanNet dataset and a self-collected dataset, we demonstrate that our proposed approach significantly surpasses the performance of previous semantic-based localization techniques. Moreover, we highlight the practical application of OVSG in real-world robot navigation and manipulation experiments.
Think and Answer ME: Benchmarking and Exploring Multi-Entity Reasoning Grounding in Remote Sensing
Beihang University, Beijing, China（北京航空航天大学）；University of Liverpool, Liverpool, UK（利物浦大学）；Institute of Acoustics, Chinese Academy of Sciences, Beijing, China（中国科学院声学研究所）
OpenMap: Instruction Grounding via Open-Vocabulary Visual-Language Mapping
School of Software, Tsinghua University（清华大学软件学院）；School of computer science and engineering, Central South University（中南大学计算机科学与工程学院）；Inspur Yunzhou Industrial Internet Co., Ltd（Inspur Yunzhou工业互联网有限公司）；Tsinghua University（清华大学）
SynCLIP: Synonym-Coherent Language-Image Pretraining for Robust Open-Vocabulary Dense Perception
Beihang University（北京航空航天大学）；State Key Laboratory of Space Information System and Integrated Application（空间信息系统与集成应用国家重点实验室）；Nanjing University（南京大学）；Harbin Engineering University（哈尔滨工程大学）；Beijing Zhongguancun Academy（北京中关村科学城）
Embodied Multimodal Grounding for Open-Vocabulary Mobile Manipulation via Semantic 3D Gaussian Splatting
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Midea Group（美的集团）；The Hong Kong University of Science and Technology（香港科技大学）
Leveraging Vision-Language Models for Open-Vocabulary Instance Segmentation and Tracking
Autonomous Intelligent Systems, University of Bonn（博恩大学自主智能系统中心）；Lamarr Institute for Machine Learning and AI（拉马尔人工智能与机器学习研究所）；Center for Robotics, University of Bonn（博恩大学机器人中心）
Depth-Only Open-Vocabulary 3D Semantic Segmentation For Privacy-Preserving Robotic Applications
Humanoid Robots Lab, University of Bonn（波恩大学仿人机器人实验室）；Lamarr Institute for Machine Learning and Artificial Intelligence（拉马尔机器学习和人工智能研究所）；Center for Robotics, Bonn, Germany（波恩机器人中心）
Large language models (LLMs) have unlocked new capabilities of task planning from human instructions. However, prior attempts to apply LLMs to real-world robotic tasks are limited by the lack of grounding in the surrounding scene. In this paper, we develop NLMap, an open-vocabulary and queryable scene representation to address this problem. NLMap serves as a framework to gather and integrate contextual information into LLM planners, allowing them to see and query available objects in the scene before generating a context-conditioned plan. NLMap first establishes a natural language queryable sc…
CLASP: Closed-loop Asynchronous Spatial Perception for Open-vocabulary Desktop Object Grasping
Harbin Institute of Technology（哈尔滨工业大学）；National Key Laboratory of Smart Farm Technologies and Systems（智慧农场技术与系统全国重点实验室）；Peng Cheng Laboratory（鹏城实验室）；Northeastern University（东北大学）
