Current open-vocabulary scene graph generation algorithms highly rely on both 3D scene point cloud data and posed RGB-D images and thus have limited applications in scenarios where RGB-D images or camera poses are not readily available. To solve this problem, we propose Point2Graph, a novel end-to-end point cloud-based 3D open-vocabulary scene graph generation framework in which the requirement of posed RGB-D image series is eliminated. This hierarchical framework contains room and object detection/segmentation and open-vocabulary classification. For the room layer, we leverage the advantage of merging the geometry-based border detection algorithm with the learning-based region detection to segment rooms and create a "Snap-Lookup" framework for open-vocabulary room classification. In addition, we create an end-to-end pipeline for the object layer to detect and classify 3D objects based solely on 3D point cloud data. Our evaluation results show that our framework can outperform the current state-of-the-art (SOTA) open-vocabulary object and room segmentation and classification algorithm on widely used real-scene datasets.
OpenObject-NAV: Open-Vocabulary Object-Oriented Navigation Based on Dynamic Carrier-Relationship Scene Graph
School of Automation, Beijing Institute of Technology（北京理工大学自动化学院）；School of Mechanical Engineering, University of Nottingham Ningbo China（宁波诺丁汉大学机械工程学院）
Tufts University（塔夫茨大学）；Bosch Center for Artificial Intelligence（博世人工智能中心）；Carnegie Mellon University（卡内基梅隆大学）
Generalisation to unseen contexts remains a challenge for embodied navigation agents. In the context of semantic audio-visual navigation (SAVi) tasks, the notion of generalisation should include both generalising to unseen indoor visual scenes as well as generalising to unheard sounding objects. However, previous SAVi task definitions do not include evaluation conditions on truly novel sounding objects, resorting instead to evaluating agents on unheard sound clips of known objects; meanwhile, previous SAVi methods do not include explicit mechanisms for incorporating domain knowledge about obje…
Open-Vocabulary Spatio-Temporal Scene Graph for Robot Perception and Teleoperation Planning
HI-Robot Lab, School of Automation Science and Engineering, South China University of Technology（华南理工大学自动化科学与工程学院）；Institute of AI Industries, Chinese Academy of Science（中国科学院人工智能产业研究所）；School of Computer Science, Central China Normal University（中央财经大学计算机科学学院）；School of Computer Science and Informatics, University of Liverpool（利物浦大学计算机科学与信息学院）
OGScene3D: Incremental Open-Vocabulary 3D Gaussian Scene Graph Mapping for Scene Understanding
Shanghai Jiao Tong University（上海交通大学）；University of Cambridge（剑桥大学）；University of Technology Nuremberg（纽伦堡技术大学）；Nanyang Technological University（南洋理工大学）；Department of Automation, Key Laboratory of System Control and Information Processing of Ministry of Education, State Key Laboratory of Avionics Integration and Aviation System-of-Systems Synthesis, Shanghai Key Laboratory of Navigation and Location Based Services, Shanghai Jiao Tong University（自动化系，教育部系统控制与信息处理重点实验室，航空系统集成与航空系统-of-系统综合国家重点实验室，上海导航与定位基于服务重点实验室，上海交通大学）
Jet Propulsion Laboratory（喷气推进实验室）；California Institute of Technology（加州理工学院）；Swiss Federal Institute of Technology（瑞士联邦理工学院）；ETH Zürich（苏黎世联邦理工学院）；FieldAI Inc.（FieldAI公司）；Stanford University（斯坦福大学）；University of California Berkeley（加州大学伯克利分校）
Deep reinforcement learning (DRL) demonstrates its potential in learning a model-free navigation policy for robot visual navigation. However, the data-demanding algorithm relies on a large number of navigation trajectories in training. Existing datasets supporting training such robot navigation algorithms consist of either 3D synthetic scenes or reconstructed scenes. Synthetic data suffers from domain gap to the real-world scenes while visual inputs rendered from 3D reconstructed scenes have undesired holes and artifacts. In this paper, we present a new dataset collected in real-world to facil…
SSF-PAN: Semantic Scene Flow-Based Perception for Autonomous Navigation in Traffic Scenarios
Research Institute of Trustworthy Autonomous Systems, Southern University of Science and Technology（南方科技大学可信自主系统研究院）；Shenzhen Deeproute.ai Co.,Ltd（深圳元戎启行科技有限公司）
Uncertainty-Informed Active Perception for Open Vocabulary Object Goal Navigation
Center for Robotics, University of Bonn（波恩大学机器人中心）；MAVLab, TU Delft（代尔夫特理工大学MAVLab）；Lamarr Institute for Machine Learning and Artificial Intelligence（机器学习与人工智能拉马尔研究所）
HyperDCM: Dynamic Cluster Memory Replay in Hyperbolic Space for Continual Robotic Navigation Across Scenes
Software Engineering Institute, East China Normal University（华东师范大学软件工程学院）；School of Geospatial Information, Information Engineering University（信息工程大学地理空间信息学院）；University of Shanghai for Science and Technology（上海理工大学）；Shanghai Jiao Tong University（上海交通大学）
GALA: Guided Attention with Language Alignment for Open Vocabulary Gaussian Splatting
Technical University of Munich（慕尼黑技术大学）；Universitat Politècnica de Catalunya（巴塞罗那技术大学）；Google（谷歌）；Munich Center for Machine Learning（慕尼黑机器学习中心）；University of Tubingen（图宾根大学）；ETH Zurich（苏黎世联邦理工学院）；Visualais
IIIT-Hyderabad（国际信息技术研究所海得拉巴分校）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）
We embark on a hitherto unreported problem of an autonomous robot (self-driving car) navigating in dynamic scenes in a manner that reduces its localization error and eventual cumulative drift or Absolute Trajectory Error, which is pronounced in such dynamic scenes. With the hugely popular Velodyne-16 3D LIDAR as the main sensing modality, and the accurate LIDAR-based Localization and Mapping algorithm, LOAM, as the state estimation framework, we show that in the absence of a navigation policy, drift rapidly accumulates in the presence of moving objects. To overcome this, we learn actions that …
Learning Robot Visual Navigation in Crowds via Intention-Aware Scene Representations
Shenzhen Key Laboratory of Robotics Perception and Intelligence, Department of Electronic and Electrical Engineering, SUSTech（南方科技大学电子与电气工程系深圳市机器人感知与智能重点实验室）；Jiaxing Research Institute, SUSTech（南方科技大学嘉兴研究院）；Research Institute of MA&EI, Peng Cheng Laboratory（鹏城实验室机器人与人工智能研究所）
We present CoMet, a novel approach for computing a group's cohesion and using that to improve a robot's navigation in crowded scenes. Our approach uses a novel cohesion-metric that builds on prior work in social psychology. We compute this metric by utilizing various visual features of pedestrians from an RGB-D camera on-board a robot. Specifically, we detect characteristics corresponding to proximity between people, their relative walking speeds, the group size, and interactions between group members. We use our cohesion-metric to design and improve a navigation scheme that accounts for diffe…
Large language models (LLMs) have unlocked new capabilities of task planning from human instructions. However, prior attempts to apply LLMs to real-world robotic tasks are limited by the lack of grounding in the surrounding scene. In this paper, we develop NLMap, an open-vocabulary and queryable scene representation to address this problem. NLMap serves as a framework to gather and integrate contextual information into LLM planners, allowing them to see and query available objects in the scene before generating a context-conditioned plan. NLMap first establishes a natural language queryable sc…
