In order to provide a robot with the ability to understand and react to a user's natural language inputs, the natural language must be connected to the robot's underlying representations of the world. Recently, large language models (LLMs) and 3D scene graphs (3DSGs) have become a popular choice for grounding natural language and representing the world. In this work, we address the challenge of using LLMs with 3DSGs to ground natural language. Existing methods encode the scene graph as serialized text within the LLM's context window, but this encoding does not scale to large or rich 3DSGs. Instead, we propose to use a form of Retrieval Augmented Generation to select a subset of the 3DSG relevant to the task. We encode a 3DSG in a graph database and provide a query language interface (Cypher) as a tool to the LLM with which it can retrieve relevant data for language grounding. We evaluate our approach on instruction following and scene question-answering tasks and compare against baseline context window and code generation methods. Our results show that using Cypher as an interface to 3D scene graphs scales significantly better to large, rich graphs on both local and cloud-based models. This leads to large performance improvements in grounded language tasks while also substantially reducing the token count of the scene graph content. A video supplement is available at https://www.youtube.com/watch?v=zY_YI9giZSA.
Wild-Drive: Off-Road Scene Captioning and Path Planning via Robust Multi-modal Routing and Efficient Large Language Model
School of Instrument Science and Engineering, Southeast University(东南大学仪器科学与工程学院);Southeast University Nanjing Jiangbei New Area Innovation Research Institute(东南大学南京江滨新区创新研究院);National Key Laboratory of Equipment State Sensing and Smart Support, National University of Defense Technology(国防科技大学装备状态感知与智能支撑国家重点实验室);School of Transportation, Shandong University of Science and Technology(山东科技大学交通学院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所)
S3Mem: Structured Spatiotemporal Scene-Event Memory for Long-Horizon Interactive Question Answering
University of Science and Technology of China(中国科学技术大学);Shanghai Jiao Tong University(上海交通大学);Shanghai AI Laboratory(上海人工智能实验室);City University of Hong Kong(香港城市大学);The Chinese University of Hong Kong(香港中文大学);Fudan University(复旦大学);The University of Sydney(悉尼大学);Beihang University(北航)
Cognitive Dual-Process Planning for Autonomous Driving with Structured Scene Knowledge and Verifiable Reasoning-Action Consistency
School of Mechanical Engineering, Beijing Institute of Technology(北京理工大学机械工程学院);National Engineering Research Center of Electric Vehicles, Beijing Institute of Technology(北京理工大学电动车辆国家工程研究中心);School of Mechanical Engineering, Southeast University(东南大学机械工程学院)
Structured Scene Memory for Vision-Language Navigation
Beijing Institute of Technology(北京理工大学);ETH Zurich(苏黎世联邦理工学院);Salesforce Research(Salesforce研究院);Inception Institute of Artificial Intelligence(Inception人工智能研究院)
Hierarchically-Structured Open-Vocabulary Indoor Scene Synthesis with Pre-trained Large Language Model
School of Software, Shandong University(山东大学软件学院);School of Computer Science, National University of Defense Technology(国防科技大学计算机学院);Shandong Research Institute of Industrial Technology(山东省产业技术研究院)
SceneTransporter: Optimal Transport-Guided Compositional Latent Diffusion for Single-Image Structured 3D Scene Generation
Xi’an Research Institute of Hi-Tech(西安高新技术研究院);Tsinghua University(清华大学);SkyWork AI;Tongji University(同济大学);Fudan University(复旦大学);Beijing Normal University(北京师范大学)
From Marginal to Joint Predictions: Evaluating Scene-Consistent Trajectory Prediction Approaches for Automated Driving
Pre-Development of Automated Driving, CARIAD SE(自动化驾驶前期开发,CARIAD SE);Technical University of Munich (TUM)(慕尼黑技术大学);Karlsruhe Institute of Technology (KIT)(卡尔斯鲁厄理工学院)
MetaScenes: Towards Automated Replica Creation for Real-world 3D Scans
State Key Laboratory of General Artificial Intelligence, BIGAI(通用人工智能国家重点实验室,BIGAI);Beijing Institute of Technology(北京理工大学);Tsinghua University(清华大学);University of Science and Technology of China(中国科学技术大学)
PFSD: A Multi-Modal Pedestrian-Focus Scene Dataset for Rich Tasks in Semi-Structured Environments
School of Information Science and Technology, University of Science and Technology of China(中国科学技术大学信息科学技术学院);State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), CASIA(中国科学院自动化研究所多模态人工智能系统国家重点实验室);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)
CUS-GS: A Compact Unified Structured Gaussian Splatting Framework for Multimodal Scene Representation
School of Computer Science, Hangzhou Dianzi University(杭州电子科技大学计算机科学学院);CAD & CG, Zhejiang University(浙江大学计算机辅助设计与图形学研究所);School of Computer Science, University of Bristol(布里斯托大学计算机科学学院)