Current approaches for 3D scene graph prediction rely on labeled datasets to train models for a fixed set of known object classes and relationship categories. We present Open3DSG, an alternative approach to learn 3D scene graph prediction in an open world without requiring labeled scene graph data. We co-embed the features from a 3D scene graph prediction backbone with the feature space of powerful open world 2D vision language foundation models. This enables us to predict 3D scene graphs from 3D point clouds in a zero-shot manner by querying object classes from an open vocabulary and predicting the inter-object relationships from a grounded LLM with scene graph features and queried object classes as context. Open3DSG is the first 3D point cloud method to predict not only explicit open-vocabulary object classes, but also open-set relationships that are not limited to a predefined label set, making it possible to express rare as well as specific objects and relationships in the predicted 3D scene graph. Our experiments show that Open3DSG is effective at predicting arbitrary object classes as well as their complex inter-object relationships describing spatial, supportive, semantic and comparative relationships.
GALA: Guided Attention with Language Alignment for Open Vocabulary Gaussian Splatting
Technical University of Munich（慕尼黑技术大学）；Universitat Politècnica de Catalunya（巴塞罗那技术大学）；Google（谷歌）；Munich Center for Machine Learning（慕尼黑机器学习中心）；University of Tubingen（图宾根大学）；ETH Zurich（苏黎世联邦理工学院）；Visualais
German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心（DFKI））；RPTU University Kaiserslautern-Landau（科布伦茨-兰道大学（RPTU））；University of Modena and Reggio Emilia（摩德纳和雷吉奥-艾米利亚大学）
Large language models (LLMs) have unlocked new capabilities of task planning from human instructions. However, prior attempts to apply LLMs to real-world robotic tasks are limited by the lack of grounding in the surrounding scene. In this paper, we develop NLMap, an open-vocabulary and queryable scene representation to address this problem. NLMap serves as a framework to gather and integrate contextual information into LLM planners, allowing them to see and query available objects in the scene before generating a context-conditioned plan. NLMap first establishes a natural language queryable sc…
Fondazione Bruno Kessler (FBK)（布鲁诺·凯斯勒基金会）；University of Trento（特伦托大学）；Queen Mary University of London（伦敦大学玛丽女王学院）；Idiap Research Institute（伊迪亚普研究所）；École polytechnique fédérale de Lausanne (EPFL)（洛桑联邦理工学院）
Fondazione Bruno Kessler（布鲁诺·凯塞勒基金会）；University of Trento（特伦托大学）；Queen Mary University of London（伦敦大学玛丽女王学院）；Idiap Research Institute（伊迪亚普研究所）；EPFL（洛桑联邦理工学院）
Scene text recognition has attracted increasing interest in recent years due to its wide range of applications in multilingual translation, autonomous driving, etc. In this report, we describe our solution to the Out of Vocabulary Scene Text Understanding (OOV-ST) Challenge, which aims to extract out-of-vocabulary (OOV) words from natural scene images. Our oCLIP-based model achieves 28.59\% in h-mean which ranks 1st in end-to-end OOV word recognition track of OOV Challenge in ECCV2022 TiE Workshop.
This report presents our 2nd place solution to ECCV 2022 challenge on Out-of-Vocabulary Scene Text Understanding (OOV-ST) : Cropped Word Recognition. This challenge is held in the context of ECCV 2022 workshop on Text in Everything (TiE), which aims to extract out-of-vocabulary words from natural scene images. In the competition, we first pre-train SCATTER on the synthetic datasets, then fine-tune the model on the training set with data augmentations. Meanwhile, two additional models are trained specifically for long and vertical texts. Finally, we combine the output from different models with…
From Pixels to Graphs: Open-Vocabulary Scene Graph Generation with Vision-Language Models
School of Information Science and Technology, ShanghaiTech University（上海科技大学信息科学与技术学院）；Shanghai AI Laboratory（上海人工智能实验室）；Shanghai Engineering Research Center of Intelligent Vision and Imaging（上海智能视觉与成像工程技术研究中心）
Open-vocabulary scene understanding aims to localize and recognize unseen categories beyond the annotated label space. The recent breakthrough of 2D open-vocabulary perception is largely driven by Internet-scale paired image-text data with rich vocabulary concepts. However, this success cannot be directly transferred to 3D scenarios due to the inaccessibility of large-scale 3D-text pairs. To this end, we propose to distill knowledge encoded in pre-trained vision-language (VL) foundation models through captioning multi-view images from 3D, which allows explicitly associating 3D and semantic-ric…
Expanding Scene Graph Boundaries: Fully Open-vocabulary Scene Graph Generation via Visual-Concept Alignment and Retention
The Hong Kong Polytechnic University（香港理工大学）；Centre for Artificial Intelligence and Robotics, HKISI-CAS（香港中文大学（深圳）人工智能与机器人中心（HKISI-CAS））；State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所 多模态人工智能系统国家重点实验室）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学 人工智能学院）
Hierarchically-Structured Open-Vocabulary Indoor Scene Synthesis with Pre-trained Large Language Model
School of Software, Shandong University（山东大学软件学院）；School of Computer Science, National University of Defense Technology（国防科技大学计算机学院）；Shandong Research Institute of Industrial Technology（山东省产业技术研究院）
From Data to Modeling: Fully Open-vocabulary Scene Graph Generation
Department of Computing, The Hong Kong Polytechnic University（香港理工大学计算机系）；CAIR, HKISI-CAS（中国科学院香港创新研究院）；MAIS, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所马克思主义学院）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
AllRead MLT；Computer Vision Center, Universitat Autonoma de Barcelona（巴塞罗那自治大学计算机视觉中心）；AWS AI Labs（亚马逊云科技人工智能实验室）
This paper presents final results of the Out-Of-Vocabulary 2022 (OOV) challenge. The OOV contest introduces an important aspect that is not commonly studied by Optical Character Recognition (OCR) models, namely, the recognition of unseen scene text instances at training time. The competition compiles a collection of public scene text datasets comprising of 326,385 images with 4,864,405 scene text instances, thus covering a wide range of data distributions. A new and independent validation and test set is formed with scene text instances that are out of vocabulary at training time. The competit…
