In recent years, 2D Vision-Language Models (VLMs) have made significant strides in image-text understanding tasks. However, their performance in 3D spatial comprehension, which is critical for embodied intelligence, remains limited. Recent advances have leveraged 3D point clouds and multi-view images as inputs, yielding promising results. However, we propose exploring a purely vision-based solution inspired by human perception, which merely relies on visual cues for 3D spatial understanding. This paper empirically investigates the limitations of VLMs in 3D spatial knowledge, revealing that their primary shortcoming lies in the lack of global-local correspondence between the scene and individual frames. To address this, we introduce GPT4Scene, a novel visual prompting paradigm in VLM training and inference that helps build the global-local relationship, significantly improving the 3D spatial understanding of indoor scenes. Specifically, GPT4Scene constructs a Bird's Eye View (BEV) image from the video and marks consistent object IDs across both frames and the BEV image. The model then inputs the concatenated BEV image and video frames with markers. In zero-shot evaluations, GPT4Scene improves performance over closed-source VLMs like GPT-4o. Additionally, we prepare a processed video dataset consisting of 165K text annotation to fine-tune open-source VLMs, achieving state-of-the-art performance on all 3D understanding tasks. Surprisingly, after training with the GPT4Scene paradigm, VLMs consistently improve during inference, even without object marker prompting and BEV image as explicit correspondence. It demonstrates that the proposed paradigm helps VLMs develop an intrinsic ability to understand 3D scenes, which paves the way for a seamless approach to extending pre-trained VLMs for 3D scene understanding.
Vision-Language Pre-training with Object Contrastive Learning for 3D Scene Understanding
Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；College of Computer Science and Software Engineering, Shenzhen University（深圳大学计算机与软件学院）；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；Research Center of Artificial Intelligence, Peng Cheng Laboratory（鹏城实验室人工智能研究中心）
From Pixels to Graphs: Open-Vocabulary Scene Graph Generation with Vision-Language Models
School of Information Science and Technology, ShanghaiTech University（上海科技大学信息科学与技术学院）；Shanghai AI Laboratory（上海人工智能实验室）；Shanghai Engineering Research Center of Intelligent Vision and Imaging（上海智能视觉与成像工程技术研究中心）
LEO-VL: Efficient Scene Representation for Scalable 3D Vision-Language Learning
Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）；State Key Laboratory of General Artificial Intelligence, BIGAI（通用人工智能国家重点实验室，BIGAI）；School of Artificial Intelligence, Beijing University of Posts and Telecommunications（北京邮电大学人工智能学院）；Department of Automation, Tsinghua University（清华大学自动化系）
HMR3D: Hierarchical Multimodal Representation for 3D Scene Understanding with Large Vision-Language Model
Institute of High-Performance Computing, Agency for Science, Technology and Research, Singapore（高性能计算研究所，科技研究局，新加坡）；Centre for Frontier AI Research, Agency for Science, Technology and Research, Singapore（前沿人工智能研究中心，科技研究局，新加坡）；College of Computing and Data Science, Nanyang Technological University, Singapore（计算与数据科学学院，南洋理工大学，新加坡）
CL4D: Contrastive Language-4D Pretraining for Vision-Language Reasoning in Dynamic Scenes
University of Moratuwa（莫拉图瓦大学）；Singapore-MIT Alliance for Research & Technology (SMART) Centre（新加坡-麻省理工研究与技术联盟（SMART）中心）；Agency for Science, Technology and Research (A*STAR)（新加坡科学、技术与研究局（A*STAR））
University of Chinese Academy of Sciences（中国科学院大学）
Answering questions that require reading texts in an image is challenging for current models. One key difficulty of this task is that rare, polysemous, and ambiguous words frequently appear in images, e.g., names of places, products, and sports teams. To overcome this difficulty, only resorting to pre-trained word embedding models is far from enough. A desired model should utilize the rich information in multiple modalities of the image to help understand the meaning of scene texts, e.g., the prominent text on a bottle is most likely to be the brand. Following this idea, we propose a novel VQA…
SceneTAP: Scene-Coherent Typographic Adversarial Planner against Vision-Language Models in Real-World Environments
Agency for Science, Technology and Research (A*STAR)（新加坡科技研究局）；College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算与数据科学学院）；University of Alberta（阿尔伯塔大学）；Tianjin University（天津大学）
MVL-Loc: Leveraging Vision-Language Model for Generalizable Multi-Scene Camera Relocalization
School of Automation Science and Engineering, South China University of Technology（自动化科学与工程学院，华南理工大学）；Thrust of Artificial Intelligence, The Hong Kong University of Science and Technology (Guangzhou)（人工智能方向，香港科学与技术大学（广州））
We study open-world 3D scene understanding, a family of tasks that require agents to reason about their 3D environment with an open-set vocabulary and out-of-domain visual inputs - a critical skill for robots to operate in the unstructured 3D world. Towards this end, we propose Semantic Abstraction (SemAbs), a framework that equips 2D Vision-Language Models (VLMs) with new 3D spatial capabilities, while maintaining their zero-shot robustness. We achieve this abstraction using relevancy maps extracted from CLIP, and learn 3D spatial and geometric reasoning skills on top of those abstractions in…
RE-VLM: Event-Augmented Vision-Language Model for Scene Understanding
School of Artificial Intelligence, Beijing University of Posts and Telecommunications（北京邮电大学人工智能学院）；State Key Laboratory of General Artificial Intelligence, BIGAI（通用人工智能国家重点实验室，BIGAI）
Toward Autonomous Laboratory Safety Monitoring with Vision Language Models: Learning to See Hazards Through Scene Structure
Electrical and Computer Engineering, University of California, Riverside, USA（加州大学河滨分校电气与计算机工程系）；Computer Science and Engineering, University of California, Riverside, USA（加州大学河滨分校计算机科学与工程系）；Lawrence Livermore National Laboratory（劳伦斯利弗莫尔国家实验室）
Grounding language to visual relations is critical to various language-and-vision applications. In this work, we tackle two fundamental language-and-vision tasks: image-text matching and image captioning, and demonstrate that neural scene graph generators can learn effective visual relation features to facilitate grounding language to visual relations and subsequently improve the two end applications. By combining relation features with the state-of-the-art models, our experiments show significant improvement on the standard Flickr30K and MSCOCO benchmarks. Our experimental results and analysi…
Search-based Testing of Vision Language Models for In-Car Scene Understanding
BMW Group, Technical University of Munich（宝马集团、慕尼黑技术大学）；Technical University of Munich（慕尼黑技术大学）；Technical University of Munich, fortiss GmbH（慕尼黑技术大学、fortiss GmbH）
