定位自然语言描述的物体对自主智能体而言是一项重大挑战。现有基于CLIP的开放词汇方法能够利用简单（bare）查询成功执行3D物体定位，但无法应对需要理解物体关系的模糊描述。为解决该问题，我们提出了一种名为BBQ（Beyond Bare Queries）的模块化方法，该方法构建具有度量与语义空间边的3D场景图表示，并通过我们的演绎场景推理算法将大语言模型用作人机交互接口。BBQ采用鲁棒的DINO驱动关联来构建3D以物体为中心的地图，并使用先进的raycasting算法配合2D视觉-语言模型将物体描述为图节点。在Replica和ScanNet数据集上，我们证明了BBQ在开放词汇3D语义分割方面相比其他零样本方法处于领先地位。此外，我们还表明利用空间关系对于包含多个同一语义类别实体的场景尤其有效。在具有挑战性的Sr3D+、Nr3D和ScanRefer基准上，我们的演绎方法展现出显著改进，相比其他最先进方法能够通过复杂查询实现物体定位。我们的设计选择与软件实现的结合在机器人机载计算机实验中带来了显著的数据处理速度提升。这一有前景的性能使我们的方法能够应用于智能机器人项目。我们已在https://linukc.github.io/BeyondBareQueries/公开代码。
英文摘要：
Locating objects described in natural language presents a significant challenge for autonomous agents. Existing CLIP-based open-vocabulary methods successfully perform 3D object grounding with simple (bare) queries, but cannot cope with ambiguous descriptions that demand an understanding of object relations. To tackle this problem, we propose a modular approach called BBQ (Beyond Bare Queries), which constructs 3D scene graph representation with metric and semantic spatial edges and utilizes a large language model as a human-to-agent interface through our deductive scene reasoning algorithm. BBQ employs robust DINO-powered associations to construct 3D object-centric map and an advanced raycasting algorithm with a 2D vision-language model to describe them as graph nodes. On the Replica and ScanNet datasets, we have demonstrated that BBQ takes a leading place in open-vocabulary 3D semantic segmentation compared to other zero-shot methods. Also, we show that leveraging spatial relations is especially effective for scenes containing multiple entities of the same semantic class. On challenging Sr3D+, Nr3D and ScanRefer benchmarks, our deductive approach demonstrates a significant improvement, enabling objects grounding by complex queries compared to other state-of-the-art methods. The combination of our design choices and software implementation has resulted in significant data processing speed in experiments on the robot on-board computer. This promising performance enables the application of our approach in intelligent robotics projects. We made the code publicly available at https://linukc.github.io/BeyondBareQueries/.
BARE: Towards Bias-Aware and Reasoning-Enhanced One-Tower Visual Grounding
School of Artificial Intelligence, Beijing University of Posts and Telecommunications（北京邮电大学人工智能学院）；Pengcheng Laboratory（鹏城实验室）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
Visual Programming for Zero-shot Open-Vocabulary 3D Visual Grounding
The Future Network of Intelligence Institute（未来智能网络研究院）；The Chinese University of Hong Kong (Shenzhen)（香港中文大学（深圳））；School of Science and Engineering, The Chinese University of Hong Kong (Shenzhen)（香港中文大学（深圳）理工学院）；IHPC, A*STAR, Singapore（新加坡科研局智能系统研究所）；The University of Hong Kong（香港大学）
Training-Free Open-Vocabulary Visual Grounding for Remote Sensing Images and Videos
School of Computer Science and Technology, Xidian University（西安电子科技大学计算机科学与技术学院）；Interdisciplinary Institute of Artificial Intelligence, Xidian University（西安电子科技大学跨学科人工智能研究院）；School of Artificial Intelligence, Xidian University（西安电子科技大学人工智能学院）；Northwest Institute of Nuclear Technology（西北核技术研究所）；School of Physics and Information Engineering, Fuzhou University（福州大学物理与信息工程学院）
Context Disentangling and Prototype Inheriting for Robust Visual Grounding
Nanjing University of Science and Technology（南京理工大学）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；SenseTime Research（商汤科技研究院）
Answer grounding aims to reveal the visual evidence for visual question answering (VQA), which entails highlighting relevant positions in the image when answering questions about images. Previous attempts typically tackle this problem using pretrained object detectors, but without the flexibility for objects not in the predefined vocabulary. However, these black-box methods solely concentrate on the linguistic generation, ignoring the visual interpretability. In this paper, we propose Dual Visual-Linguistic Interaction (DaVI), a novel unified end-to-end framework with the capability for both l…
Large language models (LLMs) have unlocked new capabilities of task planning from human instructions. However, prior attempts to apply LLMs to real-world robotic tasks are limited by the lack of grounding in the surrounding scene. In this paper, we develop NLMap, an open-vocabulary and queryable scene representation to address this problem. NLMap serves as a framework to gather and integrate contextual information into LLM planners, allowing them to see and query available objects in the scene before generating a context-conditioned plan. NLMap first establishes a natural language queryable sc…
Reasoning3D -- Grounding and Reasoning in 3D: Fine-Grained Zero-Shot Open-Vocabulary 3D Reasoning Part Segmentation via Large Vision-Language Models
KOKONI, Moxin (Huzhou) Tech. Co., LTD.（杭州魔芯科技有限公司）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；School of Information Engineering, Huzhou University（湖州师范学院信息工程学院）；Singapore University of Technology and Design（新加坡科技设计大学）；School of Information Science and Technology, University of Science and Technology of China（中国科学技术大学信息科学技术学院）；School of Software Technology, Zhejiang University（浙江大学软件学院）
Video-GroundingDINO: Towards Open-Vocabulary Spatio-Temporal Video Grounding
Mohamed bin Zayed University of AI（穆罕默德·本·扎耶德人工智能大学）；Australian National University（澳大利亚国立大学）；University of California, Merced（加州大学默塞德分校）；Google Research（谷歌研究院）；Linköping University（林雪平大学）
Grounding Everything in Tokens for Multimodal Large Language Models
MoE Key Lab of Artificial Intelligence（人工智能混合专家模型关键实验室）；AI Institute, Shanghai Jiao Tong University（上海交通大学人工智能研究院）；Central Research Institute, Huawei（华为中央研究院）
From Gaze to Meaning: A Training-Free AI Agent for Unified Grounding and Explanation
Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey（萨里大学视觉、语音与信号处理中心）；Surrey Institute for People-Centred AI, University of Surrey（萨里大学以人为本人工智能研究所）
Solving Zero-Shot 3D Visual Grounding as Constraint Satisfaction Problems
Bernoulli Institute for Mathematics, Computer Science and Artificial Intelligence, University of Groningen（格罗宁根大学伯努利学院）；Computer Vision for Human-Computer Interaction Lab (cv:hci), Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院人机交互计算机视觉实验室）
HERO: Hierarchical Embedding-Refinement for Open-Vocabulary Temporal Sentence Grounding in Videos
Laboratory of Complex Systems Modeling and Simulation, School of Computer Science and Technology, Hangzhou Dianzi University（电子科技大学复杂系统建模与仿真实验室，计算机科学与技术学院）；Zhejiang Key Laboratory of Space Information Sensing and Transmission, Hangzhou Dianzi University（浙江省空间信息感知与传输重点实验室，电子科技大学）；Department of Psychological and Cognitive Sciences, Tsinghua University（清华大学心理与认知科学系）
Why Does Grounding Hurt Medical VQA? Benchmarking, Diagnosis, and Fine-Tuning of Vision-Language Models
New York University, New York, USA（纽约大学）；Tsinghua University, Beijing, China（清华大学）；Columbia University, New York, USA（哥伦比亚大学）；University of Michigan, Ann Arbor, USA（密歇根大学）
Multiple Consistent 2D-3D Mappings for Robust Zero-Shot 3D Visual Grounding
Hangzhou Dianzi University（杭州电子科技大学）；Zhejiang University（浙江大学）；Alibaba Group（阿里巴巴集团）；Institute for Clarity in Documentation（文档清晰研究所）；Inria Paris-Rocquencourt（Inria巴黎-鲁克蒙特研究所）；Rajiv Gandhi University（拉贾·甘地大学）；Tsinghua University（清华大学）；Palmer Research Laboratories（帕勒实验室）
Uncertainty-aware Medical Diagnostic Phrase Identification and Grounding
College of Computer Science, Sichuan University（四川大学计算机学院）；Institute of High Performance Computing (IHPC), Agency for Science, Technology and Research (A*STAR)（新加坡科技研究局高性能计算研究所）；Department of Computing, The Hong Kong Polytechnic University（香港理工大学计算机系）；Department of Radiology, West China Hospital, Sichuan University（四川大学华西医院放射科）；College of Intelligence and Computing, Tianjin University（天津大学智能与计算学院）；Department of Mathematics, Sichuan University（四川大学数学系）；School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-Sen University（中山大学深圳校区网络科学与技术学院）；Department of Ophthalmology, Yong Loo Lin School of Medicine, National University of Singapore and the Singapore Eye Research Institute, Singapore National Eye Centre（新加坡国立大学 Yong Loo Lin 医学院眼科系及新加坡眼科研所、新加坡国家眼科中心）
