Traditional object detection systems are typically constrained to predefined categories, limiting their applicability in dynamic environments. In contrast, open-vocabulary object detection (OVD) enables the identification of objects from novel classes not present in the training set. Recent advances in visual-language modeling have led to significant progress of OVD. However, prior works face challenges in either adapting the single-scale image backbone from CLIP to the detection framework or ensuring robust visual-language alignment. We propose Visual-Language Detection (VLDet), a novel framework that revamps feature pyramid for fine-grained visual-language alignment, leading to improved OVD performance. With the VL-PUB module, VLDet effectively exploits the visual-language knowledge from CLIP and adapts the backbone for object detection through feature pyramid. In addition, we introduce the SigRPN block, which incorporates a sigmoid-based anchor-text contrastive alignment loss to improve detection of novel categories. Through extensive experiments, our approach achieves 58.7 AP for novel classes on COCO2017 and 24.8 AP on LVIS, surpassing all state-of-the-art methods and achieving significant improvements of 27.6% and 6.9%, respectively. Furthermore, VLDet also demonstrates superior zero-shot performance on closed-set object detection.
FGAseg: Fine-Grained Pixel-Text Alignment for Open-Vocabulary Semantic Segmentation
University of Science and Technology of China（中国科学技术大学）；Institute of Artificial Intelligence (TeleAI), China Telecom（中国电信人工智能研究院（TeleAI））；School of Artificial Intelligence, OPtics and ElectroNics (iOPEN), Northwestern Polytechnical University（西北工业大学人工智能、光学与电子学院（iOPEN））
GUIDED: Granular Understanding via Identification, Detection, and Discrimination for Fine-Grained Open-Vocabulary Object Detection
School of Computer Science and Engineering, Sun Yat-sen University（中山大学计算机科学与工程学院）；Meituan（美团）；GuangDong Province Key Laboratory of Information Security Technology（广东省信息安全技术重点实验室）；Research Institute, Sun Yat-sen University（中山大学深圳研究院）
MQADet: A Plug-and-Play Paradigm for Enhancing Open-Vocabulary Object Detection via Multimodal Question Answering
School of Computer and Information Science, Qinghai Institute of Technology（青海理工学院计算机与信息科学学院）；School of Information Science and Technology, Harbin Institute of Technology (Shen Zhen)（哈尔滨工业大学（深圳）信息科学与技术学院）；School of Computer Science and Technology, Qinghai University（青海大学计算机科学与技术学院）；Eryuan Digital Technology Co., Ltd.（洱源数字科技有限公司）
DSAA: Dual-Stage Attribute Activation for Fine-grained Open Vocabulary Detection
Beijing University of Posts and Telecommunications（北京邮电大学）；Beijing E-Hualu Information Technology Co., Ltd.（北京亿华鲁信息技术有限公司）；State Key Laboratory of General Artificial Intelligence, BIGAI, Beijing, China（通用人工智能国家重点实验室，BIGAI，北京，中国）
Decomposed Vision-Language Alignment for Fine-Grained Open-Vocabulary Segmentation
Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院航空航天信息研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Zhejiang University（浙江大学）
State University of New York at Buffalo（纽约州立大学布法罗分校）；Tianjin University（天津大学）；Horizon Robotics, Inc.（地平线机器人科技有限公司）
Despite the previous success of object analysis, detecting and segmenting a large number of object categories with a long-tailed data distribution remains a challenging problem and is less investigated. For a large-vocabulary classifier, the chance of obtaining noisy logits is much higher, which can easily lead to a wrong recognition. In this paper, we exploit prior knowledge of the relations among object categories to cluster fine-grained classes into coarser parent classes, and construct a classification tree that is responsible for parsing an object instance into a fine-grained category via…
Dynamic-DINO: Fine-Grained Mixture of Experts Tuning for Real-time Open-Vocabulary Object Detection
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；Polytechnic Institute, Zhejiang University（浙江大学 polytechnic 院）；ZTE（ZTE 公司）
SAM-MI: A Mask-Injected Framework for Enhancing Open-Vocabulary Semantic Segmentation with SAM
State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS)（多模态人工智能系统国家重点实验室）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
Monash University（莫纳什大学）；ByteDance（字节跳动）；The University of Hong Kong（香港大学）
Existing object detection methods are bounded in a fixed-set vocabulary by costly labeled data. When dealing with novel categories, the model has to be retrained with more bounding box annotations. Natural language supervision is an attractive alternative for its annotation-free attributes and broader object concepts. However, learning open-vocabulary object detection from language is challenging since image-text pairs do not contain fine-grained object-language alignments. Previous solutions rely on either expensive grounding annotations or distilling classification-oriented vision models. In…
Renovating Names in Open-Vocabulary Segmentation Benchmarks
Bosch IoC Lab（博世物联网计算实验室）；University of Tübingen（蒂宾根大学）；Tübingen AI Center（蒂宾根人工智能中心）；Autonomous Vision Group, University of Tübingen（蒂宾根大学自主视觉组）；ETH Zurich（苏黎世联邦理工学院）；MPI for Intelligent Systems, Tübingen（蒂宾根智能系统马克斯·普朗克研究所）；Bosch Center for Artificial Intelligence（博世人工智能中心）
ExpAlign: Expectation-Guided Vision-Language Alignment for Open-Vocabulary Grounding
Department of Automation, Tsinghua University, China（清华大学自动化系）；University of Electronic Science and Technology of China, China（电子科技大学）；University of Copenhagen, Denmark（哥本哈根大学）；Lingsu Lab, China（灵素实验室）
O2V-Mapping: Online Open-Vocabulary Mapping with Neural Implicit Representation
Academy for Engineering & Technology（工程与技术学院）；Fudan University（复旦大学）；State Key Laboratory of Intelligent Vehicle Safety Technology（智能车辆安全技术国家重点实验室）；School of Future Technology（未来技术学院）；Harbin Institute of Technology（哈尔滨工业大学）；Department of Computer Science and Engineering（计算机科学与工程系）；Shanghai Jiao Tong University（上海交通大学）
ExDet: Open-Domain Open-Vocabulary Detection with Cross-modal Extrapolation and Rectification
College of Intelligence and Computing, Tianjin University（天津大学智能与计算学部）；Faculty of Computer Science and Artificial Intelligence, Shenzhen University of Advanced Technology（深圳理工大学计算机科学与人工智能学院）；School of Artificial Intelligence, Nanchang University（南昌大学人工智能学院）
Recent open-vocabulary detection methods aim to detect novel objects by distilling knowledge from vision-language models (VLMs) trained on a vast amount of image-text pairs. To improve the effectiveness of these methods, researchers have utilized datasets with a large vocabulary that contains a large number of object classes, under the assumption that such data will enable models to extract comprehensive knowledge on the relationships between various objects and better generalize to unseen object classes. In this study, we argue that more fine-grained labels are necessary to extract richer kno…
MARIS: Marine Open-Vocabulary Instance Segmentation with Geometric Enhancement and Semantic Alignment
University of Science and Technology of China（中国科学技术大学）；Institute of Artificial Intelligence (TeleAI)（人工智能研究院（TeleAI））；China Telecom（中国电信）；Fudan University（复旦大学）；Northwestern Polytechnical University（西北工业大学）
