Learning Background Prompts to Discover Implicit Knowledge for Open Vocabulary Object Detection
School of Computer Science and Engineering, Sun Yat-sen University（中山大学计算机科学与工程学院）；The University of Hong Kong（香港大学）；SECE, Shenzhen Graduate School, Peking University（北京大学深圳研究生院电子与计算机工程学院）；Institute of Artificial Intelligence, Beihang University（北京航空航天大学人工智能研究院）；Research Institute, Sun Yat-sen University, Shenzhen（中山大学深圳研究院）
Jiaming Li, Jiacheng Zhang, Jichang Li, Ge Li, Si Liu, Liang Lin, Guanbin Li
Open vocabulary object detection (OVD) aims at seeking an optimal object detector capable of recognizing objects from both base and novel categories. Recent advances leverage knowledge distillation to transfer insightful knowledge from pre-trained large-scale vision-language models to the task of object detection, significantly generalizing the powerful capabilities of the detector to identify more unknown object categories. However, these methods face significant challenges in background interpretation and model overfitting and thus often result in the loss of crucial background knowledge, giving rise to sub-optimal inference performance of the detector. To mitigate these issues, we present a novel OVD framework termed LBP to propose learning background prompts to harness explored implicit background knowledge, thus enhancing the detection performance w.r.t. base and novel categories. Specifically, we devise three modules: Background Category-specific Prompt, Background Object Discovery, and Inference Probability Rectification, to empower the detector to discover, represent, and leverage implicit object knowledge explored from background proposals. Evaluation on two benchmark datasets, OV-COCO and OV-LVIS, demonstrates the superiority of our proposed method over existing state-of-the-art approaches in handling the OVD tasks.
O2V-Mapping: Online Open-Vocabulary Mapping with Neural Implicit Representation
Academy for Engineering & Technology（工程与技术学院）；Fudan University（复旦大学）；State Key Laboratory of Intelligent Vehicle Safety Technology（智能车辆安全技术国家重点实验室）；School of Future Technology（未来技术学院）；Harbin Institute of Technology（哈尔滨工业大学）；Department of Computer Science and Engineering（计算机科学与工程系）；Shanghai Jiao Tong University（上海交通大学）
University of Pittsburgh（匹兹堡大学）；Google Research（谷歌研究院）
To alleviate the cost of obtaining accurate bounding boxes for training today's state-of-the-art object detection models, recent weakly supervised detection work has proposed techniques to learn from image-level labels. However, requiring discrete image-level labels is both restrictive and suboptimal. Real-world "supervision" usually consists of more unstructured text, such as captions. In this work we learn association maps between images and captions. We then use a novel objectness criterion to rank the resulting candidate boxes, such that high-ranking boxes have strong gradients along all e…
NoOVD: Novel Category Discovery and Embedding for Open-Vocabulary Object Detection
College of Intelligence and Computing, Tianjin University（天津大学智能与计算学院）；Key Research Center for Surface Monitoring and Analysis of Relics, State Administration of Cultural Heritage（文物表面监测与分析国家重点研究中心）；Faculty of Computer Science and Artificial Intelligence, Shenzhen University of Advanced Technology（深圳先进技术大学计算机科学与人工智能学院）；School of Artificial Intelligence, Nanchang University（南昌大学人工智能学院）
Nanyang Technological University（南洋理工大学）；The University of Hong Kong（香港大学）；SenseTime Research and Tetras.AI（商汤研究院与Tetras.AI）；Shanghai AI Laboratory（上海人工智能实验室）
Pre-trained vision-language models (VLMs) learn to align vision and language representations on large-scale datasets, where each image-text pair usually contains a bag of semantic concepts. However, existing open-vocabulary object detectors only align region embeddings individually with the corresponding features extracted from the VLMs. Such a design leaves the compositional structure of semantic concepts in a scene under-exploited, although the structure may be implicitly learned by the VLMs. In this work, we propose to align the embedding of bag of regions beyond individual regions. The pro…
Open-Vocabulary 3D Object Detection with Co-Distillation Discovery and Dual Guidance Robust Training
School of Computer Science and Engineering, University of Electronic Science and Technology of China（电子科技大学计算机科学与工程学院）；Singapore University of Technology and Design（新加坡科技设计大学）；School of Computer Science and Technology, Hainan University（海南大学计算机科学与技术学院）
Towards Domain-Generalized Open-Vocabulary Object Detection: A Progressive Domain-invariant Cross-modal Alignment Method
MAIS, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所多模态人工智能系统实验室）；Institute of Microelectronics, University of Chinese Academy of Sciences（中国科学院大学微电子学院）；School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences（中国科学院大学前沿交叉科学学院）
ExpAlign: Expectation-Guided Vision-Language Alignment for Open-Vocabulary Grounding
Department of Automation, Tsinghua University, China（清华大学自动化系）；University of Electronic Science and Technology of China, China（电子科技大学）；University of Copenhagen, Denmark（哥本哈根大学）；Lingsu Lab, China（灵素实验室）
Unveiling the Unknown: Open Vocabulary Object Detection with Scene Graphs
Faculty of Electrical Engineering and Computer Science, Ningbo University（宁波大学电气工程与计算机科学学院）；Faculty of Computing, Georg-August-Universität Göttingen（哥廷根大学计算机学院）；Merchants’ Guild Economics and Cultural Intelligent Computing Laboratory, Ningbo University（宁波大学商帮经济与文化智能计算实验室）；School of Software Technology, Zhejiang University（浙江大学软件学院）
OV-Uni3DETR: Towards Unified Open-Vocabulary 3D Object Detection via Cycle-Modality Propagation
Tsinghua University（清华大学）；Rutgers University（罗格斯大学）；Beijing National Research Center for Information Science and Technology (BNRist)（北京信息科学与技术国家研究中心）；The University of Hong Kong（香港大学）
OVS-DINO: Open-Vocabulary Segmentation via Structure-Aligned SAM-DINO with Language Guidance
Tongji University（同济大学）；Hong Kong Polytechnic University（香港理工大学）；University of Electronic Science and Technology of China（电子科技大学）；Chongqing University（重庆大学）
We discover that DALLE-2 seems to have a hidden vocabulary that can be used to generate images with absurd prompts. For example, it seems that \texttt{Apoploe vesrreaitais} means birds and \texttt{Contarra ccetnxniams luryca tanniounons} (sometimes) means bugs or pests. We find that these prompts are often consistent in isolation but also sometimes in combinations. We present our black-box method to discover words that seem random but have some correspondence to visual concepts. This creates important security and interpretability challenges.
Swiss Federal Institute of Technology in Zürich（苏黎世联邦理工学院）；The University of Queensland（昆士兰大学）
Recently, groundbreaking results have been presented on open-vocabulary semantic image segmentation. Such methods segment each pixel in an image into arbitrary categories provided at run-time in the form of text prompts, as opposed to a fixed set of classes defined at training time. In this work, we present a zero-shot volumetric open-vocabulary semantic scene segmentation method. Our method builds on the insight that we can fuse image features from a vision-language model into a neural implicit representation. We show that the resulting feature field can be segmented into different classes by…
Domain Adaptation for Large-Vocabulary Object Detectors
Xidian University（西安电子科技大学）；Nanyang Technological University（南洋理工大学）；University of Technology Sydney（悉尼科技大学）；University of Chinese Academy of Sciences（中国科学院大学）
In Open Set Domain Adaptation (OSDA), large amounts of target samples are drawn from the implicit categories that never appear in the source domain. Due to the lack of their specific belonging, existing methods indiscriminately regard them as a single class unknown. We challenge this broadly-adopted practice that may arouse unexpected detrimental effects because the decision boundaries between the implicit categories have been fully ignored. Instead, we propose Self-supervised Class-Discovering Adapter (SCDA) that attempts to achieve OSDA by gradually discovering those implicit classes, then i…
We propose an approach for annotating object classes using free-form text written by undirected and untrained annotators. Free-form labeling is natural for annotators, they intuitively provide very specific and exhaustive labels, and no training stage is necessary. We first collect 729 labels on 15k images using 124 different annotators. Then we automatically enrich the structure of these free-form annotations by discovering a natural vocabulary of 4020 classes within them. This vocabulary represents the natural distribution of objects well and is learned directly from data, instead of being a…
Attribute-based recognition models, due to their impressive performance and their ability to generalize well on novel categories, have been widely adopted for many computer vision applications. However, usually both the attribute vocabulary and the class-attribute associations have to be provided manually by domain experts or large number of annotators. This is very costly and not necessarily optimal regarding recognition performance, and most importantly, it limits the applicability of attribute-based models to large scale data sets. To tackle this problem, we propose an end-to-end unsupervis…
End-to-end Open-vocabulary Video Visual Relationship Detection using Multi-modal Prompting
Beijing Institute of Technology（北京理工大学）；School of Computer Science and Technology, Beijing Institute of Technology（北京理工大学计算机科学与技术学院）；Shenzhen MSU-BIT University（深圳北理莫斯科大学）；University of Rochester（罗切斯特大学）
