Open-vocabulary object detection focusing on detecting novel categories guided by natural language. In this report, we propose Open-Vocabulary Light-Weighted Detection Transformer (OVLW-DETR), a deployment friendly open-vocabulary detector with strong performance and low latency. Building upon OVLW-DETR, we provide an end-to-end training recipe that transferring knowledge from vision-language model (VLM) to object detector with simple alignment. We align detector with the text encoder from VLM by replacing the fixed classification layer weights in detector with the class-name embeddings extracted from the text encoder. Without additional fusing module, OVLW-DETR is flexible and deployment friendly, making it easier to implement and modulate. improving the efficiency of interleaved attention computation. Experimental results demonstrate that the proposed approach is superior over existing real-time open-vocabulary detectors on standard Zero-Shot LVIS benchmark. Source code and pre-trained models are available at [https://github.com/Atten4Vis/LW-DETR].
Open-vocabulary object detection, which is concerned with the problem of detecting novel objects guided by natural language, has gained increasing attention from the community. Ideally, we would like to extend an open-vocabulary detector such that it can produce bounding box predictions based on user inputs in form of either natural language or exemplar image. This offers great flexibility and user experience for human-computer interaction. To this end, we propose a novel open-vocabulary detector based on DETR -- hence the name OV-DETR -- which, once trained, can detect any object given its cl…
DeCo-DETR: Decoupled Cognition DETR for efficient Open-Vocabulary Object Detection
Jiangsu University（江苏大学）；Brown University（布朗大学）；Nanyang Technological University（南洋理工大学）；MBZUAI；University of New South Wales（新南威尔士大学）；USC；University of Toronto（多伦多大学）；Data61 CSIRO
OV-DEIM: Real-time DETR-Style Open-Vocabulary Object Detection with GridSynthetic Augmentation
Intellindust AI Lab（Intellindust AI实验室）；Guangdong Laboratory of Artificial Intelligence（广东人工智能实验室）；College of Computer Science（计算机科学学院）；Software Engineering, Shenzhen University, China（软件工程，深圳大学，中国）；School of Information Technology, Carleton University, Canada（信息科技学院，卡尔顿大学，加拿大）；Institute for Research in Biomedicine, Bellinzona, Switzerland（生物医学研究 institute，贝尔林扎，瑞士）
OV-DQUO: Open-Vocabulary DETR with Denoising Text Query Training and Open-World Unknown Objects Supervision
Harbin Institute of Technology（哈尔滨工业大学）；Shenzhen University（深圳大学）；University of Chinese Academy of Sciences（中国科学院大学）；Harbin Institute of Technology, Chongqing Research Institute（哈尔滨工业大学重庆研究院）；CECloud Computing Technology Co., Ltd（天翼云计算技术有限公司）
Prompt-OVD is an efficient and effective framework for open-vocabulary object detection that utilizes class embeddings from CLIP as prompts, guiding the Transformer decoder to detect objects in both base and novel classes. Additionally, our novel RoI-based masked attention and RoI pruning techniques help leverage the zero-shot classification ability of the Vision Transformer-based CLIP, resulting in improved detection performance at minimal computational cost. Our experiments on the OV-COCO and OVLVIS datasets demonstrate that Prompt-OVD achieves an impressive 21.2 times faster inference speed…
From Data to Modeling: Fully Open-vocabulary Scene Graph Generation
Department of Computing, The Hong Kong Polytechnic University（香港理工大学计算机系）；CAIR, HKISI-CAS（中国科学院香港创新研究院）；MAIS, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所马克思主义学院）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
The Chinese University of Hong Kong（香港中文大学）；SenseTime Research（商汤科技研究院）；Shanghai Jiao Tong University（上海交通大学）；Qing Yuan Research Institute（清源研究院）；Centre for Perceptual and Interactive Intelligence（感知与交互智能研究中心）
Open-vocabulary detection (OVD) is an object detection task aiming at detecting objects from novel categories beyond the base categories on which the detector is trained. Recent OVD methods rely on large-scale visual-language pre-trained models, such as CLIP, for recognizing novel objects. We identify the two core obstacles that need to be tackled when incorporating these models into detector training: (1) the distribution mismatch that happens when applying a VL-model trained on whole images to region recognition tasks; (2) the difficulty of localizing objects of unseen classes. To overcome t…
Prompt Sensitivity in Vision-Language Grounding: How Small Changes in Wording Affect Object Detection
Department of Aerospace Engineering, Indian Institute of Technology Bombay（印度理工学院孟买分校航空航天工程系）；Department of Electrical Engineering, Indian Institute of Technology Bombay（印度理工学院孟买分校电气工程系）
