In pursuit of detecting unstinted objects that extend beyond predefined categories, prior arts of open-vocabulary object detection (OVD) typically resort to pretrained vision-language models (VLMs) for base-to-novel category generalization. However, to mitigate the misalignment between upstream image-text pretraining and downstream region-level perception, additional supervisions are indispensable, eg, image-text pairs or pseudo annotations generated via self-training strategies. In this work, we propose CCKT-Det trained without any extra supervision. The proposed framework constructs a cyclic and dynamic knowledge transfer from language queries and visual region features extracted from VLMs, which forces the detector to closely align with the visual-semantic space of VLMs. Specifically, 1) we prefilter and inject semantic priors to guide the learning of queries, and 2) introduce a regional contrastive loss to improve the awareness of queries on novel objects. CCKT-Det can consistently improve performance as the scale of VLMs increases, all while requiring the detector at a moderate level of computation overhead. Comprehensive experimental results demonstrate that our method achieves performance gain of +2.9% and +10.2% AP50 over previous state-of-the-arts on the challenging COCO benchmark, both without and with a stronger teacher model.
ExpAlign: Expectation-Guided Vision-Language Alignment for Open-Vocabulary Grounding
Department of Automation, Tsinghua University, China(清华大学自动化系);University of Electronic Science and Technology of China, China(电子科技大学);University of Copenhagen, Denmark(哥本哈根大学);Lingsu Lab, China(灵素实验室)
Instance Brownian Bridge as Texts for Open-vocabulary Video Instance Segmentation
School of Electronic and Computer Engineering, Peking University(北京大学电子与计算机工程学院);Tsinghua University(清华大学);Xiamen University(厦门大学);Peng Cheng Laboratory(鹏城实验室)
本文提出 BriVIS,通过 TIR 将帧级实例查询连接为布朗桥,并利用 BTA 对比学习实现桥级实例表示与文本对齐,在 BURST 上显著优于 OV2Seg。
CLIPSelf: Vision Transformer Distills Itself for Open-Vocabulary Dense Prediction
Nanyang Technological University(南洋理工大学);The Chinese University of Hong Kong(香港中文大学);The University of Hong Kong(香港大学);SenseTime Research and Tetras.AI(商汤科技研究院与Tetras.AI);Shanghai AI Laboratory(上海人工智能实验室)
针对 CLIP ViT 从全局图像迁移到局部区域时的域偏移问题,本文提出 CLIPSelf 自蒸馏方法,在无需区域-文本对的情况下增强区域-语言对齐,并在多项开放词汇密集预测基准上达到最优性能。
DSAA: Dual-Stage Attribute Activation for Fine-grained Open Vocabulary Detection
Beijing University of Posts and Telecommunications(北京邮电大学);Beijing E-Hualu Information Technology Co., Ltd.(北京亿华鲁信息技术有限公司);State Key Laboratory of General Artificial Intelligence, BIGAI, Beijing, China(通用人工智能国家重点实验室,BIGAI,北京,中国)
CLIP-VIS: Adapting CLIP for Open-Vocabulary Video Instance Segmentation
School of Electrical and Information Engineering, Tianjin University(天津大学电气与信息工程学院);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);School of Big Data and Software Engineering, Chongqing University(重庆大学大数据与软件学院)
Contrastive Learning-Driven Traffic Sign Perception: Multi-Modal Fusion of Text and Vision
School of Intelligent Systems Engineering, Sun Yat-sen University(中山大学智能系统工程学院);Guangdong Provincial Key Laboratory of Intelligent Transportation System(广东省智能交通系统重点实验室)
A Hierarchical Semantic Distillation Framework for Open-Vocabulary Object Detection
School of Computer Science and Engineering, Sun Yat-sen University(中山大学计算机科学与工程学院);Peng Cheng Laboratory(鹏城实验室);Tongyi Lab, Alibaba Group(阿里巴巴集团通义实验室);Guangdong Key Laboratory of Information Security Technology, Sun Yat-sen University(中山大学广东省信息安全技术重点实验室);Key Laboratory of Machine Intelligence and Advanced Computing, Sun Yat-sen University, Ministry of Education(中山大学机器智能与先进计算教育部重点实验室)
OV-DQUO: Open-Vocabulary DETR with Denoising Text Query Training and Open-World Unknown Objects Supervision
Harbin Institute of Technology(哈尔滨工业大学);Shenzhen University(深圳大学);University of Chinese Academy of Sciences(中国科学院大学);Harbin Institute of Technology, Chongqing Research Institute(哈尔滨工业大学重庆研究院);CECloud Computing Technology Co., Ltd(天翼云计算技术有限公司)
CLIM: Contrastive Language-Image Mosaic for Region Representation
Nanyang Technological University(南洋理工大学);The Chinese University of Hong Kong(香港中文大学);The University of Hong Kong(香港大学);SenseTime Research(商汤科技研究院);Shanghai AI Laboratory(上海人工智能实验室)