Open-vocabulary semantic segmentation requires models to effectively integrate visual representations with open-vocabulary semantic labels. While Contrastive Language-Image Pre-training (CLIP) models shine in recognizing visual concepts from text, they often struggle with segment coherence due to their limited localization ability. In contrast, Vision Foundation Models (VFMs) excel at acquiring spatially consistent local visual representations, yet they fall short in semantic understanding. This paper introduces ProxyCLIP, an innovative framework designed to harmonize the strengths of both CLIP and VFMs, facilitating enhanced open-vocabulary semantic segmentation. ProxyCLIP leverages the spatial feature correspondence from VFMs as a form of proxy attention to augment CLIP, thereby inheriting the VFMs' robust local consistency and maintaining CLIP's exceptional zero-shot transfer capacity. We propose an adaptive normalization and masking strategy to get the proxy attention from VFMs, allowing for adaptation across different VFMs. Remarkably, as a training-free approach, ProxyCLIP significantly improves the average mean Intersection over Union (mIoU) across eight benchmarks from 40.3 to 44.4, showcasing its exceptional efficacy in bridging the gap between spatial precision and semantic richness for the open-vocabulary segmentation task.
CLIP-VIS: Adapting CLIP for Open-Vocabulary Video Instance Segmentation
School of Electrical and Information Engineering, Tianjin University（天津大学电气与信息工程学院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；School of Big Data and Software Engineering, Chongqing University（重庆大学大数据与软件学院）
CLIPSelf: Vision Transformer Distills Itself for Open-Vocabulary Dense Prediction
Nanyang Technological University（南洋理工大学）；The Chinese University of Hong Kong（香港中文大学）；The University of Hong Kong（香港大学）；SenseTime Research and Tetras.AI（商汤科技研究院与Tetras.AI）；Shanghai AI Laboratory（上海人工智能实验室）
针对 CLIP ViT 从全局图像迁移到局部区域时的域偏移问题，本文提出 CLIPSelf 自蒸馏方法，在无需区域-文本对的情况下增强区域-语言对齐，并在多项开放词汇密集预测基准上达到最优性能。
We introduce Patch Aligned Contrastive Learning (PACL), a modified compatibility function for CLIP's contrastive loss, intending to train an alignment between the patch tokens of the vision encoder and the CLS token of the text encoder. With such an alignment, a model can identify regions of an image corresponding to a given text input, and therefore transfer seamlessly to the task of open vocabulary semantic segmentation without requiring any segmentation annotations during training. Using pre-trained CLIP encoders with PACL, we are able to set the state-of-the-art on the task of open vocabul…
Boosting Open-Vocabulary Object Detection by Handling Background Samples
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Nanjing Artificial Intelligence Research of IA（中科院南京人工智能研究院）
Prompt-OVD is an efficient and effective framework for open-vocabulary object detection that utilizes class embeddings from CLIP as prompts, guiding the Transformer decoder to detect objects in both base and novel classes. Additionally, our novel RoI-based masked attention and RoI pruning techniques help leverage the zero-shot classification ability of the Vision Transformer-based CLIP, resulting in improved detection performance at minimal computational cost. Our experiments on the OV-COCO and OVLVIS datasets demonstrate that Prompt-OVD achieves an impressive 21.2 times faster inference speed…
Collaborative Vision-Text Representation Optimizing for Open-Vocabulary Segmentation
Institute of Information Science, Beijing Jiaotong University（北京交通大学信息科学研究所）；Peng Cheng Laboratory（鹏城实验室）；Georgia Institute of Technology（佐治亚理工学院）；Picsart AI Research (PAIR)（Picsart人工智能研究院）
Huazhong University of Science and Technology（华中科技大学）；Microsoft Research Asia（微软亚洲研究院）
This paper presents a new framework for open-vocabulary semantic segmentation with the pre-trained vision-language model, named Side Adapter Network (SAN). Our approach models the semantic segmentation task as a region recognition problem. A side network is attached to a frozen CLIP model with two branches: one for predicting mask proposals, and the other for predicting attention bias which is applied in the CLIP model to recognize the class of masks. This decoupled design has the benefit CLIP in recognizing the class of mask proposals. Since the attached side network can reuse CLIP features, …
Open-Vocabulary Semantic Segmentation with Image Embedding Balancing
National Key Laboratory of Multispectral Information Intelligent Processing Technology（多光谱信息智能处理技术国家重点实验室）；School of Artificial Intelligence and Automation, Huazhong University of Science and Technology（华中科技大学人工智能与自动化学院）
本文提出 EBSeg 框架，通过 AdaB Decoder 自适应平衡训练类与新类图像嵌入，并结合 SSC Loss 与冻结 SAM 编码器，提升开放词汇语义分割的泛化和空间定位性能。
Mitigating Objectness Bias and Region-to-Text Misalignment for Open-Vocabulary Panoptic Segmentation
University of Ljubljana（卢布尔雅那大学）；ETH Zurich（苏黎世联邦理工学院）；University of Zagreb（扎格reb大学）；Faculty of Comp. and Inf. Science（计算机与信息科学系）；Dept. of Computer Science（计算机科学系）；Faculty of Elec. Eng. and Computing（电子工程与计算科学系）
The Chinese University of Hong Kong（香港中文大学）；SenseTime Research（商汤科技研究院）；Shanghai Jiao Tong University（上海交通大学）；Qing Yuan Research Institute（清源研究院）；Centre for Perceptual and Interactive Intelligence（感知与交互智能研究中心）
Open-vocabulary detection (OVD) is an object detection task aiming at detecting objects from novel categories beyond the base categories on which the detector is trained. Recent OVD methods rely on large-scale visual-language pre-trained models, such as CLIP, for recognizing novel objects. We identify the two core obstacles that need to be tackled when incorporating these models into detector training: (1) the distribution mismatch that happens when applying a VL-model trained on whole images to region recognition tasks; (2) the difficulty of localizing objects of unseen classes. To overcome t…
DeCLIP: Decoupled Learning for Open-Vocabulary Dense Perception
School of Computer Science and Technology, HIT, Shenzhen（哈尔滨工业大学深圳校区计算机科学与技术学院）；International Research Institute for Artificial Intelligence, HIT, Shenzhen（哈尔滨工业大学深圳国际人工智能研究院）；University of Chinese Academy of Sciences（中国科学院大学）
A Hierarchical Semantic Distillation Framework for Open-Vocabulary Object Detection
School of Computer Science and Engineering, Sun Yat-sen University（中山大学计算机科学与工程学院）；Peng Cheng Laboratory（鹏城实验室）；Tongyi Lab, Alibaba Group（阿里巴巴集团通义实验室）；Guangdong Key Laboratory of Information Security Technology, Sun Yat-sen University（中山大学广东省信息安全技术重点实验室）；Key Laboratory of Machine Intelligence and Advanced Computing, Sun Yat-sen University, Ministry of Education（中山大学机器智能与先进计算教育部重点实验室）
CL-CLIP: CLIP-Based Continual Learning Framework with Cost-Volume Category Decoupling for Object Detection
National College for Excellent Engineers, Beihang University（卓越工程师学院，北京航空航天大学）；AI Research, Qihoo 360（360人工智能研究院，奇虎360）；School of Electronic Information Engineering, Beihang University（电子信息学院，北京航空航天大学）；School of Cyber Science and Technology, Beihang University（网络安全科学与技术学院，北京航空航天大学）；School of Computer Science and Engineering, Beihang University（计算机科学与工程学院，北京航空航天大学）；State Key Laboratory of Media Convergence and Communication, Communication University of China（媒体融合与传播国家重点实验室，中国传媒大学）；Institute of Information Technology, Mathematics and Mechanics, Lobachebsky University（信息技术、数学与力学学院，洛瓦茨基大学）；School of Artificial Intelligence, Beihang University（人工智能学院，北京航空航天大学）
Nanyang Technological University（南洋理工大学）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；University of Ottawa（渥太华大学）；Carnegie Mellon University（卡内基梅隆大学）；MBZUAI（穆罕默德·本·扎耶德人工智能大学）
