A sliding-window inference strategy is commonly adopted in recent training-free open-vocabulary semantic segmentation methods to overcome limitation of the CLIP in processing high-resolution images. However, this approach introduces a new challenge: each window is processed independently, leading to semantic discrepancy across windows. To address this issue, we propose Global-Local Aligned CLIP~(GLA-CLIP), a framework that facilitates comprehensive information exchange across windows. Rather than limiting attention to tokens within individual windows, GLA-CLIP extends key-value tokens to incorporate contextual cues from all windows. Nevertheless, we observe a window bias: outer-window tokens are less likely to be attended, since query features are produced through interactions within the inner window patches, thereby lacking semantic grounding beyond their local context. To mitigate this, we introduce a proxy anchor, constructed by aggregating tokens highly similar to the given query from all windows, which provides a unified semantic reference for measuring similarity across both inner- and outer-window patches. Furthermore, we propose a dynamic normalization scheme that adjusts attention strength according to object scale by dynamically scaling and thresholding the attention map to cope with small-object scenarios. Moreover, GLA-CLIP can be equipped on existing methods and broad their receptive field. Extensive experiments validate the effectiveness of GLA-CLIP in enhancing training-free open-vocabulary semantic segmentation performance. Code is available at https://github.com/2btlFe/GLA-CLIP.
Huazhong University of Science and Technology（华中科技大学）；Microsoft Research Asia（微软亚洲研究院）
This paper presents a new framework for open-vocabulary semantic segmentation with the pre-trained vision-language model, named Side Adapter Network (SAN). Our approach models the semantic segmentation task as a region recognition problem. A side network is attached to a frozen CLIP model with two branches: one for predicting mask proposals, and the other for predicting attention bias which is applied in the CLIP model to recognize the class of masks. This decoupled design has the benefit CLIP in recognizing the class of mask proposals. Since the attached side network can reuse CLIP features, …
The Chinese University of Hong Kong（香港中文大学）；SenseTime Research（商汤科技研究院）；Shanghai Jiao Tong University（上海交通大学）；Qing Yuan Research Institute（清源研究院）；Centre for Perceptual and Interactive Intelligence（感知与交互智能研究中心）
Open-vocabulary detection (OVD) is an object detection task aiming at detecting objects from novel categories beyond the base categories on which the detector is trained. Recent OVD methods rely on large-scale visual-language pre-trained models, such as CLIP, for recognizing novel objects. We identify the two core obstacles that need to be tackled when incorporating these models into detector training: (1) the distribution mismatch that happens when applying a VL-model trained on whole images to region recognition tasks; (2) the difficulty of localizing objects of unseen classes. To overcome t…
CLIP-VIS: Adapting CLIP for Open-Vocabulary Video Instance Segmentation
School of Electrical and Information Engineering, Tianjin University（天津大学电气与信息工程学院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；School of Big Data and Software Engineering, Chongqing University（重庆大学大数据与软件学院）
CLIPSelf: Vision Transformer Distills Itself for Open-Vocabulary Dense Prediction
Nanyang Technological University（南洋理工大学）；The Chinese University of Hong Kong（香港中文大学）；The University of Hong Kong（香港大学）；SenseTime Research and Tetras.AI（商汤科技研究院与Tetras.AI）；Shanghai AI Laboratory（上海人工智能实验室）
针对 CLIP ViT 从全局图像迁移到局部区域时的域偏移问题，本文提出 CLIPSelf 自蒸馏方法，在无需区域-文本对的情况下增强区域-语言对齐，并在多项开放词汇密集预测基准上达到最优性能。
We introduce Patch Aligned Contrastive Learning (PACL), a modified compatibility function for CLIP's contrastive loss, intending to train an alignment between the patch tokens of the vision encoder and the CLS token of the text encoder. With such an alignment, a model can identify regions of an image corresponding to a given text input, and therefore transfer seamlessly to the task of open vocabulary semantic segmentation without requiring any segmentation annotations during training. Using pre-trained CLIP encoders with PACL, we are able to set the state-of-the-art on the task of open vocabul…
Australian National University（澳大利亚国立大学）；Linköping University（林雪平大学）；Mohamed bin Zayed University of AI（穆罕默德·本·扎耶德人工智能大学）
Existing open-vocabulary object detectors typically enlarge their vocabulary sizes by leveraging different forms of weak supervision. This helps generalize to novel objects at inference. Two popular forms of weak-supervision used in open-vocabulary detection (OVD) include pretrained CLIP model and image-level supervision. We note that both these modes of supervision are not optimally aligned for the detection task: CLIP is trained with image-text pairs and lacks precise localization of objects while the image-level supervision has been used with heuristics that do not accurately specify local …
Boosting Open-Vocabulary Object Detection by Handling Background Samples
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Nanjing Artificial Intelligence Research of IA（中科院南京人工智能研究院）
Prompt-OVD is an efficient and effective framework for open-vocabulary object detection that utilizes class embeddings from CLIP as prompts, guiding the Transformer decoder to detect objects in both base and novel classes. Additionally, our novel RoI-based masked attention and RoI pruning techniques help leverage the zero-shot classification ability of the Vision Transformer-based CLIP, resulting in improved detection performance at minimal computational cost. Our experiments on the OV-COCO and OVLVIS datasets demonstrate that Prompt-OVD achieves an impressive 21.2 times faster inference speed…
Collaborative Vision-Text Representation Optimizing for Open-Vocabulary Segmentation
Institute of Information Science, Beijing Jiaotong University（北京交通大学信息科学研究所）；Peng Cheng Laboratory（鹏城实验室）；Georgia Institute of Technology（佐治亚理工学院）；Picsart AI Research (PAIR)（Picsart人工智能研究院）
Open-Vocabulary Semantic Segmentation with Image Embedding Balancing
National Key Laboratory of Multispectral Information Intelligent Processing Technology（多光谱信息智能处理技术国家重点实验室）；School of Artificial Intelligence and Automation, Huazhong University of Science and Technology（华中科技大学人工智能与自动化学院）
本文提出 EBSeg 框架，通过 AdaB Decoder 自适应平衡训练类与新类图像嵌入，并结合 SSC Loss 与冻结 SAM 编码器，提升开放词汇语义分割的泛化和空间定位性能。
Mitigating Objectness Bias and Region-to-Text Misalignment for Open-Vocabulary Panoptic Segmentation
University of Ljubljana（卢布尔雅那大学）；ETH Zurich（苏黎世联邦理工学院）；University of Zagreb（扎格reb大学）；Faculty of Comp. and Inf. Science（计算机与信息科学系）；Dept. of Computer Science（计算机科学系）；Faculty of Elec. Eng. and Computing（电子工程与计算科学系）
DeCLIP: Decoupled Learning for Open-Vocabulary Dense Perception
School of Computer Science and Technology, HIT, Shenzhen（哈尔滨工业大学深圳校区计算机科学与技术学院）；International Research Institute for Artificial Intelligence, HIT, Shenzhen（哈尔滨工业大学深圳国际人工智能研究院）；University of Chinese Academy of Sciences（中国科学院大学）
Plug-in Feedback Self-adaptive Attention in CLIP for Training-free Open-Vocabulary Segmentation
University of Toronto（多伦多大学）；China Agricultural University（中国农业大学）；Concordia University（Concordia大学）；McMaster University（麦马斯特大学）；Beijing Jiaotong University（北京交通大学）
A Hierarchical Semantic Distillation Framework for Open-Vocabulary Object Detection
School of Computer Science and Engineering, Sun Yat-sen University（中山大学计算机科学与工程学院）；Peng Cheng Laboratory（鹏城实验室）；Tongyi Lab, Alibaba Group（阿里巴巴集团通义实验室）；Guangdong Key Laboratory of Information Security Technology, Sun Yat-sen University（中山大学广东省信息安全技术重点实验室）；Key Laboratory of Machine Intelligence and Advanced Computing, Sun Yat-sen University, Ministry of Education（中山大学机器智能与先进计算教育部重点实验室）
