In this paper, we consider the problem of open-vocabulary semantic segmentation (OVS), which aims to segment objects of arbitrary classes instead of pre-defined, closed-set categories. The main contributions are as follows: First, we propose a transformer-based model for OVS, termed as OVSegmentor, which only exploits web-crawled image-text pairs for pre-training without using any mask annotations. OVSegmentor assembles the image pixels into a set of learnable group tokens via a slot-attention based binding module, and aligns the group tokens to the corresponding caption embedding. Second, we propose two proxy tasks for training, namely masked entity completion and cross-image mask consistency. The former aims to infer all masked entities in the caption given the group tokens, that enables the model to learn fine-grained alignment between visual groups and text entities. The latter enforces consistent mask predictions between images that contain shared entities, which encourages the model to learn visual invariance. Third, we construct CC4M dataset for pre-training by filtering CC12M with frequently appeared entities, which significantly improves training efficiency. Fourth, we perform zero-shot transfer on three benchmark datasets, PASCAL VOC 2012, PASCAL Context, and COCO Object. Our model achieves superior segmentation results over the state-of-the-art method by using only 3\% data (4M vs 134M) for pre-training. Code and pre-trained models will be released for future research.
Nanyang Technological University（南洋理工大学）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；University of Ottawa（渥太华大学）；Carnegie Mellon University（卡内基梅隆大学）；MBZUAI（穆罕默德·本·扎耶德人工智能大学）
We design an open-vocabulary image segmentation model to organize an image into meaningful regions indicated by arbitrary texts. Recent works (CLIP and ALIGN), despite attaining impressive open-vocabulary classification accuracy with image-level caption labels, are unable to segment visual concepts with pixels. We argue that these models miss an important step of visual grouping, which organizes pixels into groups before learning visual-semantic alignments. We propose OpenSeg to address the above issue while still making use of scalable image-level supervision of captions. First, it learns to …
USE: Universal Segment Embeddings for Open-Vocabulary Image Segmentation
Bosch Research North America（博世北美研究院）；Bosch Center for Artificial Intelligence (BCAI)（博世人工智能中心）；The Ohio State University（俄亥俄州立大学）；University of California Davis（加州大学戴维斯分校）
LLMDet: Learning Strong Open-Vocabulary Object Detectors under the Supervision of Large Language Models
School of Computer Science and Engineering, Sun Yat-sen University（中山大学计算机科学与工程学院）；Tongyi Lab, Alibaba Group（阿里巴巴集团通义实验室）；Peng Cheng Laboratory（鹏城实验室）；Pazhou Laboratory (Huangpu)（琶洲实验室（黄埔））
We introduce Patch Aligned Contrastive Learning (PACL), a modified compatibility function for CLIP's contrastive loss, intending to train an alignment between the patch tokens of the vision encoder and the CLS token of the text encoder. With such an alignment, a model can identify regions of an image corresponding to a given text input, and therefore transfer seamlessly to the task of open vocabulary semantic segmentation without requiring any segmentation annotations during training. Using pre-trained CLIP encoders with PACL, we are able to set the state-of-the-art on the task of open vocabul…
Lost in Translation? Vocabulary Alignment for Source-Free Adaptation in Open-Vocabulary Semantic Segmentation
The Good AI Lab（好AI实验室）；ETH Zurich（苏黎世联邦理工学院）；AMD Silo AI；Google（谷歌）；Technical University of Munich（慕尼黑技术大学）；INSAIT, Sofia University, St. Kliment Ohridski（INSAIT，索菲亚大学，圣克莱孟·奥赫里茨基）；University of Bologna（博洛尼亚大学）
School of Electronic Information, Wuhan University of Science and Technology（武汉科技大学电子信息学院）；State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications（北京邮电大学网络与交换技术国家重点实验室）；Oriental Space Port Research Institute（东方航天港研究院）；School of Artificial Intelligence, Wuhan University（武汉大学人工智能学院）
FreeSeg: Unified, Universal and Open-Vocabulary Image Segmentation
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；ByteDance Inc（字节跳动公司）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
The Hong Kong University of Science and Technology（香港科技大学）；International Digital Economy Academy (IDEA)（国际数字经济研究院（IDEA））；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；Tsinghua University（清华大学）；Microsoft Research at Redmond（微软雷德蒙德研究院）
We present OpenSeeD, a simple Open-vocabulary Segmentation and Detection framework that jointly learns from different segmentation and detection datasets. To bridge the gap of vocabulary and annotation granularity, we first introduce a pre-trained text encoder to encode all the visual concepts in two tasks and learn a common semantic space for them. This gives us reasonably good results compared with the counterparts trained on segmentation task only. To further reconcile them, we locate two discrepancies: $i$) task discrepancy -- segmentation requires extracting masks for both foreground obje…
Renovating Names in Open-Vocabulary Segmentation Benchmarks
Bosch IoC Lab（博世物联网计算实验室）；University of Tübingen（蒂宾根大学）；Tübingen AI Center（蒂宾根人工智能中心）；Autonomous Vision Group, University of Tübingen（蒂宾根大学自主视觉组）；ETH Zurich（苏黎世联邦理工学院）；MPI for Intelligent Systems, Tübingen（蒂宾根智能系统马克斯·普朗克研究所）；Bosch Center for Artificial Intelligence（博世人工智能中心）
Beyond-Labels: Advancing Open-Vocabulary Segmentation With Vision-Language Models
Artificial Intelligence Creative Research Lab, ETRI University of Science（人工智能创意研究实验室，ETRI大学）；Field Robotics Research Section, ETRI University of Science（机器人领域研究部，ETRI大学）；Artificial Intelligence Creative Research Lab, ETRI Daejeon, South Korea（人工智能创意研究实验室，ETRI大田，韩国）
SegEarth-OV3: Exploring SAM 3 for Open-Vocabulary Semantic Segmentation in Remote Sensing Images
School of Software Engineering, Xi’an Jiaotong University（西安交通大学软件工程学院）；School of Computer Science and Technology, Xi’an Jiaotong University（西安交通大学计算机科学与技术学院）；School of Information and Communications Engineering, Xi’an Jiaotong University（西安交通大学信息与通信工程学院）；Ministry of Education Key Lab of Intelligent Network Security, Xi’an, China（教育部智能网络安全重点实验室）；Pazhou Laboratory (Huangpu), Radarweg 29, Guangzhou, China（琶洲实验室（黄埔），雷达路29号，广州，中国）
OV-PARTS: Towards Open-Vocabulary Part Segmentation
Shanghai AI Laboratory（上海人工智能实验室）；The University of Hong Kong（香港大学）；The University of Sydney（悉尼大学）；University of Macau（澳门大学）；Texas A&M University（得克萨斯农工大学）
