In this paper, we address the challenge of performing open-vocabulary video instance segmentation (OV-VIS) in real-time. We analyze the computational bottlenecks of state-of-the-art foundation models that performs OV-VIS, and propose a new method, TROY-VIS, that significantly improves processing speed while maintaining high accuracy. We introduce three key techniques: (1) Decoupled Attention Feature Enhancer to speed up information interaction between different modalities and scales; (2) Flash Embedding Memory for obtaining fast text embeddings of object categories; and, (3) Kernel Interpolation for exploiting the temporal continuity in videos. Our experiments demonstrate that TROY-VIS achieves the best trade-off between accuracy and speed on two large-scale OV-VIS benchmarks, BURST and LV-VIS, running 20x faster than GLEE-Lite (25 FPS v.s. 1.25 FPS) with comparable or even better accuracy. These results demonstrate TROY-VIS's potential for real-time applications in dynamic environments such as mobile robotics and augmented reality. Code and model will be released at https://github.com/google-research/troyvis.
Nanyang Technological University（南洋理工大学）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；University of Ottawa（渥太华大学）；Carnegie Mellon University（卡内基梅隆大学）；MBZUAI（穆罕默德·本·扎耶德人工智能大学）
We design an open-vocabulary image segmentation model to organize an image into meaningful regions indicated by arbitrary texts. Recent works (CLIP and ALIGN), despite attaining impressive open-vocabulary classification accuracy with image-level caption labels, are unable to segment visual concepts with pixels. We argue that these models miss an important step of visual grouping, which organizes pixels into groups before learning visual-semantic alignments. We propose OpenSeg to address the above issue while still making use of scalable image-level supervision of captions. First, it learns to …
USE: Universal Segment Embeddings for Open-Vocabulary Image Segmentation
Bosch Research North America（博世北美研究院）；Bosch Center for Artificial Intelligence (BCAI)（博世人工智能中心）；The Ohio State University（俄亥俄州立大学）；University of California Davis（加州大学戴维斯分校）
We introduce Patch Aligned Contrastive Learning (PACL), a modified compatibility function for CLIP's contrastive loss, intending to train an alignment between the patch tokens of the vision encoder and the CLS token of the text encoder. With such an alignment, a model can identify regions of an image corresponding to a given text input, and therefore transfer seamlessly to the task of open vocabulary semantic segmentation without requiring any segmentation annotations during training. Using pre-trained CLIP encoders with PACL, we are able to set the state-of-the-art on the task of open vocabul…
Lost in Translation? Vocabulary Alignment for Source-Free Adaptation in Open-Vocabulary Semantic Segmentation
The Good AI Lab（好AI实验室）；ETH Zurich（苏黎世联邦理工学院）；AMD Silo AI；Google（谷歌）；Technical University of Munich（慕尼黑技术大学）；INSAIT, Sofia University, St. Kliment Ohridski（INSAIT，索菲亚大学，圣克莱孟·奥赫里茨基）；University of Bologna（博洛尼亚大学）
School of Electronic Information, Wuhan University of Science and Technology（武汉科技大学电子信息学院）；State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications（北京邮电大学网络与交换技术国家重点实验室）；Oriental Space Port Research Institute（东方航天港研究院）；School of Artificial Intelligence, Wuhan University（武汉大学人工智能学院）
Facebook AI Research (FAIR)（Facebook人工智能研究院（FAIR））
Progress on object detection is enabled by datasets that focus the research community's attention on open challenges. This process led us from simple images to complex scenes and from bounding boxes to segmentation masks. In this work, we introduce LVIS (pronounced `el-vis'): a new dataset for Large Vocabulary Instance Segmentation. We plan to collect ~2 million high-quality instance segmentation masks for over 1000 entry-level object categories in 164k images. Due to the Zipfian distribution of categories in natural images, LVIS naturally has a long tail of categories with few training sample…
FreeSeg: Unified, Universal and Open-Vocabulary Image Segmentation
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；ByteDance Inc（字节跳动公司）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
The Hong Kong University of Science and Technology（香港科技大学）；International Digital Economy Academy (IDEA)（国际数字经济研究院（IDEA））；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；Tsinghua University（清华大学）；Microsoft Research at Redmond（微软雷德蒙德研究院）
We present OpenSeeD, a simple Open-vocabulary Segmentation and Detection framework that jointly learns from different segmentation and detection datasets. To bridge the gap of vocabulary and annotation granularity, we first introduce a pre-trained text encoder to encode all the visual concepts in two tasks and learn a common semantic space for them. This gives us reasonably good results compared with the counterparts trained on segmentation task only. To further reconcile them, we locate two discrepancies: $i$) task discrepancy -- segmentation requires extracting masks for both foreground obje…
Renovating Names in Open-Vocabulary Segmentation Benchmarks
Bosch IoC Lab（博世物联网计算实验室）；University of Tübingen（蒂宾根大学）；Tübingen AI Center（蒂宾根人工智能中心）；Autonomous Vision Group, University of Tübingen（蒂宾根大学自主视觉组）；ETH Zurich（苏黎世联邦理工学院）；MPI for Intelligent Systems, Tübingen（蒂宾根智能系统马克斯·普朗克研究所）；Bosch Center for Artificial Intelligence（博世人工智能中心）
Recent object detection and instance segmentation tasks mainly focus on datasets with a relatively small set of categories, e.g. Pascal VOC with 20 classes and COCO with 80 classes. The new large vocabulary dataset LVIS brings new challenges to conventional methods. In this work, we propose an equalization loss to solve the long tail of rare categories problem. Combined with exploiting the data from detection datasets to alleviate the effect of missing-annotation problems during the training, our method achieves 5.1\% overall AP gain and 11.4\% AP gain of rare categories on LVIS benchmark with…
Beyond-Labels: Advancing Open-Vocabulary Segmentation With Vision-Language Models
Artificial Intelligence Creative Research Lab, ETRI University of Science（人工智能创意研究实验室，ETRI大学）；Field Robotics Research Section, ETRI University of Science（机器人领域研究部，ETRI大学）；Artificial Intelligence Creative Research Lab, ETRI Daejeon, South Korea（人工智能创意研究实验室，ETRI大田，韩国）
Instance Brownian Bridge as Texts for Open-vocabulary Video Instance Segmentation
School of Electronic and Computer Engineering, Peking University（北京大学电子与计算机工程学院）；Tsinghua University（清华大学）；Xiamen University（厦门大学）；Peng Cheng Laboratory（鹏城实验室）
本文提出 BriVIS，通过 TIR 将帧级实例查询连接为布朗桥，并利用 BTA 对比学习实现桥级实例表示与文本对齐，在 BURST 上显著优于 OV2Seg。
SegEarth-OV3: Exploring SAM 3 for Open-Vocabulary Semantic Segmentation in Remote Sensing Images
School of Software Engineering, Xi’an Jiaotong University（西安交通大学软件工程学院）；School of Computer Science and Technology, Xi’an Jiaotong University（西安交通大学计算机科学与技术学院）；School of Information and Communications Engineering, Xi’an Jiaotong University（西安交通大学信息与通信工程学院）；Ministry of Education Key Lab of Intelligent Network Security, Xi’an, China（教育部智能网络安全重点实验室）；Pazhou Laboratory (Huangpu), Radarweg 29, Guangzhou, China（琶洲实验室（黄埔），雷达路29号，广州，中国）
OV-PARTS: Towards Open-Vocabulary Part Segmentation
Shanghai AI Laboratory（上海人工智能实验室）；The University of Hong Kong（香港大学）；The University of Sydney（悉尼大学）；University of Macau（澳门大学）；Texas A&M University（得克萨斯农工大学）
