开放世界目标检测(OWOD)旨在解决现实场景中出现未知目标类别的问题,使训练于已知类别的检测器能够检测未知目标并逐步整合其提供的知识。尽管现有OWOD方法主要聚焦于检测未知目标,但往往忽略了检测目标之间的丰富语义关系,这对于场景理解和开放世界环境中的应用(如开放世界跟踪和新类发现)至关重要。在本文中,我们扩展OWOD框架,联合检测未知目标并学习语义丰富的实例嵌入,使检测器能够捕捉实例之间的细粒度语义关系。为此,我们提出了两个模块,利用Vision Foundation Models(VFMs)的丰富且可泛化知识,并可集成到开放世界目标检测器中。首先,未知框细化模块利用Segment Anything Model的实例掩码来准确定位未知目标。然后,嵌入转移模块通过放松对比损失将实例级语义相似性从VFM特征蒸馏到检测器的嵌入中,使检测器能够学习具有语义意义且可泛化的实例特征。大量实验表明,我们的方法显著提高了未知目标检测和实例嵌入质量,同时在下游任务如开放世界跟踪中也提升了性能。
英文摘要:
Open World Object Detection(OWOD) addresses realistic scenarios where unseen object classes emerge, enabling detectors trained on known classes to detect unknown objects and incrementally incorporate the knowledge they provide. While existing OWOD methods primarily focus on detecting unknown objects, they often overlook the rich semantic relationships between detected objects, which are essential for scene understanding and applications in open-world environments (e.g., open-world tracking and novel class discovery). In this paper, we extend the OWOD framework to jointly detect unknown objects and learn semantically rich instance embeddings, enabling the detector to capture fine-grained semantic relationships between instances. To this end, we propose two modules that leverage the rich and generalizable knowledge of Vision Foundation Models(VFMs) and can be integrated into open-world object detectors. First, the Unknown Box Refine Module uses instance masks from the Segment Anything Model to accurately localize unknown objects. The Embedding Transfer Module then distills instance-wise semantic similarities from VFM features to the detector's embeddings via a relaxed contrastive loss, enabling the detector to learn a semantically meaningful and generalizable instance feature. Extensive experiments show that our method significantly improves both unknown object detection and instance embedding quality, while also enhancing performance in downstream tasks such as open-world tracking.
Mohamed Bin Zayed University of Artificial Intelligence (MBZUAI)(穆罕默德·本·扎耶德人工智能大学(MBZUAI));Aalto University(阿尔托大学);Australian National University(澳大利亚国立大学);Linköping University(林雪平大学)
Mohamed bin Zayed University of AI(穆罕默德·本·扎耶德人工智能大学);Technology Innovation Institute(技术创新研究院);Aalto University(阿尔托大学);University of Central Florida(中佛罗里达大学);Linköping University(林雪平大学)
Unseen Object Instance Segmentation with Fully Test-time RGB-D Embeddings Adaptation
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);Nanjing Artificial Intelligence Research of IA(南京人工智能创新研究院)
FISBe: A real-world benchmark dataset for instance segmentation of long-range thin filamentous structures
Max-Delbrueck-Center for Molecular Medicine in the Helmholtz Association (MDC)(亥姆霍兹联合会马克斯·德尔布吕克分子医学中心);Helmholtz Imaging(亥姆霍兹成像中心);HHMI Janelia Research Campus(HHMI贾尼亚研究校区);University of Potsdam(波茨坦大学);German Cancer Research Center (DKFZ)(德国癌症研究中心)
OV-MAP: Open-Vocabulary Zero-Shot 3D Instance Segmentation Map for Robots
Interdisciplinary Program in AI, Seoul National University(人工智能交叉学科项目,首尔国立大学);Artificial Intelligence Institute, Seoul National University(人工智能研究所,首尔国立大学);Department of Computer Science, Seoul National University(计算机科学系,首尔国立大学)
EvObj: Learning Evolving Object-centric Representations for 3D Instance Segmentation without Scene Supervision
Shenzhen Research Institute, The Hong Kong Polytechnic University(深圳研究 institute,香港理工大学);vLAR Group, The Hong Kong Polytechnic University(vLAR 团队,香港理工大学)
UC-OWOD: Unknown-Classified Open World Object Detection
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Peking University(北京大学)
Indian Institute of Technology Hyderabad(印度理工学院海得拉巴分校);Mohamed bin Zayed University of AI(穆罕默德·本·扎耶德人工智能大学);Australian National University(澳大利亚国立大学);Linköping University(林雪平大学)
Synthetic Data Augmentation for Enhanced Chicken Carcass Instance Segmentation
Department of Biological and Agricultural Engineering, University of Arkansas(生物与农业工程系,阿肯色大学);Department of Food Science, University of Arkansas(食品科学系,阿肯色大学);Department of Electrical Engineering, University of South Florida(电气工程系,佛罗里达州立大学);Department of Mechanical Engineering, University of Arkansas(机械工程系,阿肯色大学);Department of Industrial Engineering, Purdue University(工业工程系,普渡大学)
Disco: Densely-overlapping Cell Instance Segmentation via Adjacency-aware Collaborative Coloring
Shanghai Academy of Artificial Intelligence for Science(上海人工智能科学研究院);Fudan University(复旦大学);Shanghai Innovation Institute(上海创新研究院);School of Life Sciences and Biotechnology, Shanghai Jiao Tong University(上海交通大学生命科学与生物技术学院);Lingang Laboratory(临港实验室);Renji Hospital, School of Medicine, Shanghai Jiao Tong University(仁济医院,上海交通大学医学院)
Semi-supervised classification of dental conditions in panoramic radiographs using large language model and instance segmentation: A real-world dataset evaluation
Institute of Computing, Federal University of Bahia(巴伊亚联邦大学计算学院);Federal Institute of Bahia(巴伊亚联邦学院);Federal University of Maranhão(马拉尼昂联邦大学);Harvard School of Public Health(哈佛大学公共卫生学院);Instituto de Ciência e Inovação em Engenharia Mecânica e Engenharia Industrial(机械与工业工程科学与创新研究所);Faculdade de Engenharia, Universidade do Porto(波尔图大学工程学院);Dentistry Faculty, Federal University of Bahia(巴伊亚联邦大学牙科学院)
Colon Nuclei Instance Segmentation using a Probabilistic Two-Stage Detector
University of Porto(波尔图大学);Institute for Systems and Computer Engineering Technology and Science (INESC TEC)(系统与计算机工程技术与科学研究所(INESC TEC));Czech Technical University(捷克理工大学)
Understanding Image2Video Domain Shift in Food Segmentation: An Instance-level Analysis on Apples
Interdisciplinary Program in Artificial Intelligence, Seoul National University(人工智能交叉学科项目,首尔国立大学);Department of Biological Environmental Science, College of Life Science and Biotechnology, Dongguk University(生命科学与生物技术学院生物环境科学系,东国大学);Department of Food Science and Biotechnology, Dongguk University(食品科学与生物技术系,东国大学)
Graduate School of Info. Sci. \& Eng. Ritsumeikan University, Osaka, Japan;College of Info. Sci. \& Eng. Ritsumeikan University, Osaka, Japan;Faculty of Sci.;Eng. Soka University, Tokyo, Japan;Research Organization of Sci. \& Tech. Ritsumeikan University, Shiga, Japan;Graduate School of Informatics Kyoto University, Kyoto, Japan
Fast Model-guided Instance-wise Adaptation Framework for Real-world Pansharpening with Fidelity Constraints
University of Electronic Science and Technology of China(电子科技大学);National Research Council, Institute of Methodologies for Environmental Analysis (CNR-IMAA)(国家研究委员会,环境分析方法研究所(CNR-IMAA))