This paper studies open-vocabulary segmentation (OVS) through calibrating in-vocabulary and domain-biased embedding space with generalized contextual prior of CLIP. As the core of open-vocabulary understanding, alignment of visual content with the semantics of unbounded text has become the bottleneck of this field. To address this challenge, recent works propose to utilize CLIP as an additional classifier and aggregate model predictions with CLIP classification results. Despite their remarkable progress, performance of OVS methods in relevant scenarios is still unsatisfactory compared with supervised counterparts. We attribute this to the in-vocabulary embedding and domain-biased CLIP prediction. To this end, we present a Semantic-assisted CAlibration Network (SCAN). In SCAN, we incorporate generalized semantic prior of CLIP into proposal embedding to avoid collapsing on known categories. Besides, a contextual shift strategy is applied to mitigate the lack of global context and unnatural background noise. With above designs, SCAN achieves state-of-the-art performance on all popular open-vocabulary segmentation benchmarks. Furthermore, we also focus on the problem of existing evaluation system that ignores semantic duplication across categories, and propose a new metric called Semantic-Guided IoU (SG-IoU).
Bridging 3D Gaussians and Semantic Occupancy for Comprehensive Open-Vocabulary Scene Understanding from Unposed Images
The Hong Kong Polytechnic University(香港理工大学);Eastern Institute of Technology(东方理工大学);Shanghai Jiao Tong University(上海交通大学);Wuhan University(武汉大学);Great Wall Motor(长城汽车);Tsinghua University(清华大学);Ningbo Institute of Digital Twin, Eastern Institute of Technology(东方理工大学宁波数字孪生研究院)
Taming SAM3 in the Wild: A Concept Bank for Open-Vocabulary Segmentation
Department of Electrical and Computer Engineering, Sungkyunkwan University(Sungkyunkwan 大学电气与计算机工程系);School of Computing and Artificial Intelligence, Southwest Jiaotong University(西南交通大学计算机与人工智能学院);School of Computer Science and Engineering, Nanjing University of Science and Technology(南京理工大学计算机科学与工程学院);School of Computer Science and Engineering, University of Electronic Science and Technology of China(中国电子科技大学计算机科学与工程学院)
School of Artificial Intelligence, Xidian University(西安电子科技大学人工智能学院);MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University(上海交通大学人工智能研究院)
Open-Vocabulary HOI Detection with Interaction-aware Prompt and Concept Calibration
Wangxuan Institute of Computer Technology, Peking University(北京大学王璇计算机技术研究所);National Institute of Health Data Science, Peking University(北京大学国家健康数据科学研究院)
Stereo obstacle detection for unmanned surface vehicles by IMU-assisted semantic segmentation
University of Ljubljana, Faculty of Computer and Information Science(卢布尔雅那大学计算机与信息科学学院);University of Ljubljana, Faculty of Electrical Engineering(卢布尔雅那大学电气工程学院)
Observation Quality in DSM-Assisted Aerial Localization: A Benchmark and Base-Anchored Uncertainty Calibration
Aerospace Information Research Institute, Chinese Academy of Sciences(中国科学院空天信息研究院);School of Electronic Engineering, Beijing University of Posts and Telecommunications(北京邮电大学电子工程学院)
Physical Color Calibration of Digital Pathology Scanners for Robust Artificial Intelligence Assisted Cancer Diagnosis
Karolinska Institutet(卡罗林斯卡学院);FFEI Ltd(FFEI有限公司);University of Turku(图尔库大学);Stavanger University Hospital(斯塔万格大学医院);University of Stavanger(斯塔万格大学);Aarhus University Hospital(奥胡斯大学医院);Aarhus University(奥胡斯大学);Tampere University(坦佩雷大学)
SurgCalib: Gaussian Splatting-Based Hand-Eye Calibration for Robot-Assisted Minimally Invasive Surgery
Robotics and Control Laboratory, University of British Columbia(机器人与控制实验室,不列颠哥伦比亚大学);Department of Biomedical Engineering, National University of Singapore(生物医学工程系,新加坡国立大学);Division of Otolaryngology – Head and Neck Surgery, University of British Columbia(耳鼻喉科–头颈外科部,不列颠哥伦比亚大学)