arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2312.04089cs.CV
收藏

基于语义辅助校准的开放词汇分割

Open-Vocabulary Segmentation with Semantic-Assisted Calibration

Shenzhen International Graduate School Tsinghua University(清华大学深圳国际研究生院);Sun Yat-sen University(中山大学);ByteDance Inc.(字节跳动公司)

Yong Liu, Sule Bai, Guanbin Li, Yitong Wang, Yansong Tang

AI总结:

针对开放词汇分割中视觉-文本语义对齐瓶颈,提出语义辅助校准网络(SCAN),通过CLIP语义先验校准嵌入空间并优化预测,在主流基准上实现SOTA性能,还提出SG-IoU指标解决评估缺陷。

摘要:

本文通过利用CLIP的广义上下文先验校准词汇内和领域偏差的嵌入空间,研究开放词汇分割(OVS)。作为开放词汇理解的核心,视觉内容与无界文本语义的对齐已成为该领域的瓶颈。为应对这一挑战,近期研究提出将CLIP用作额外分类器,并将模型预测与CLIP分类结果聚合。尽管取得显著进展,但OVS方法在相关场景中的性能仍不及有监督方法。我们将此归因于词汇内嵌入和领域偏差的CLIP预测。为此,我们提出语义辅助校准网络(SCAN):将CLIP的广义语义先验融入提案嵌入以避免已知类别坍缩,同时采用上下文偏移策略缓解全局上下文缺失和非自然背景噪声问题。通过上述设计,SCAN在所有主流开放词汇分割基准上实现了最先进性能;此外,我们还关注现有评估体系忽略类别间语义重复的问题,提出新指标语义引导IoU(SG-IoU)。

英文摘要:

This paper studies open-vocabulary segmentation (OVS) through calibrating in-vocabulary and domain-biased embedding space with generalized contextual prior of CLIP. As the core of open-vocabulary understanding, alignment of visual content with the semantics of unbounded text has become the bottleneck of this field. To address this challenge, recent works propose to utilize CLIP as an additional classifier and aggregate model predictions with CLIP classification results. Despite their remarkable progress, performance of OVS methods in relevant scenarios is still unsatisfactory compared with supervised counterparts. We attribute this to the in-vocabulary embedding and domain-biased CLIP prediction. To this end, we present a Semantic-assisted CAlibration Network (SCAN). In SCAN, we incorporate generalized semantic prior of CLIP into proposal embedding to avoid collapsing on known categories. Besides, a contextual shift strategy is applied to mitigate the lack of global context and unnatural background noise. With above designs, SCAN achieves state-of-the-art performance on all popular open-vocabulary segmentation benchmarks. Furthermore, we also focus on the problem of existing evaluation system that ignores semantic duplication across categories, and propose a new metric called Semantic-Guided IoU (SG-IoU).

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑