使用 VLM 在不一致分类体系上进行跨域语义分割
Cross-Domain Semantic Segmentation on Inconsistent Taxonomy using VLMs
AI总结：
本文提出 CSI 框架，利用 VLM 的语义知识结合传统 UDA 的分割推理，在无目标标签且类别分类体系不一致时完成跨域语义分割。
摘要：
无监督域适应（UDA）中语义分割的挑战不仅源于源图像与目标图像之间的域偏移，还源于跨域类别分类体系的差异。传统 UDA 研究假设源域与目标域之间具有一致的分类体系，因此限制了其识别并适应目标域分类体系的能力。本文提出一种新方法，即使用视觉语言模型在不一致分类体系上进行跨域语义分割（CSI），即使在源域与目标域类别不匹配的情况下，也能有效执行域自适应语义分割。CSI 利用视觉语言模型（VLM）的语义泛化潜力，与以往 UDA 方法形成协同。它利用通过传统 UDA 方法获得的分割推理，并结合 VLM 中嵌入的丰富语义知识，对目标域中的新类别重新标注。该方法无需任何目标域真值标签，即可有效适应扩展后的分类体系。我们的方法在不一致分类体系设置（从粗到细分类体系和开放分类体系）下的多个基准上均表现出有效性，并且在与以往最先进 UDA 方法结合时展现出稳定的协同效应。实现代码可在 http://github.com/jkee58/CSI 获取。
英文摘要：
The challenge of semantic segmentation in Unsupervised Domain Adaptation (UDA) emerges not only from domain shifts between source and target images but also from discrepancies in class taxonomies across domains. Traditional UDA research assumes consistent taxonomy between the source and target domains, thereby limiting their ability to recognize and adapt to the taxonomy of the target domain. This paper introduces a novel approach, Cross-Domain Semantic Segmentation on Inconsistent Taxonomy using Vision Language Models (CSI), which effectively performs domain-adaptive semantic segmentation even in situations of source-target class mismatches. CSI leverages the semantic generalization potential of Visual Language Models (VLMs) to create synergy with previous UDA methods. It leverages segment reasoning obtained through traditional UDA methods, combined with the rich semantic knowledge embedded in VLMs, to relabel new classes in the target domain. This approach allows for effective adaptation to extended taxonomies without requiring any ground truth label for the target domain. Our method has shown to be effective across various benchmarks in situations of inconsistent taxonomy settings (coarse-to-fine taxonomy and open taxonomy) and demonstrates consistent synergy effects when integrated with previous state-of-the-art UDA methods. The implementation is available at http://github.com/jkee58/CSI.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
