通过查询词进行视觉分割：用于半监督图像分割的语言锚点
Segmenting Visuals With Querying Words: Language Anchors For Semi-Supervised Image Segmentation
AI总结：
本文提出HVLFormer，通过生成多尺度文本查询和引入跨视角一致性正则化，提升视觉与文本对齐，以实现更准确的半监督图像分割。
摘要：
视觉语言模型（VLMs）提供了丰富的语义先验，但尚未在半监督语义分割中得到充分探索。最近尝试将VLMs整合以注入高层语义，但忽略了由于使用领域不变文本嵌入而不适应数据集和图像特定上下文而产生的视觉与文本表示之间的语义不一致。这种缺乏领域意识，加上标注有限，削弱了模型的语义理解，阻碍了有效的视觉语言对齐。为此，我们提出层级视觉语言转换器（HVLFormer），在掩码转换器架构中实现视觉与文本表示的领域意识和领域鲁棒对齐。首先，我们将预训练VLMs的文本嵌入转换为文本对象查询，生成多尺度、数据集感知的查询，从粗到细捕捉类别语义并增强上下文推理。接着，通过注入图像特定的视觉上下文来细化这些查询，使文本语义与局部场景结构对齐并增强类别区分。最后，为实现领域鲁棒性，我们引入跨视角和模态一致性正则化，强制在掩码转换器架构内跨增强视角保持预测一致性。此外，它在解码过程中确保了稳定的视觉语言对齐。在少于1%的训练数据下，HVLFormer在Pascal VOC、COCO、ADE20K和Cityscapes上优于现有方法。我们的代码和结果将在GitHub上发布。
英文摘要：
Vision Language Models (VLMs) provide rich semantic priors but are underexplored in Semi supervised Semantic Segmentation. Recent attempts to integrate VLMs to inject high level semantics overlook the semantic misalignment between visual and textual representations that arises from using domain invariant text embeddings without adapting them to dataset and image specific contexts. This lack of domain awareness, coupled with limited annotations, weakens the model semantic understanding by preventing effective vision language alignment. As a result, the model struggles with contextual reasoning, shows weak intra class discrimination, and confuses similar classes. To address these challenges, we propose Hierarchical Vision Language transFormer (HVLFormer), which achieves domain aware and domain robust alignment between visual and textual representations within a mask transformer architecture. Firstly, we transform text embeddings from pretrained VLMs into textual object queries, enabling the generation of multi scale, dataset aware queries that capture class semantics from coarse to fine granularity and enhance contextual reasoning. Next, we refine these queries by injecting image specific visual context to align textual semantics with local scene structures and enhance class discrimination. Finally, to achieve domain robustness, we introduce cross view and modal consistency regularization, which enforces prediction consistency within mask-transformer architecture across augmented views. Moreover, it ensures stable vision language alignment during decoding. With less than 1% training data, HVLFormer outperforms state of the art methods on Pascal VOC, COCO, ADE20K, and Cityscapes. Our code and results will be available on GitHub.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
