CLIP-DINOiser：教CLIP一些DINO技巧以实现开放词汇语义分割
CLIP-DINOiser: Teaching CLIP a few DINO tricks for open-vocabulary semantic segmentation
AI总结：
结合CLIP与自监督特征优势，提出无需标注的开放词汇语义分割方法CLIP-DINOiser，在多个细粒度基准上达到SOTA。
摘要：
流行的CLIP模型凭借其与任意文本提示的无缝交互，展现出令人印象深刻的零样本能力。然而，它缺乏空间感知能力，如果不进行额外的微调步骤，便不适合密集的计算机视觉任务（例如语义分割），而这种微调通常使用标注，并可能抑制其原有的开放词汇特性。与此同时，自监督表示方法已经证明，在没有人工标注或显式监督的情况下也能具备良好的定位属性。在这项工作中，我们结合两者的优势，提出了一种不需要任何标注的开放词汇语义分割方法。我们提出，通过整合从自监督特征中提取的定位先验，来局部改进密集的MaskCLIP特征（该特征是通过对CLIP最后一个池化层进行简单修改计算得出的）。通过这样做，我们极大地提升了MaskCLIP的性能，并产生了平滑的输出。此外，我们展示了所使用的自监督特征属性可以直接从CLIP特征中学习到。我们的方法CLIP-DINOiser在推理时仅需CLIP的单次前向传播和两个轻量卷积层，无需额外监督或额外内存，并在COCO、Pascal Context、Cityscapes和ADE20k等具有挑战性的细粒度基准上达到了最先进的结果。用于复现我们结果的代码可在https://github.com/wysoczanska/clip_dinoiser获取。
英文摘要：
The popular CLIP model displays impressive zero-shot capabilities thanks to its seamless interaction with arbitrary text prompts. However, its lack of spatial awareness makes it unsuitable for dense computer vision tasks, e.g., semantic segmentation, without an additional fine-tuning step that often uses annotations and can potentially suppress its original open-vocabulary properties. Meanwhile, self-supervised representation methods have demonstrated good localization properties without human-made annotations nor explicit supervision. In this work, we take the best of both worlds and propose an open-vocabulary semantic segmentation method, which does not require any annotations. We propose to locally improve dense MaskCLIP features, which are computed with a simple modification of CLIP's last pooling layer, by integrating localization priors extracted from self-supervised features. By doing so, we greatly improve the performance of MaskCLIP and produce smooth outputs. Moreover, we show that the used self-supervised feature properties can directly be learnt from CLIP features. Our method CLIP-DINOiser needs only a single forward pass of CLIP and two light convolutional layers at inference, no extra supervision nor extra memory and reaches state-of-the-art results on challenging and fine-grained benchmarks such as COCO, Pascal Context, Cityscapes and ADE20k. The code to reproduce our results is available at https://github.com/wysoczanska/clip_dinoiser.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
