DINOv2结合文本：面向图像级和像素级视觉-语言对齐的统一框架
DINOv2 Meets Text: A Unified Framework for Image- and Pixel-Level Vision-Language Alignment
AI总结：
提出dino.txt框架，通过改进LiT训练策略并引入[CLS]词元与patch平均值拼接及多模态数据整理，使DINOv2实现视觉-语言对齐，以极低计算成本在零样本分类和开放词汇语义分割任务上达到最先进水平。
摘要：
自监督视觉基础模型能生成强大的嵌入，在广泛的下游任务中取得卓越性能。然而，与CLIP等视觉-语言模型不同，自监督视觉特征难以直接与语言对齐，阻碍了其在开放词汇任务中的应用。我们提出名为dino.txt的方法，为广泛使用的自监督视觉编码器DINOv2解锁了这一新能力。我们基于LiT训练策略，该策略训练文本编码器与冻结的视觉模型对齐，但在密集任务上效果不佳。我们提出了几个关键要素来提升全局和密集任务的性能，例如将[CLS]词元与patch平均值拼接以训练对齐，并利用文本和图像两种模态来整理数据。通过这些方法，我们仅用CLIP一小部分的计算成本，就成功训练出类似CLIP的模型，并在零样本分类和开放词汇语义分割中取得了最先进的结果。
英文摘要：
Self-supervised visual foundation models produce powerful embeddings that achieve remarkable performance on a wide range of downstream tasks. However, unlike vision-language models such as CLIP, self-supervised visual features are not readily aligned with language, hindering their adoption in open-vocabulary tasks. Our method, named dino.txt, unlocks this new ability for DINOv2, a widely used self-supervised visual encoder. We build upon the LiT training strategy, which trains a text encoder to align with a frozen vision model but leads to unsatisfactory results on dense tasks. We propose several key ingredients to improve performance on both global and dense tasks, such as concatenating the [CLS] token with the patch average to train the alignment and curating data using both text and image modalities. With these, we successfully train a CLIP-like model with only a fraction of the computational cost compared to CLIP while achieving state-of-the-art results in zero-shot classification and open-vocabulary semantic segmentation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
