基于视觉基础模型的无标注语义分割
Annotation Free Semantic Segmentation with Vision Foundation Models
AI总结：
本研究针对语义分割需大量昂贵像素级标注的问题，利用CLIP、SAM等基础模型生成免费标注，结合DinoV2构建轻量对齐模块，实现了仅需少量训练、无人工标注的零样本语义分割。
摘要：
语义分割是最具挑战性的视觉任务之一，通常需要大量训练数据及成本高昂的像素级标注。随着基础模型尤其是视觉-语言模型的成功，近期研究尝试实现零样本语义分割，但仍需大规模训练或额外的图像/像素级标注。本研究利用现有基础模型为任意语义分割数据集生成免费标注：使用CLIP（对比语言-图像预训练模型）检测目标，用SAM（分割一切模型）生成高质量目标掩码。随后，我们在自监督视觉编码器DinoV2之上构建轻量模块，将图像块特征与预训练文本编码器对齐以实现零样本语义分割。该方法仅需极少训练即可为任意预训练视觉编码器赋予基于语言的语义能力，以基础模型作为唯一监督来源，仅需少量训练数据且无需人工标注即可实现泛化。
英文摘要：
Semantic Segmentation is one of the most challenging vision tasks, usually requiring large amounts of training data with expensive pixel level annotations. With the success of foundation models and especially vision-language models, recent works attempt to achieve zeroshot semantic segmentation while requiring either large-scale training or additional image/pixel level annotations. In this work, we generate free annotations for any semantic segmentation dataset using existing foundation models. We use CLIP to detect objects and SAM to generate high quality object masks. Next, we build a lightweight module on top of a self-supervised vision encoder, DinoV2, to align the patch features with a pretrained text encoder for zeroshot semantic segmentation. Our approach can bring language-based semantics to any pretrained vision encoder with minimal training, uses foundation models as the sole source of supervision and generalizes from little training data with no annotation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
