Ref-Diff：基于生成模型的零样本指代图像分割
Ref-Diff: Zero-shot Referring Image Segmentation with Generative Models
AI总结：
提出Ref-Diff，利用生成模型（如Stable Diffusion）的细粒度多模态信息实现零样本指代图像分割，无需提议生成器即可媲美现有弱监督方法，与判别模型结合后性能显著提升。
摘要：
零样本指代图像分割是一项具有挑战性的任务，因为其目标是根据给定的指代表述找到对应的实例分割掩码，而无需在此类配对数据上进行训练。当前的零样本方法主要侧重于使用预训练的判别模型（如CLIP）。然而，我们观察到生成模型（如Stable Diffusion）可能已经理解了各种视觉元素与文本描述之间的关系，而这一潜力在该任务中鲜有研究。在这项工作中，我们引入了一种新颖的指代扩散分割器（Ref-Diff）用于该任务，它利用了生成模型中的细粒度多模态信息。我们证明，在没有提议生成器的情况下，仅凭生成模型就能达到与现有最先进的弱监督模型相当的性能。当我们结合生成模型和判别模型时，我们的Ref-Diff显著优于这些竞争方法。这表明生成模型对此任务同样有益，并能与判别模型互补，以实现更好的指代分割。我们的代码公开于https://github.com/kodenii/Ref-Diff。
英文摘要：
Zero-shot referring image segmentation is a challenging task because it aims to find an instance segmentation mask based on the given referring descriptions, without training on this type of paired data. Current zero-shot methods mainly focus on using pre-trained discriminative models (e.g., CLIP). However, we have observed that generative models (e.g., Stable Diffusion) have potentially understood the relationships between various visual elements and text descriptions, which are rarely investigated in this task. In this work, we introduce a novel Referring Diffusional segmentor (Ref-Diff) for this task, which leverages the fine-grained multi-modal information from generative models. We demonstrate that without a proposal generator, a generative model alone can achieve comparable performance to existing SOTA weakly-supervised models. When we combine both generative and discriminative models, our Ref-Diff outperforms these competing methods by a significant margin. This indicates that generative models are also beneficial for this task and can complement discriminative models for better referring segmentation. Our code is publicly available at https://github.com/kodenii/Ref-Diff.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
