基于文本到图像扩散模型的开放词汇全景分割
Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models
AI总结：
提出ODISE，统一预训练文本-图像扩散与判别模型的冻结内部表示，执行开放词汇全景分割，在ADE20K等数据集上大幅超越先前最优方法。
摘要：
我们提出了ODISE：基于扩散的开放词汇全景分割，它统一了预训练的文本-图像扩散模型和判别模型来执行开放词汇全景分割。文本到图像扩散模型具有生成带有各种开放词汇语言描述的高质量图像的卓越能力。这表明它们的内部表示空间与现实世界中的开放概念高度相关。另一方面，像CLIP这样的文本-图像判别模型擅长将图像分类为开放词汇标签。我们利用这两个模型的冻结内部表示，对野外的任何类别进行全景分割。我们的方法在开放词汇全景分割和语义分割任务上，均以显著优势超越了先前的最优方法。特别是，仅使用COCO训练，我们的方法在ADE20K数据集上达到了23.4 PQ和30.0 mIoU，比先前的最优方法绝对提升了8.3 PQ和7.9 mIoU。我们在https://github.com/NVlabs/ODISE开源了代码和模型。
英文摘要：
We present ODISE: Open-vocabulary DIffusion-based panoptic SEgmentation, which unifies pre-trained text-image diffusion and discriminative models to perform open-vocabulary panoptic segmentation. Text-to-image diffusion models have the remarkable ability to generate high-quality images with diverse open-vocabulary language descriptions. This demonstrates that their internal representation space is highly correlated with open concepts in the real world. Text-image discriminative models like CLIP, on the other hand, are good at classifying images into open-vocabulary labels. We leverage the frozen internal representations of both these models to perform panoptic segmentation of any category in the wild. Our approach outperforms the previous state of the art by significant margins on both open-vocabulary panoptic and semantic segmentation tasks. In particular, with COCO training only, our method achieves 23.4 PQ and 30.0 mIoU on the ADE20K dataset, with 8.3 PQ and 7.9 mIoU absolute improvement over the previous state of the art. We open-source our code and models at https://github.com/NVlabs/ODISE .
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
