开放词汇全景分割与检索增强
Open Vocabulary Panoptic Segmentation With Retrieval Augmentation
AI总结：
本文提出RetCLIP方法，通过检索增强提升开放词汇全景分割的性能，实现对未见类别的有效分割。
摘要：
给定一个输入图像和一组类名，全景分割旨在将图像中的每个像素标记为类别标签和实例标签。相比之下，开放词汇全景分割旨在根据用户输入对任意类别进行分割。挑战在于，一个在特定数据集上训练的全景分割系统通常难以泛化到训练数据之外的未见类别。在本文中，我们提出了RetCLIP，一种检索增强的全景分割方法，以提高未见类别的性能。具体来说，我们利用配对的图像-文本数据构建了一个掩码段特征数据库。在推理时，我们使用输入图像的掩码段特征作为查询键，从数据库中检索相似特征和相关类别标签。根据查询特征与检索特征之间的相似性，为掩码段分配分类分数。检索基于的分类分数与CLIP基于的分数相结合，以产生最终输出。我们将我们的解决方案与先前的SOTA方法（FC-CLIP）结合。当在COCO上训练时，所提出的方法在ADE20k数据集上实现了30.9 PQ、19.3 mAP、44.0 mIoU，比基线提高了+4.5 PQ、+2.5 mAP、+10.0 mIoU的绝对提升。
英文摘要：
Given an input image and set of class names, panoptic segmentation aims to label each pixel in an image with class labels and instance labels. In comparison, Open Vocabulary Panoptic Segmentation aims to facilitate the segmentation of arbitrary classes according to user input. The challenge is that a panoptic segmentation system trained on a particular dataset typically does not generalize well to unseen classes beyond the training data. In this work, we propose RetCLIP, a retrieval-augmented panoptic segmentation method that improves the performance of unseen classes. In particular, we construct a masked segment feature database using paired image-text data. At inference time, we use masked segment features from the input image as query keys to retrieve similar features and associated class labels from the database. Classification scores for the masked segment are assigned based on the similarity between query features and retrieved features. The retrieval-based classification scores are combined with CLIP-based scores to produce the final output. We incorporate our solution with a previous SOTA method (FC-CLIP). When trained on COCO, the proposed method demonstrates 30.9 PQ, 19.3 mAP, 44.0 mIoU on the ADE20k dataset, achieving +4.5 PQ, +2.5 mAP, +10.0 mIoU absolute improvement over the baseline.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
