被字幕“背叛”：面向开放词汇实例分割的字幕定位与生成联合框架
Betrayed by Captions: Joint Caption Grounding and Generation for Open Vocabulary Instance Segmentation
AI总结：
针对开放词汇实例分割现有方法存在监督噪声、未利用上下文的问题，提出联合字幕定位与生成框架，通过名词匹配定位损失和字幕生成头互补提升，在COCO数据集的两项任务上实现新类别性能显著提升。
摘要：
本研究聚焦开放词汇实例分割任务，旨在扩展分割模型的能力，使其能够对实例级别的新类别进行分类和分割。现有方法依赖大规模字幕数据集与复杂流水线，来建立图像区域与字幕词汇之间的一一映射关系。但这类方法会将不可见的词汇（如形容词、动词）与图像区域匹配，从而产生带噪声的监督信号。与此同时，上下文词汇与新类别存在高度相关性，对于推断新物体的存在同样重要。\n为解决上述局限，我们设计了联合字幕定位与生成（Caption Grounding and Generation，CGG）框架，该框架引入了一种仅聚焦于物体名词匹配的新型定位损失，以提升学习效率。我们还添加了一个字幕生成头，可提供额外监督并实现上下文建模，作为定位损失的补充。我们的分析与实验结果表明，定位组件与生成组件能够相互补充，显著提升模型对新类别的分割性能。\n我们在COCO数据集上开展了两组设置的实验：开放词汇实例分割（Open Vocabulary Instance Segmentation，OVIS）和开放集全景分割（Open Set Panoptic Segmentation，OSPS），实验结果证明了CGG框架的优越性。具体而言，在OVIS任务上，CGG无需额外数据即可使新类别的平均精度均值（mAP）大幅提升6.8%；在OSPS基准上，新类别的全景质量（PQ）提升了15%。
英文摘要：
In this work, we focus on open vocabulary instance segmentation to expand a segmentation model to classify and segment instance-level novel categories. Previous approaches have relied on massive caption datasets and complex pipelines to establish one-to-one mappings between image regions and words in captions. However, such methods build noisy supervision by matching non-visible words to image regions, such as adjectives and verbs. Meanwhile, context words are also important for inferring the existence of novel objects as they show high inter-correlations with novel categories. To overcome these limitations, we devise a joint \textbf{Caption Grounding and Generation (CGG)} framework, which incorporates a novel grounding loss that only focuses on matching object nouns to improve learning efficiency. We also introduce a caption generation head that enables additional supervision and contextual modeling as a complementation to the grounding loss. Our analysis and results demonstrate that grounding and generation components complement each other, significantly enhancing the segmentation performance for novel classes. Experiments on the COCO dataset with two settings: Open Vocabulary Instance Segmentation (OVIS) and Open Set Panoptic Segmentation (OSPS) demonstrate the superiority of the CGG. Specifically, CGG achieves a substantial improvement of 6.8% mAP for novel classes without extra data on the OVIS task and 15% PQ improvements for novel classes on the OSPS benchmark.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
