半监督全景叙事定位
Semi-Supervised Panoptic Narrative Grounding
AI总结：
本文提出半监督全景叙事定位方案 SS-PNG-NW+，通过 Burn-In、数据增强和基于质量驱动的损失调整（QLA）利用少量标注数据，在仅 30% 和 50% 监督数据下超越全监督模型。
摘要：
尽管取得了相当大的进展，全景叙事定位（Panoptic Narrative Grounding，PNG）的发展仍然受到高昂标注成本的制约。在本文中，我们提出了一种新颖的半监督全景叙事定位（Semi-Supervised Panoptic Narrative Grounding，SS-PNG）学习方案，利用较少的已标注图像-文本对和较多的未标注图像-文本对来实现具有竞争力的性能。与视觉分割任务不同，PNG 涉及一个像素属于多个开放式名词。因此，现有的基于多类别的半监督分割框架无法直接应用于该任务。为了解决这一挑战，我们首先开发了一种专门针对 SS-PNG 设置的新型 SS-PNG 网络（SS-PNG-NW）。我们深入研究了 Burn-In 和数据增强等策略，以确定 SS-PNG-NW 的最佳通用配置。此外，为了解决伪标签质量不平衡的问题，我们提出了一种基于质量驱动的损失调整（Quality-Based Loss Adjustment，QLA）方法来调整半监督目标，从而得到增强的 SS-PNG-NW+。采用我们提出的 QLA，我们分别在像素级和掩码级改进了 BCE Loss 和 Dice loss。我们在 PNG 数据集上进行了大量实验，我们的 SS-PNG-NW+ 在所有数据比例下都表现出与全监督模型相当的有希望的结果。值得注意的是，我们的 SS-PNG-NW+ 在仅使用 30% 和 50% 监督数据的情况下优于全监督模型，分别超过其性能 0.8% 和 1.1%。这凸显了我们提出的 SS-PNG-NW+ 在克服有限标注带来的挑战和增强 PNG 任务适用性方面的有效性。源代码可在 https://github.com/nini0919/SSPNG 获取。
英文摘要：
Despite considerable progress, the advancement of Panoptic Narrative Grounding (PNG) remains hindered by costly annotations. In this paper, we introduce a novel Semi-Supervised Panoptic Narrative Grounding (SS-PNG) learning scheme, capitalizing on a smaller set of labeled image-text pairs and a larger set of unlabeled pairs to achieve competitive performance. Unlike visual segmentation tasks, PNG involves one pixel belonging to multiple open-ended nouns. As a result, existing multi-class based semi-supervised segmentation frameworks cannot be directly applied to this task. To address this challenge, we first develop a novel SS-PNG Network (SS-PNG-NW) tailored to the SS-PNG setting. We thoroughly investigate strategies such as Burn-In and data augmentation to determine the optimal generic configuration for the SS-PNG-NW. Additionally, to tackle the issue of imbalanced pseudo-label quality, we propose a Quality-Based Loss Adjustment (QLA) approach to adjust the semi-supervised objective, resulting in an enhanced SS-PNG-NW+. Employing our proposed QLA, we improve BCE Loss and Dice loss at pixel and mask levels, respectively. We conduct extensive experiments on PNG datasets, with our SS-PNG-NW+ demonstrating promising results comparable to fully-supervised models across all data ratios. Remarkably, our SS-PNG-NW+ outperforms fully-supervised models with only 30% and 50% supervision data, exceeding their performance by 0.8% and 1.1% respectively. This highlights the effectiveness of our proposed SS-PNG-NW+ in overcoming the challenges posed by limited annotations and enhancing the applicability of PNG tasks. The source code is available at https://github.com/nini0919/SSPNG.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
