arXiv 2311.010912024-03-10cs.CV

通过耦合像素与物体上下文丰富短语表征以实现全景叙事定位

Enriching Phrases with Coupled Pixel and Object Contexts for Panoptic Narrative Grounding

Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；Meituan（美团）；Institute of Artificial Intelligence, Beihang University（北京航空航天大学人工智能研究院）；Hangzhou Innovation Institute, Beihang University（北京航空航天大学杭州创新研究院）

Tianrui Hui, Zihan Ding, Junshi Huang, Xiaoming Wei, Xiaolin Wei, Jiao Dai, Jizhong Han, Si Liu

备注：: Accepted by IJCAI 2023. Since the PNG benchmark adopts a different data partition manner from ours, we update the experimental results on the things/stuff/singulars/plurals subsets based on the PNG's code

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对全景叙事定位任务中现有方法上下文聚合存在噪声、语义错位的问题，提出融合像素与物体上下文的PPO-TD结构及POCL损失，在基准上取得大幅领先的SOTA性能。

摘要：

全景叙事定位（PNG）旨在对叙事字幕中名词短语描述的图像中的事物与物体类别进行分割。作为一项多模态任务，PNG的核心在于图像与字幕间的视觉-语言交互。此前的两阶段方法从离线生成的掩码候选框中聚合视觉上下文到短语特征，这类上下文往往存在噪声且碎片化。近期的单阶段方法仅从图像特征中聚合像素上下文到短语特征，由于缺乏物体先验，可能导致语义错位。为实现更全面的视觉-语言交互，我们提出通过设计短语-像素-物体Transformer解码器（PPO-TD），耦合像素与物体上下文来丰富短语表征，同时将细粒度的局部细节与粗粒度的实体线索聚合到短语特征中。此外，我们还提出短语-物体对比损失（POCL），拉近匹配的短语-物体对、推远不匹配的对，以从更多与短语相关的物体token中聚合更精准的物体上下文。在PNG基准上的大量实验表明，我们的方法以大幅优势取得了新的state-of-the-art性能。

英文摘要：

Panoptic narrative grounding (PNG) aims to segment things and stuff objects in an image described by noun phrases of a narrative caption. As a multimodal task, an essential aspect of PNG is the visual-linguistic interaction between image and caption. The previous two-stage method aggregates visual contexts from offline-generated mask proposals to phrase features, which tend to be noisy and fragmentary. The recent one-stage method aggregates only pixel contexts from image features to phrase features, which may incur semantic misalignment due to lacking object priors. To realize more comprehensive visual-linguistic interaction, we propose to enrich phrases with coupled pixel and object contexts by designing a Phrase-Pixel-Object Transformer Decoder (PPO-TD), where both fine-grained part details and coarse-grained entity clues are aggregated to phrase features. In addition, we also propose a PhraseObject Contrastive Loss (POCL) to pull closer the matched phrase-object pairs and push away unmatched ones for aggregating more precise object contexts from more phrase-relevant object tokens. Extensive experiments on the PNG benchmark show our method achieves new state-of-the-art performance with large margins.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。