利用文本到图像扩散模型探索短语级定位
Exploring Phrase-Level Grounding with Text-to-Image Diffusion Model
AI总结：
本文提出DiffPNG框架，利用文本到图像扩散模型的交叉与自注意力及SAM细化模块，实现零样本短语级全景叙事定位。
摘要：
近年来，扩散模型在视觉理解方面日益展现出强大能力。通过利用基于提示的学习来构造句子，这些模型已在分类和视觉定位任务中表现出熟练水平。然而，现有方法主要展示了其执行句子级定位的能力，而利用上下文信息进行短语级理解的潜力在很大程度上仍未得到探索。本文以全景叙事定位（Panoptic Narrative Grounding，PNG）作为代理任务，进一步研究这种能力。PNG旨在分割给定叙事文本中由多个名词短语提及的对象实例。具体而言，我们提出DiffPNG框架，这是一种简单而有效的方法，通过将过程分解为定位、分割和细化步骤序列，充分利用扩散模型架构进行分割。该框架首先使用交叉注意力机制识别锚点，随后利用自注意力执行分割，以实现零样本PNG。此外，我们引入基于SAM的细化模块，以提升分割掩码的质量。我们在PNG数据集上的大量实验表明，DiffPNG在零样本PNG任务设置下取得了强劲性能，确凿证明了扩散模型具备上下文感知的短语级理解能力。源代码可在https://github.com/nini0919/DiffPNG获取。
英文摘要：
Recently, diffusion models have increasingly demonstrated their capabilities in vision understanding. By leveraging prompt-based learning to construct sentences, these models have shown proficiency in classification and visual grounding tasks. However, existing approaches primarily showcase their ability to perform sentence-level localization, leaving the potential for leveraging contextual information for phrase-level understanding largely unexplored. In this paper, we utilize Panoptic Narrative Grounding (PNG) as a proxy task to investigate this capability further. PNG aims to segment object instances mentioned by multiple noun phrases within a given narrative text. Specifically, we introduce the DiffPNG framework, a straightforward yet effective approach that fully capitalizes on the diffusion's architecture for segmentation by decomposing the process into a sequence of localization, segmentation, and refinement steps. The framework initially identifies anchor points using cross-attention mechanisms and subsequently performs segmentation with self-attention to achieve zero-shot PNG. Moreover, we introduce a refinement module based on SAM to enhance the quality of the segmentation masks. Our extensive experiments on the PNG dataset demonstrate that DiffPNG achieves strong performance in the zero-shot PNG task setting, conclusively proving the diffusion model's capability for context-aware, phrase-level understanding. Source code is available at \url{https://github.com/nini0919/DiffPNG}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
