面向弱监督定位图像描述的自上而下框架
Top-Down Framework for Weakly-supervised Grounded Image Captioning
AI总结：
针对两阶段WSGIC依赖独立候选框、忽视物体关系的问题，提出单阶段自上而下框架，通过RGM、VLAMs和关系语义模块实现描述生成与定位，并在两个数据集上达到SOTA定位性能。
摘要：
弱监督定位图像描述（WSGIC）旨在不使用边界框监督的情况下，生成图像描述，并在输入图像中对预测出的物体词进行定位。近期的两阶段解决方案大多采用自下而上的流程：（1）使用物体检测器将输入图像编码为多个区域特征；（2）利用区域特征进行描述生成和定位。然而，使用物体检测器产生的独立候选框，往往会使后续的定位描述器过度拟合于寻找正确的物体词，忽视物体之间的关系，并为定位选择不兼容的候选区域。为解决这些问题，我们提出一种单阶段弱监督定位描述器，直接以 RGB 图像为输入，在自上而下的图像层面执行描述生成和定位。具体而言，我们将图像编码为视觉 token 表示，并在解码器中提出循环定位模块（RGM），以获得精确的视觉-语言注意图（VLAMs），用于识别物体的空间位置。此外，我们显式地将关系模块注入单阶段框架，通过多标签分类促进关系理解。该关系语义作为上下文信息，有助于预测描述中的关系词和物体词。我们观察到，关系语义不仅帮助定位描述器生成更准确的描述，还提升了定位性能。我们在两个具有挑战性的数据集（Flickr30k Entities captioning 和 MSCOCO captioning）上验证了所提方法的有效性。实验结果表明，我们的方法取得了最先进的定位性能。
英文摘要：
Weakly-supervised grounded image captioning (WSGIC) aims to generate the caption and ground (localize) predicted object words in the input image without using bounding box supervision. Recent two-stage solutions mostly apply a bottom-up pipeline: (1) encode the input image into multiple region features using an object detector; (2) leverage region features for captioning and grounding. However, utilizing independent proposals produced by object detectors tends to make the subsequent grounded captioner overfitted in finding the correct object words, overlooking the relation between objects, and selecting incompatible proposal regions for grounding. To address these issues, we propose a one-stage weakly-supervised grounded captioner that directly takes the RGB image as input to perform captioning and grounding at the top-down image level. Specifically, we encode the image into visual token representations and propose a Recurrent Grounding Module (RGM) in the decoder to obtain precise Visual Language Attention Maps (VLAMs), which recognize the spatial locations of the objects. In addition, we explicitly inject a relation module into our one-stage framework to encourage relation understanding through multi-label classification. This relation semantics served as contextual information facilitating the prediction of relation and object words in the caption. We observe that the relation semantic not only assists the grounded captioner in generating a more accurate caption but also improves the grounding performance. We validate the effectiveness of our proposed method on two challenging datasets (Flick30k Entities captioning and MSCOCO captioning). The experimental results demonstrate that our method achieves state-of-the-art grounding performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
