arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2212.09737cs.CV
收藏

用于视觉-语言预训练的位置引导文本提示

Position-guided Text Prompt for Vision-Language Pre-training

National University of Singapore(新加坡国立大学);Sea AI Lab(Sea人工智能实验室)

Alex Jinpeng Wang, Pan Zhou, Mike Zheng Shou, Shuicheng Yan

AI总结:

本文提出位置引导文本提示 PTP,通过图像分块、目标检测和填空式预训练增强 VLP 模型的视觉定位能力,并在多个框架与基准上取得显著提升。

摘要:

视觉-语言预训练(Vision-Language Pre-Training,VLP)已在对齐图像与文本对方面展现出可观能力,推动了种类广泛的跨模态学习任务。然而,我们观察到 VLP 模型往往缺乏视觉 grounding/定位能力,而这种能力对视觉推理等许多下游任务至关重要。在本工作中,我们提出一种新颖的位置引导文本提示(Position-guided Text Prompt,PTP)范式,以增强采用 VLP 训练的跨模态模型的视觉 grounding 能力。具体而言,在 VLP 阶段,PTP 将图像划分为 $N\ imes N$ 个块,并通过 VLP 中广泛使用的目标检测器识别每个块中的目标。随后,它在给定 PTP 的情况下,将视觉 grounding 任务重构为填空题问题,鼓励模型预测给定块中的目标,或回归给定目标所在的块,例如在 PTP“The block P has a O”中填入“P”或“O”。该机制提升了 VLP 模型的视觉 grounding 能力,从而帮助它们更好地处理各类下游任务。通过将 PTP 引入若干最先进的 VLP 框架,我们在具有代表性的跨模态学习模型架构和多个基准上观察到一致且显著的改进,例如 ViLT 基线在零样本 Flickr30K Retrieval 上 average recall@1 提升 4.8,SOTA BLIP 基线在 COCO Captioning 上 CIDEr 提升 5.3。此外,PTP 取得了与基于目标检测器的方法相当的结果,并且推理速度快得多,因为 PTP 在推理时丢弃其目标检测器,而后者无法这样做。我们的代码和预训练权重将在 https://github.com/sail-sg/ptp 发布。

英文摘要:

Vision-Language Pre-Training (VLP) has shown promising capabilities to align image and text pairs, facilitating a broad variety of cross-modal learning tasks. However, we observe that VLP models often lack the visual grounding/localization capability which is critical for many downstream tasks such as visual reasoning. In this work, we propose a novel Position-guided Text Prompt (PTP) paradigm to enhance the visual grounding ability of cross-modal models trained with VLP. Specifically, in the VLP phase, PTP divides the image into $N\times N$ blocks, and identifies the objects in each block through the widely used object detector in VLP. It then reformulates the visual grounding task into a fill-in-the-blank problem given a PTP by encouraging the model to predict the objects in the given blocks or regress the blocks of a given object, e.g. filling `P" or ``O" in aPTP ``The block P has a O". This mechanism improves the visual grounding capability of VLP models and thus helps them better handle various downstream tasks. By introducing PTP into several state-of-the-art VLP frameworks, we observe consistently significant improvements across representative cross-modal learning model architectures and several benchmarks, e.g. zero-shot Flickr30K Retrieval (+4.8 in average recall@1) for ViLT \cite{vilt} baseline, and COCO Captioning (+5.3 in CIDEr) for SOTA BLIP \cite{blip} baseline. Moreover, PTP achieves comparable results with object-detector based methods, and much faster inference speed since PTP discards its object detector for inference while the later cannot. Our code and pre-trained weight will be released at \url{https://github.com/sail-sg/ptp}.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑