PiTL: 通过提示式弱监督视觉语言预训练实现跨模态检索
PiTL: Cross-modal Retrieval with Weakly-supervised Vision-language Pre-training via Prompting
AI总结：
针对视觉语言预训练依赖大量标注图文对的问题，提出PiTL弱监督方法，通过提示大语言模型生成图像描述构建数据集，在跨模态检索任务上以更少监督超越现有弱监督方法。
摘要：
视觉语言（VL）预训练（VLP）已被证明能让VL模型在广泛的VL下游任务中具备良好的泛化能力，尤其是在跨模态检索任务上。然而，该方法依赖大量图文对，需要繁琐且成本高昂的数据整理工作。与之相对，弱监督VLP（W-VLP）探索了利用由预训练目标检测器（OD）从图像中生成的目标标签的方法，但这类方法仍需要配对信息，即图像与目标级标注，作为训练目标检测器的监督信号。为进一步减少监督数据量，我们提出Prompts-in-The-Loop（PiTL，提示在环）方法，通过提示从大语言模型（LLM）中获取知识来描述图像。具体而言，给定一张图像的类别标签（例如“炼油厂”），由LLM提取的知识（例如“炼油厂中可以看到大型储油罐、管道系统以及……”）会被用作对应的语言模态数据。这些知识补充了诸如场景中最可能出现的实体之间的常见关系等信息。我们构建了IN14K，这是一个全新的VL数据集，包含来自ImageNet21K的14K个类别的900万张图像和100万条由PiTL生成的描述。实验表明，使用PiTL生成的图文对预训练的VL模型，在图像到文本（I2T）和文本到图像（T2I）检索任务上，以更少的监督数据显著优于其他W-VLP方法。该结果证明了PiTL生成的配对数据用于VLP的有效性。
英文摘要：
Vision-language (VL) Pre-training (VLP) has shown to well generalize VL models over a wide range of VL downstream tasks, especially for cross-modal retrieval. However, it hinges on a huge amount of image-text pairs, which requires tedious and costly curation. On the contrary, weakly-supervised VLP (W-VLP) explores means with object tags generated by a pre-trained object detector (OD) from images. Yet, they still require paired information, i.e. images and object-level annotations, as supervision to train an OD. To further reduce the amount of supervision, we propose Prompts-in-The-Loop (PiTL) that prompts knowledge from large language models (LLMs) to describe images. Concretely, given a category label of an image, e.g. refinery, the knowledge, e.g. a refinery could be seen with large storage tanks, pipework, and ..., extracted by LLMs is used as the language counterpart. The knowledge supplements, e.g. the common relations among entities most likely appearing in a scene. We create IN14K, a new VL dataset of 9M images and 1M descriptions of 14K categories from ImageNet21K with PiTL. Empirically, the VL models pre-trained with PiTL-generated pairs are strongly favored over other W-VLP works on image-to-text (I2T) and text-to-image (T2I) retrieval tasks, with less supervision. The results reveal the effectiveness of PiTL-generated pairs for VLP.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
