基于文本数据中心与交互式提示的图像描述生成
Text Data-Centric Image Captioning with Interactive Prompts
AI总结：
本文提出TIPCap方法，通过多元高斯映射和提示交互模块，在四种逐步减少配对数据依赖的设置下实现以文本数据为中心的图像描述生成，并在MS-COCO和Flickr30K上取得最优性能。
摘要：
有监督图像描述生成方法已取得巨大进展，但收集高质量的人工标注图像-文本数据仍然具有挑战性。近年来，大规模视觉与语言模型（如CLIP）和大规模生成式语言模型（如GPT-2）在多种任务中展现出强大性能，这也为利用网络配对数据、非配对数据甚至纯文本数据进行图像描述生成提供了一些新的解决方案。其中，主流方案是在CLIP模型图像-文本对一致表示的辅助下，将图像嵌入投影到文本嵌入空间中。然而，当前方法在以下方面仍面临若干挑战：在统一解决方案中适应多样化的数据配置、准确估计图像-文本嵌入偏差，以及在推理阶段纠正不理想的预测结果。本文提出了一种新的以文本数据为中心、采用交互式提示的图像描述生成方法，命名为TIPCap。1）我们考虑了四种不同设置，逐步降低对配对数据的依赖。2）我们构建了一个由多元高斯分布驱动的映射模块，以缓解模态差距，该模块适用于上述四种不同设置。3）我们提出了一个提示交互模块，可以在生成描述之前融入可选的提示信息。大量实验表明，我们的TIPCap优于其他弱监督或无监督图像描述生成方法，并在两个广泛使用的数据集MS-COCO和Flickr30K上取得了新的最先进性能。
英文摘要：
Supervised image captioning approaches have made great progress, but it is challenging to collect high-quality human-annotated image-text data. Recently, large-scale vision and language models (e.g., CLIP) and large-scale generative language models (e.g., GPT-2) have shown strong performances in various tasks, which also provide some new solutions for image captioning with web paired data, unpaired data or even text-only data. Among them, the mainstream solution is to project image embeddings into the text embedding space with the assistance of consistent representations between image-text pairs from the CLIP model. However, the current methods still face several challenges in adapting to the diversity of data configurations in a unified solution, accurately estimating image-text embedding bias, and correcting unsatisfactory prediction results in the inference stage. This paper proposes a new Text data-centric approach with Interactive Prompts for image Captioning, named TIPCap. 1) We consider four different settings which gradually reduce the dependence on paired data. 2) We construct a mapping module driven by multivariate Gaussian distribution to mitigate the modality gap, which is applicable to the above four different settings. 3) We propose a prompt interaction module that can incorporate optional prompt information before generating captions. Extensive experiments show that our TIPCap outperforms other weakly or unsupervised image captioning methods and achieves a new state-of-the-art performance on two widely used datasets, i.e., MS-COCO and Flickr30K.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
