通过纯文本训练挖掘细粒度图文对齐以实现零样本图像描述
Mining Fine-Grained Image-Text Alignment for Zero-Shot Captioning via Text-Only Training
AI总结：
本文提出一种基于纯文本训练的零样本图像描述框架MacCap，通过子区域特征聚合、噪声注入和CLIP重排序策略减小CLIP潜在空间中的模态鸿沟，在MSCOCO等数据集上取得显著性能提升。
摘要：
图像描述生成旨在生成图像的描述性且有意义的文本描述，从而支持广泛的视觉-语言应用。先前的研究表明，利用对比图像语言预训练（CLIP）的强大能力为实现零样本图像描述生成提供了一种有前景的方法，从而无需昂贵的描述标注。然而，在CLIP的潜在空间中普遍观察到的模态鸿沟会破坏成对图文特征之间的对齐，从而损害零样本图像描述的性能。为了解决这个问题，我们对CLIP潜在空间进行了分析，得出了两点发现。首先，我们观察到由于文本描述中固有的信息丢失，CLIP的图像子区域视觉特征能够更接近配对的描述。此外，我们表明成对图文之间的模态鸿沟可以在经验上被建模为零均值高斯分布。受这些发现的启发，我们提出了一个新颖的零样本图像描述框架，该框架仅使用文本进行训练以减小模态鸿沟。具体而言，我们引入了子区域特征聚合来利用局部区域信息，从而生成紧凑的视觉表示以匹配文本表示。此外，我们结合了噪声注入和CLIP重排序策略来提升描述性能。我们还将该框架扩展以构建零样本VQA流程，展示了其通用性。通过在MSCOCO、Flickr30k和VQAV2等常见描述和VQA数据集上的大量实验，我们表明我们的方法取得了显著的性能提升。代码可在https://github.com/Artanic30/MacCap获取。
英文摘要：
Image captioning aims at generating descriptive and meaningful textual descriptions of images, enabling a broad range of vision-language applications. Prior works have demonstrated that harnessing the power of Contrastive Image Language Pre-training (CLIP) offers a promising approach to achieving zero-shot captioning, eliminating the need for expensive caption annotations. However, the widely observed modality gap in the latent space of CLIP harms the performance of zero-shot captioning by breaking the alignment between paired image-text features. To address this issue, we conduct an analysis on the CLIP latent space which leads to two findings. Firstly, we observe that the CLIP's visual feature of image subregions can achieve closer proximity to the paired caption due to the inherent information loss in text descriptions. In addition, we show that the modality gap between a paired image-text can be empirically modeled as a zero-mean Gaussian distribution. Motivated by the findings, we propose a novel zero-shot image captioning framework with text-only training to reduce the modality gap. In particular, we introduce a subregion feature aggregation to leverage local region information, which produces a compact visual representation for matching text representation. Moreover, we incorporate a noise injection and CLIP reranking strategy to boost captioning performance. We also extend our framework to build a zero-shot VQA pipeline, demonstrating its generality. Through extensive experiments on common captioning and VQA datasets such as MSCOCO, Flickr30k and VQAV2, we show that our method achieves remarkable performance improvements. Code is available at https://github.com/Artanic30/MacCap.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
