ViPCap：基于检索文本的视觉提示用于轻量级图像描述生成
ViPCap: Retrieval Text-Based Visual Prompts for Lightweight Image Captioning
AI总结：
ViPCap提出基于检索文本的视觉提示方法，通过将文本映射到CLIP空间并采样高斯分布增强视觉特征，在COCO等数据集上显著提升轻量级图像描述生成的效率与效果。
摘要：
近期使用检索数据的轻量级图像描述生成模型主要关注文本提示。然而，先前的工作仅将检索到的文本用作文本提示，视觉信息仅依赖于CLIP视觉嵌入。由于这一问题，提示中固有的图像描述未能在视觉嵌入空间中得到充分反映。为解决此问题，我们提出了ViPCap，一种新颖的基于检索文本的视觉提示方法，用于轻量级图像描述生成。ViPCap利用带有图像信息的检索文本作为视觉提示，以增强模型捕获相关视觉信息的能力。通过将文本提示映射到CLIP空间并生成多个随机化的高斯分布，我们的方法利用采样来探索随机增强的分布，并有效检索包含图像信息的语义特征。这些检索到的特征被整合到图像中并指定为视觉提示，从而在COCO、Flickr30k和NoCaps等数据集上带来性能提升。实验结果表明，ViPCap在效率和有效性上显著优于先前的轻量级描述生成模型，展示了其作为即插即用解决方案的潜力。源代码可在https://github.com/taewhankim/VIPCAP获取。
英文摘要：
Recent lightweight image captioning models using retrieved data mainly focus on text prompts. However, previous works only utilize the retrieved text as text prompts, and the visual information relies only on the CLIP visual embedding. Because of this issue, there is a limitation that the image descriptions inherent in the prompt are not sufficiently reflected in the visual embedding space. To tackle this issue, we propose ViPCap, a novel retrieval text-based visual prompt for lightweight image captioning. ViPCap leverages the retrieved text with image information as visual prompts to enhance the ability of the model to capture relevant visual information. By mapping text prompts into the CLIP space and generating multiple randomized Gaussian distributions, our method leverages sampling to explore randomly augmented distributions and effectively retrieves the semantic features that contain image information. These retrieved features are integrated into the image and designated as the visual prompt, leading to performance improvements on the datasets such as COCO, Flickr30k, and NoCaps. Experimental results demonstrate that ViPCap significantly outperforms prior lightweight captioning models in efficiency and effectiveness, demonstrating the potential for a plug-and-play solution. The source code is available at https://github.com/taewhankim/VIPCAP.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
