利用无监督提示学习提升图像描述泛化能力
Improving Generalization of Image Captioning with Unsupervised Prompt Learning
AI总结：
本文提出无监督提示学习方法 GeneIC，通过 CLIP 对齐视觉与语言模态，从属性和语义一致性两方面优化领域特定提示向量，在不需标注数据的情况下提升图像描述模型的领域泛化能力。
摘要：
预训练的视觉-语言模型在配合手工设计的提示时，已在图像描述任务中展现出令人印象深刻的零样本能力。与此同时，手工设计的提示利用人类先验知识来引导模型。然而，由于不同领域之间存在差异，这种提供不变先验知识的手工提示可能导致某些领域出现模式坍缩。一些研究尝试引入专家知识和指令数据集，但结果成本高昂且会导致幻觉。在本文中，我们提出了一种无监督提示学习方法，用于提升图像描述的泛化能力（GeneIC），该方法无需标注数据即可为目标领域学习领域特定的提示向量。GeneIC 通过预训练的对比语言-图像预训练（CLIP）模型对齐视觉与语言模态，从而从属性和语义一致性两个方面优化领域特定的提示向量。具体而言，GeneIC 首先生成具有不同属性的属性迁移图像，同时保持与原始图像的语义相似性。然后，GeneIC 使用 CLIP 测量图像与生成句子之间的相似度。通过探索原始图像和属性迁移图像中的可变特征与不变特征，属性一致性约束图像和句子的属性变化方向，以学习领域特定知识。语义一致性直接测量生成句子与图像之间的相似度，以确保生成句子的准确性和全面性。因此，GeneIC 仅优化提示向量，有效保留了大模型中的知识并引入了领域特定知识。
英文摘要：
Pretrained visual-language models have demonstrated impressive zero-shot abilities in image captioning, when accompanied by hand-crafted prompts. Meanwhile, hand-crafted prompts utilize human prior knowledge to guide the model. However, due to the diversity between different domains, such hand-crafted prompt that provide invariant prior knowledge may result in mode collapse for some domains. Some researches attempted to incorporate expert knowledge and instruction datasets, but the results were costly and led to hallucinations. In this paper, we propose an unsupervised prompt learning method to improve Generalization of Image Captioning (GeneIC), which learns a domain-specific prompt vector for the target domain without requiring annotated data. GeneIC aligns visual and language modalities with a pre-trained Contrastive Language-Image Pre-Training (CLIP) model, thus optimizing the domain-specific prompt vector from two aspects: attribute and semantic consistency. Specifically, GeneIC first generates attribute-transferred images with differing attributes, while retaining semantic similarity with original images. Then, GeneIC uses CLIP to measure the similarity between the images and the generated sentences. By exploring the variable and invariant features in the original images and attribute-transferred images, attribute consistency constrains the attribute change direction of both images and sentences to learn domain-specific knowledge. The semantic consistency directly measures the similarity between the generated sentences and images to ensure the accuracy and comprehensiveness of the generated sentences. Consequently, GeneIC only optimizes the prompt vectors, which effectively retains the knowledge in the large model and introduces domain-specific knowledge.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
