用GPT-4增强CLIP：利用视觉描述作为提示
Enhancing CLIP with GPT-4: Harnessing Visual Descriptions as Prompts
AI总结：
本文提出利用GPT-4生成视觉描述性文本作为提示来增强CLIP的零样本迁移能力，并设计少样本适配器选择最优句子构建分类器，在多个细粒度数据集上显著超越默认提示和CoCoOP方法。
摘要：
像CLIP这样的对比预训练大型视觉-语言模型（VLM）通过在下游数据集上提供良好性能，彻底改变了视觉表示学习。VLM通过设计与数据集相关的提示，以零样本方式适配下游数据集。这种提示工程利用了领域专业知识和验证数据集。与此同时，生成式预训练模型（如GPT-4）的最新发展意味着它们可以被用作高级互联网搜索工具。它们还可以被操控以任何结构提供视觉信息。在这项工作中，我们展示了GPT-4可用于生成具有视觉描述性的文本，以及如何利用这些文本将CLIP适配到下游任务。我们展示了与CLIP的默认提示相比，在专门的细粒度数据集（如EuroSAT（约7%）、DTD（约7%）、SUN397（约4.6%）和CUB（约3.3%））上零样本迁移准确率的显著提升。我们还设计了一个简单的少样本适配器，该适配器学习选择最佳句子来构建可泛化的分类器，其平均性能比最近提出的CoCoOP高出约2%，在4个专门的细粒度数据集上高出超过4%。代码、提示和辅助文本数据集可在https://github.com/mayug/VDT-Adapter获取。
英文摘要：
Contrastive pretrained large Vision-Language Models (VLMs) like CLIP have revolutionized visual representation learning by providing good performance on downstream datasets. VLMs are 0-shot adapted to a downstream dataset by designing prompts that are relevant to the dataset. Such prompt engineering makes use of domain expertise and a validation dataset. Meanwhile, recent developments in generative pretrained models like GPT-4 mean they can be used as advanced internet search tools. They can also be manipulated to provide visual information in any structure. In this work, we show that GPT-4 can be used to generate text that is visually descriptive and how this can be used to adapt CLIP to downstream tasks. We show considerable improvements in 0-shot transfer accuracy on specialized fine-grained datasets like EuroSAT (~7%), DTD (~7%), SUN397 (~4.6%), and CUB (~3.3%) when compared to CLIP's default prompt. We also design a simple few-shot adapter that learns to choose the best possible sentences to construct generalizable classifiers that outperform the recently proposed CoCoOP by ~2% on average and by over 4% on 4 specialized fine-grained datasets. The code, prompts, and auxiliary text dataset is available at https://github.com/mayug/VDT-Adapter.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
