面向CLIP下游泛化的聚合-自适应自然语言提示
Aggregate-and-Adapt Natural Language Prompts for Downstream Generalization of CLIP
AI总结：
针对CLIP在专业领域、细粒度分类等场景下因概念表征不足泛化受限的问题，提出聚合-自适应提示嵌入AAPE，通过聚合自然语言提示知识并联合优化生成适配提示，在多类下游任务取得有竞争力性能，还降低了推理成本、提升了扩展性。
摘要：
CLIP等大型预训练视觉语言模型已展现出可观的泛化能力，但在专业领域（如卫星图像）或细粒度分类（如车型识别）场景中，若视觉概念在预训练阶段未出现或表征不足，模型性能可能会受限。提示学习提供了一种参数高效的微调框架，即便标注数据有限，也能让CLIP适配下游任务。本文通过从自然语言提示（人工生成或大语言模型生成）中提取文本知识来改进提示学习，为这些表征不足的概念提供丰富先验。我们首先通过一个可学习的提示聚合器，得到与每张输入图像对齐的提示“摘要”；随后联合训练一个提示生成器，优化目标是生成的提示嵌入既贴近聚合摘要，同时又能最小化任务损失。我们将这种提示嵌入命名为聚合-自适应提示嵌入（Aggregate-and-Adapted Prompt Embedding, AAPE）。实验表明，AAPE能够泛化到不同的下游数据分布和任务中，涵盖视觉语言理解任务（如少样本分类、视觉问答VQA）和生成任务（图像描述），并取得了具有竞争力的性能。我们还发现AAPE对处理非规范样本和分布外（OOD）样本尤其有效。此外，AAPE的学习过程消除了基线方法所需的基于大语言模型（LLM）的推理成本，且随数据规模和LLM模型规模增长的扩展性更好。
英文摘要：
Large pretrained vision-language models like CLIP have shown promising generalization capability, but may struggle in specialized domains (e.g., satellite imagery) or fine-grained classification (e.g., car models) where the visual concepts are unseen or under-represented during pretraining. Prompt learning offers a parameter-efficient finetuning framework that can adapt CLIP to downstream tasks even when limited annotation data are available. In this paper, we improve prompt learning by distilling the textual knowledge from natural language prompts (either human- or LLM-generated) to provide rich priors for those under-represented concepts. We first obtain a prompt ``summary'' aligned to each input image via a learned prompt aggregator. Then we jointly train a prompt generator, optimized to produce a prompt embedding that stays close to the aggregated summary while minimizing task loss at the same time. We dub such prompt embedding as Aggregate-and-Adapted Prompt Embedding (AAPE). AAPE is shown to be able to generalize to different downstream data distributions and tasks, including vision-language understanding tasks (e.g., few-shot classification, VQA) and generation tasks (image captioning) where AAPE achieves competitive performance. We also show AAPE is particularly helpful to handle non-canonical and OOD examples. Furthermore, AAPE learning eliminates LLM-based inference cost as required by baselines, and scales better with data and LLM model size.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
