利用大语言模型自动化零样本视觉识别的元提示方法
Meta-Prompting for Automating Zero-shot Visual Recognition with LLMs
AI总结：
本文提出MPVR方法，仅需任务简短描述和类别标签即可自动生成多样化类别特定提示，显著提升零样本图像识别性能。
摘要：
对大语言模型（LLM）生成的类别特定提示进行提示集成，已成为增强视觉-语言模型（VLM）零样本识别能力的有效方法。为了获得这些类别特定提示，现有方法依赖人工为LLM编写提示，以生成用于下游任务的VLM提示。然而，这需要手动组合这些任务特定提示，并且这些提示仍可能无法覆盖与感兴趣类别相关的多样化视觉概念和任务特定风格。为了有效地将人类从流程中移除并完全自动化零样本识别的提示生成过程，我们提出了面向视觉识别的元提示方法（MPVR）。MPVR仅以目标任务的简短自然语言描述和相关类别标签列表作为最小输入信息，即可自动生成多样化的类别特定提示，从而构建出强大的零样本分类器。在使用多个LLM和VLM进行测试时，MPVR能够有效泛化到属于广泛不同领域的各种流行零样本图像识别基准上。例如，利用GPT和Mixtral LLM，MPVR在CLIP基础上分别获得了最高19.8%和18.2%的零样本识别提升（在20个数据集上平均提升5.0%和4.5%）。
英文摘要：
Prompt ensembling of Large Language Model (LLM) generated category-specific prompts has emerged as an effective method to enhance zero-shot recognition ability of Vision-Language Models (VLMs). To obtain these category-specific prompts, the present methods rely on hand-crafting the prompts to the LLMs for generating VLM prompts for the downstream tasks. However, this requires manually composing these task-specific prompts and still, they might not cover the diverse set of visual concepts and task-specific styles associated with the categories of interest. To effectively take humans out of the loop and completely automate the prompt generation process for zero-shot recognition, we propose Meta-Prompting for Visual Recognition (MPVR). Taking as input only minimal information about the target task, in the form of its short natural language description, and a list of associated class labels, MPVR automatically produces a diverse set of category-specific prompts resulting in a strong zero-shot classifier. MPVR generalizes effectively across various popular zero-shot image recognition benchmarks belonging to widely different domains when tested with multiple LLMs and VLMs. For example, MPVR obtains a zero-shot recognition improvement over CLIP by up to 19.8% and 18.2% (5.0% and 4.5% on average over 20 datasets) leveraging GPT and Mixtral LLMs, respectively
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
