视觉语言模型的属性树提示学习
Tree of Attributes Prompt Learning for Vision-Language Models
AI总结：
提出属性树提示学习（TAP），利用大语言模型生成结构化属性树并学习层级视觉与文本提示，以解决类别名称上下文利用不足问题，在零样本泛化、跨数据集迁移和少样本分类上超越现有方法。
摘要：
提示学习已被证明在使视觉语言模型适应下游任务方面是有效的。然而，现有方法通常仅将可学习的提示令牌与类别名称拼接以获取文本特征，这未能充分利用类别名称中所蕴含的丰富上下文。为解决这一问题，我们提出了属性树提示学习（TAP），该方法首先指示大语言模型为每个类别生成具有“概念-属性-描述”结构的属性树，然后利用视觉和文本提示令牌学习该层级结构。与现有方法仅用一组非结构化描述来增强类别名称不同，我们的方法实质上从大语言模型中提炼出与类别名称相关联的结构化知识图谱。此外，我们的方法引入了专门设计用于显式学习相应视觉属性的文本和视觉提示，有效充当领域专家。另外，基于类别名称生成的通用且多样的描述在特定给定图像中可能是错误的或缺失的。为解决这种不对齐问题，我们进一步引入了一个视觉条件池化模块来提取实例特定的文本特征。大量实验结果表明，我们的方法在11个不同数据集上的零样本基类到新类泛化、跨数据集迁移以及少样本分类任务中均优于现有最先进方法。代码可在 https://github.com/HHenryD/TAP 获取。
英文摘要：
Prompt learning has proven effective in adapting vision language models for downstream tasks. However, existing methods usually append learnable prompt tokens solely with the category names to obtain textual features, which fails to fully leverage the rich context indicated in the category name. To address this issue, we propose the Tree of Attributes Prompt learning (TAP), which first instructs LLMs to generate a tree of attributes with a "concept - attribute - description" structure for each category, and then learn the hierarchy with vision and text prompt tokens. Unlike existing methods that merely augment category names with a set of unstructured descriptions, our approach essentially distills structured knowledge graphs associated with class names from LLMs. Furthermore, our approach introduces text and vision prompts designed to explicitly learn the corresponding visual attributes, effectively serving as domain experts. Additionally, the general and diverse descriptions generated based on the class names may be wrong or absent in the specific given images. To address this misalignment, we further introduce a vision-conditional pooling module to extract instance-specific text features. Extensive experimental results demonstrate that our approach outperforms state-of-the-art methods on the zero-shot base-to-novel generalization, cross-dataset transfer, as well as few-shot classification across 11 diverse datasets. Code is available at https://github.com/HHenryD/TAP.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
