面向视觉-语言模型的多模态属性提示
Multi-modal Attribute Prompting for Vision-Language Models
AI总结：
针对预训练视觉-语言模型少样本场景下忽略多模态属性、细粒度感知与泛化受限的问题，提出协同文本、视觉属性提示及属性级对齐的MAP方法，首次实现CLIP少样本适配的跨模态属性级对齐，在11个数据集上性能优于现有最优方法。
摘要：
CLIP等预训练视觉-语言模型（VLM）对下游任务展现出较强的泛化能力，但在少样本场景中表现不佳。现有提示技术主要聚焦于全局文本与图像表征，却忽略了多模态属性特征。这一局限性阻碍了模型感知细粒度视觉细节的能力，也限制了其对更广泛未见类别的泛化能力。为解决该问题，我们提出多模态属性提示方法（MAP），该方法协同探索文本属性提示、视觉属性提示以及属性级对齐。所提出的MAP具备多项优势：首先，我们引入经文本属性语义增强的可学习视觉属性提示，自适应捕获未知类别图像的视觉属性，提升CLIP的细粒度视觉感知能力；其次，所提出的属性级对齐对全局对齐形成补充，增强了开放词汇对象跨模态对齐的鲁棒性。据我们所知，这是首项为基于CLIP的少样本适配建立跨模态属性级对齐的研究。在11个数据集上的大量实验结果表明，该方法性能优于现有最优方法。
英文摘要：
Pre-trained Vision-Language Models (VLMs), like CLIP, exhibit strong generalization ability to downstream tasks but struggle in few-shot scenarios. Existing prompting techniques primarily focus on global text and image representations, yet overlooking multi-modal attribute characteristics. This limitation hinders the model's ability to perceive fine-grained visual details and restricts its generalization ability to a broader range of unseen classes. To address this issue, we propose a Multi-modal Attribute Prompting method (MAP) by jointly exploring textual attribute prompting, visual attribute prompting, and attribute-level alignment. The proposed MAP enjoys several merits. First, we introduce learnable visual attribute prompts enhanced by textual attribute semantics to adaptively capture visual attributes for images from unknown categories, boosting fine-grained visual perception capabilities for CLIP. Second, the proposed attribute-level alignment complements the global alignment to enhance the robustness of cross-modal alignment for open-vocabulary objects. To our knowledge, this is the first work to establish cross-modal attribute-level alignment for CLIP-based few-shot adaptation. Extensive experimental results on 11 datasets demonstrate that our method performs favorably against state-of-the-art approaches.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
