APoLLo：面向视觉语言模型的统一适配器与提示学习
APoLLo: Unified Adapter and Prompt Learning for Vision Language Models
AI总结：
APoLLo提出统一适配器与提示学习的多模态方法，通过交叉注意力适配层和编码器一致性约束提升视觉语言模型少样本泛化能力，在新类别、跨数据集和域偏移任务上优于现有方法。
摘要：
输入文本提示的选择对CLIP等视觉语言预训练（VLP）模型的性能起着关键作用。我们提出APoLLo，一种统一的、将适配器（Adapter）与提示（Prompt）学习相结合用于视觉语言模型的多模态方法。我们的方法旨在显著提升VLP模型在少样本微调场景下的泛化能力。我们引入可训练的基于交叉注意力的适配器层，与视觉和语言编码器协同工作，以加强两种模态之间的对齐。我们强制各编码器分支（接收增强输入）之间保持一致，以防止在下游任务中过拟合。我们的方法在三个代表性任务上进行了评估：对新类别的泛化、跨数据集评估以及未见过域偏移。实际中，在10个多样化图像识别数据集的新类别上，APoLLo相对于MaPLe（SOTA）实现了最高6.03%的相对增益。
英文摘要：
The choice of input text prompt plays a critical role in the performance of Vision-Language Pretrained (VLP) models such as CLIP. We present APoLLo, a unified multi-modal approach that combines Adapter and Prompt learning for Vision-Language models. Our method is designed to substantially improve the generalization capabilities of VLP models when they are fine-tuned in a few-shot setting. We introduce trainable cross-attention-based adapter layers in conjunction with vision and language encoders to strengthen the alignment between the two modalities. We enforce consistency between the respective encoder branches (receiving augmented inputs) to prevent overfitting in downstream tasks. Our method is evaluated on three representative tasks: generalization to novel classes, cross-dataset evaluation, and unseen domain shifts. In practice, APoLLo achieves a relative gain up to 6.03% over MaPLe (SOTA) on novel classes for 10 diverse image recognition datasets.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
