用于视觉-语言模型的多模态适配器
Multi-Modal Adapter for Vision-Language Models
AI总结：
针对现有CLIP轻量级适配方法分别处理视觉与文本模态、忽视跨模态交互的问题，提出Multi-Modal Adapter，通过可训练多头注意力融合图文特征并加性适配，提升了未见类别上的泛化性。
摘要：
诸如CLIP之类的大型预训练视觉-语言模型，已在广泛的图像分类任务上展示出最先进的性能，且无需重新训练。少样本CLIP与在下游任务上训练的现有专用架构相比具有竞争力。近期研究表明，使用轻量级适配方法可以进一步提升CLIP的性能。然而，先前方法分别对CLIP模型的不同模态进行适配，忽视了视觉表征与文本表征之间的交互和关系。在本工作中，我们提出Multi-Modal Adapter，这是一种用于CLIP多模态适配的方法。具体而言，我们加入一个可训练的Multi-Head Attention层，将文本特征与图像特征结合，以产生对二者的加性适配。基于其在未见过类别上的表现，与现有适配方法相比，Multi-Modal Adapter展现出更好的泛化能力。我们还进行了额外的消融实验和研究，以验证并解释所提出的方法。
英文摘要：
Large pre-trained vision-language models, such as CLIP, have demonstrated state-of-the-art performance across a wide range of image classification tasks, without requiring retraining. Few-shot CLIP is competitive with existing specialized architectures that were trained on the downstream tasks. Recent research demonstrates that the performance of CLIP can be further improved using lightweight adaptation approaches. However, previous methods adapt different modalities of the CLIP model individually, ignoring the interactions and relationships between visual and textual representations. In this work, we propose Multi-Modal Adapter, an approach for Multi-Modal adaptation of CLIP. Specifically, we add a trainable Multi-Head Attention layer that combines text and image features to produce an additive adaptation of both. Multi-Modal Adapter demonstrates improved generalizability, based on its performance on unseen classes compared to existing adaptation methods. We perform additional ablations and investigations to validate and interpret the proposed approach.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
