跨模态适配器：视觉-语言模型的参数高效迁移学习方法
Cross-Modal Adapter: Parameter-Efficient Transfer Learning Approach for Vision-Language Models
AI总结：
提出跨模态适配器XMAdapter，通过双模态缓存与检索实现参数高效迁移，动态融合并挖掘困难样本，在基准上显著提升准确率、泛化与效率。
摘要：
基于适配器的参数高效迁移学习在视觉-语言模型中已取得令人振奋的成果。传统适配器方法通常需要训练或微调，面临样本不足或资源受限等挑战。尽管一些方法通过利用图像模态缓存和检索来克服训练需求，但它们忽视了文本模态的重要性以及跨模态线索在视觉-语言模型参数高效适配中的作用。本工作引入了一种名为XMAdapter的跨模态参数高效方法。XMAdapter为文本和图像模态分别建立缓存模型，然后利用视觉-语言双模态信息进行检索以收集推理线索。通过动态调整亲和度比率，实现跨模态融合，解耦不同模态的相似性以评估它们各自的贡献。此外，它基于跨模态亲和度的差异探索困难样本，并通过自适应调整样本学习强度来增强模型性能。在基准数据集上的大量实验结果表明，XMAdapter在准确性、泛化性和效率方面显著优于以往的适配器方法。
英文摘要：
Adapter-based parameter-efficient transfer learning has achieved exciting results in vision-language models. Traditional adapter methods often require training or fine-tuning, facing challenges such as insufficient samples or resource limitations. While some methods overcome the need for training by leveraging image modality cache and retrieval, they overlook the text modality's importance and cross-modal cues for the efficient adaptation of parameters in visual-language models. This work introduces a cross-modal parameter-efficient approach named XMAdapter. XMAdapter establishes cache models for both text and image modalities. It then leverages retrieval through visual-language bimodal information to gather clues for inference. By dynamically adjusting the affinity ratio, it achieves cross-modal fusion, decoupling different modal similarities to assess their respective contributions. Additionally, it explores hard samples based on differences in cross-modal affinity and enhances model performance through adaptive adjustment of sample learning intensity. Extensive experimental results on benchmark datasets demonstrate that XMAdapter outperforms previous adapter-based methods significantly regarding accuracy, generalization, and efficiency.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
