UniAdapter：用于跨模态建模的统一参数高效迁移学习
UniAdapter: Unified Parameter-Efficient Transfer Learning for Cross-modal Modeling
AI总结：
本文提出 UniAdapter，通过统一单模态与多模态适配器并采用部分权重共享，在仅需1.0%-2.0%参数下实现视觉-语言模型的高效跨模态迁移，在6个基准上超越SOTA甚至全量微调。
摘要：
大规模视觉-语言预训练模型在各种下游任务中展现出良好的迁移能力。随着这些基础模型规模和下游任务数量的增长，标准的全量微调范式因沉重的计算和存储成本而变得不可持续。本文提出 UniAdapter，它统一了单模态和多模态适配器，用于在预训练视觉-语言模型上进行参数高效的跨模态适配。具体而言，适配器被分配到不同模态及其交互中，通过部分权重共享减少了可调参数总数。这种统一且知识共享的设计能够产生强大的跨模态表示，有益于各种下游任务，且仅需预训练模型 1.0%-2.0% 的可调参数。在 6 个跨模态下游基准（包括视频-文本检索、图像-文本检索、VideoQA 和 VQA）上的大量实验表明，在大多数情况下，UniAdapter 不仅优于当前最先进方法，甚至超越了全量微调策略。特别是在 MSRVTT 检索任务上，UniAdapter 以 2.2% 的模型参数量实现了 49.7% 的 recall@1，较最新竞争者提升了 2.0%。代码和模型可在 https://github.com/RERV/UniAdapter 获取。
英文摘要：
Large-scale vision-language pre-trained models have shown promising transferability to various downstream tasks. As the size of these foundation models and the number of downstream tasks grow, the standard full fine-tuning paradigm becomes unsustainable due to heavy computational and storage costs. This paper proposes UniAdapter, which unifies unimodal and multimodal adapters for parameter-efficient cross-modal adaptation on pre-trained vision-language models. Specifically, adapters are distributed to different modalities and their interactions, with the total number of tunable parameters reduced by partial weight sharing. The unified and knowledge-sharing design enables powerful cross-modal representations that can benefit various downstream tasks, requiring only 1.0%-2.0% tunable parameters of the pre-trained model. Extensive experiments on 6 cross-modal downstream benchmarks (including video-text retrieval, image-text retrieval, VideoQA, and VQA) show that in most cases, UniAdapter not only outperforms the state-of-the-arts, but even beats the full fine-tuning strategy. Particularly, on the MSRVTT retrieval task, UniAdapter achieves 49.7% recall@1 with 2.2% model parameters, outperforming the latest competitors by 2.0%. The code and models are available at https://github.com/RERV/UniAdapter.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
