RMAdapter: 基于重建的多模态适配器用于视觉-语言模型
RMAdapter: Reconstruction-based Multi-Modal Adapter for Vision-Language Models
AI总结：
RMAdapter通过双分支架构平衡通用与任务特定知识，提升视觉-语言模型在多模态迁移学习中的性能。
摘要：
预训练的视觉-语言模型（VLMs），例如CLIP，已成为多模态迁移学习中的关键工具。然而，在少样本场景中对VLMs进行微调面临显著挑战，即在获得的模型中平衡任务特定适应与泛化能力。同时，当前研究主要集中在基于提示的适应方法上，使基于适配器的方法被忽视，并揭示了明显的性能差距。为了解决这些挑战，我们引入了一种新的基于重建的多模态适配器（RMAdapter），它利用双分支架构。与传统的单分支适配器不同，RMAdapter包括：（1）一个适应分支，通过参数高效微调注入任务特定知识；（2）一个重建分支，通过将潜在空间特征重新映射回原始特征空间来保留通用知识。这种设计促进了通用知识与任务特定知识之间的动态平衡。重要的是，尽管RMAdapter引入了一个额外的重建分支，但它是精心优化以保持轻量的。通过在每一层本地计算重建损失并共享投影模块，总体计算开销被保持最小。还纳入了一致性约束，以更好地调节可区分性和泛化之间的权衡。我们全面评估了RMAdapter在三个代表性任务上的有效性：向新类别泛化、向新目标数据集泛化和领域泛化。不依赖数据增强或重复提示设计，我们的RMAdapter在所有评估指标上都优于最先进的方法。
英文摘要：
Pre-trained Vision-Language Models (VLMs), \textit{e.g.} CLIP, have become essential tools in multimodal transfer learning. However, fine-tuning VLMs in few-shot scenarios poses significant challenges in balancing task-specific adaptation and generalization in the obtained model. Meanwhile, current researches have predominantly focused on prompt-based adaptation methods, leaving adapter-based approaches underexplored and revealing notable performance gaps. To address these challenges, we introduce a novel Reconstruction-based Multimodal Adapter (RMAdapter), which leverages a dual-branch architecture. Unlike conventional single-branch adapters, RMAdapter consists of: (1) an adaptation branch that injects task-specific knowledge through parameter-efficient fine-tuning, and (2) a reconstruction branch that preserves general knowledge by reconstructing latent space features back into the original feature space. This design facilitates a dynamic balance between general and task-specific knowledge. Importantly, although RMAdapter introduces an additional reconstruction branch, it is carefully optimized to remain lightweight. By computing reconstruction loss locally at each layer and sharing projection modules, the overall computational overhead is kept minimal. A consistency constraint is also incorporated to better regulate the trade-off between discriminability and generalization. We comprehensively evaluate the effectiveness of RMAdapter on three representative tasks: generalization to new categories, generalization to new target datasets, and domain generalization. Without relying on data augmentation or duplicate prompt designs, our RMAdapter consistently outperforms state-of-the-art approaches across all evaluation metrics.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
