HeGraphAdapter：用异构图适配器微调多模态视觉-语言模型
HeGraphAdapter: Tuning Multi-Modal Vision-Language Models with Heterogeneous Graph Adapter
AI总结：
本文提出HeGraphAdapter，通过构建包含多模态节点与多种边连接的异构图并利用异构图神经网络挖掘结构知识，同时构建文本与视觉分类器，全面增强了CLIP模型在下游任务中的性能。
摘要：
基于适配器的微调方法在将预训练视觉-语言模型的知识迁移到下游任务方面展现出巨大潜力。然而，在审视现有适配器后，我们发现它们在构建任务特定知识时，通常未能充分探索不同模态之间的交互。此外，现有工作通常仅关注正文本提示之间的相似性匹配，这使得区分具有高相似视觉内容的类别变得困难。为了解决这些问题，本文提出一种新颖的异构图适配器，以实现对下游任务的VLMs微调。具体而言，我们首先构建一个统一的异构图模式，其中包含i）视觉节点、正文本节点和负文本节点，以及ii）多种类型的边连接，以共同全面建模模态内、模态间和类间结构知识。接下来，我们采用特定的异构图神经网络挖掘多模态结构知识，以适配下游任务的视觉和文本特征。最后，在HeGraphAdapter之后，我们同时构建基于文本和基于视觉的分类器，以全面增强CLIP模型的性能。在11个基准数据集上的实验结果证明了所提出的HeGraphAdapter的有效性和优势。
英文摘要：
Adapter-based tuning methods have shown significant potential in transferring knowledge from pre-trained Vision-Language Models to the downstream tasks. However, after reviewing existing adapters, we find they generally fail to fully explore the interactions between different modalities in constructing task-specific knowledge. Also, existing works usually only focus on similarity matching between positive text prompts, making it challenging to distinguish the classes with high similar visual contents. To address these issues, in this paper, we propose a novel Heterogeneous Graph Adapter to achieve tuning VLMs for the downstream tasks. To be specific, we first construct a unified heterogeneous graph mode, which contains i) visual nodes, positive text nodes and negative text nodes, and ii) several types of edge connections to comprehensively model the intra-modality, inter-modality and inter-class structure knowledge together. Next, we employ a specific Heterogeneous Graph Neural Network to excavate multi-modality structure knowledge for adapting both visual and textual features for the downstream tasks. Finally, after HeGraphAdapter, we construct both text-based and visual-based classifiers simultaneously to comprehensively enhance the performance of the CLIP model. Experimental results on 11 benchmark datasets demonstrate the effectiveness and benefits of the proposed HeGraphAdapter.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
