arXiv 2309.136252023-09-24cs.CVcs.AI

GraphAdapter：利用双知识图谱微调视觉-语言模型

GraphAdapter: Tuning Vision-Language Models With Dual Knowledge Graph

University of Science and Technology of China（中国科学技术大学）；National University of Singapore（新加坡国立大学）；Tsinghua University（清华大学）

Xin Li, Dongze Lian, Zhihe Lu, Jiawang Bai, Zhibo Chen, Xinchao Wang

备注：: Accepted by NeurIPS 2023. The manuscript will be further revised based on the reviews

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出GraphAdapter，通过双知识图谱同时建模文本与视觉模态中的类间结构知识，以增强适配器微调视觉-语言模型的效果，并在11个基准数据集上显著超越已有方法。

摘要：

适配器风格的高效迁移学习（ETL）在低数据场景下的视觉-语言模型（VLM）微调中表现出色，其仅引入少量额外参数，基于VLM通用且强大的表示来挖掘任务特定知识。然而，大多数适配器风格的工作面临两个局限：（i）仅用单一模态建模任务特定知识；（ii）忽视了对下游任务中类间关系的利用，从而导致次优解。为缓解上述问题，我们提出了一种有效的适配器风格微调策略，名为GraphAdapter，它通过使用双知识图谱显式建模双模态结构知识（即文本和视觉模态中不同语义/类别的相关性）来执行文本适配器。具体而言，双知识图谱由两个子图构成，即文本知识子图和视觉知识子图，其中节点和边分别表示两种模态中的语义/类别及其相关性。这使得每个提示的文本特征能够同时利用来自文本和视觉模态的任务特定结构知识，从而为下游任务产生更有效的分类器。在11个基准数据集上的大量实验结果表明，我们的GraphAdapter显著优于以往基于适配器的方法。代码将发布在https://github.com/lixinustc/GraphAdapter。

英文摘要：

Adapter-style efficient transfer learning (ETL) has shown excellent performance in the tuning of vision-language models (VLMs) under the low-data regime, where only a few additional parameters are introduced to excavate the task-specific knowledge based on the general and powerful representation of VLMs. However, most adapter-style works face two limitations: (i) modeling task-specific knowledge with a single modality only; and (ii) overlooking the exploitation of the inter-class relationships in downstream tasks, thereby leading to sub-optimal solutions. To mitigate that, we propose an effective adapter-style tuning strategy, dubbed GraphAdapter, which performs the textual adapter by explicitly modeling the dual-modality structure knowledge (i.e., the correlation of different semantics/classes in textual and visual modalities) with a dual knowledge graph. In particular, the dual knowledge graph is established with two sub-graphs, i.e., a textual knowledge sub-graph, and a visual knowledge sub-graph, where the nodes and edges represent the semantics/classes and their correlations in two modalities, respectively. This enables the textual feature of each prompt to leverage the task-specific structure knowledge from both textual and visual modalities, yielding a more effective classifier for downstream tasks. Extensive experimental results on 11 benchmark datasets reveal that our GraphAdapter significantly outperforms previous adapter-based methods. The code will be released at https://github.com/lixinustc/GraphAdapter

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。