面向生成式预训练视觉-语言模型的p-Laplacian适配方法
p-Laplacian Adaptation for Generative Pre-trained Vision-Language Models
AI总结：
该研究将VLM注意力后适配器调优重构为图消息传递过程，针对异配图问题提出基于p-Laplacian的p-adapter架构，在多类VLM和多模态任务上验证了其优于其他参数高效迁移学习方法的性能。
摘要：
在大规模语料上预训练的视觉-语言模型（VLM）已在诸多下游任务中取得显著成效。鉴于预训练VLM的规模快速增长，参数高效迁移学习（PETL）作为全量微调的可行替代方案受到广泛关注。适配器（adapter）是这类方法的一种，它在预训练模型中引入少量可训练参数，且适配过程中保留原始参数不变。本文提出一种全新的建模框架，将注意力后的适配器调优重构为注意力图上的图消息传递过程，其中投影后的查询与值特征分别构成节点特征，注意力矩阵构成图邻接矩阵。在该框架下，由于投影查询空间与值空间存在差异，VLM中的适配器调优需要处理异配图问题。为应对这一挑战，我们提出一种新的适配器架构p-adapter，它在图神经网络（GNN）中采用p-Laplacian消息传递机制。具体而言，注意力权重基于特征进行重归一化，随后使用校准后的注意力矩阵聚合特征，从而能够动态利用异配注意力图中不同频率的信息。我们在不同的预训练VLM和多模态任务上开展了大量实验，任务包括视觉问答、视觉蕴含和图像描述。实验结果证实，我们的方法相较于其他PETL方法具有显著优势。
英文摘要：
Vision-Language models (VLMs) pre-trained on large corpora have demonstrated notable success across a range of downstream tasks. In light of the rapidly increasing size of pre-trained VLMs, parameter-efficient transfer learning (PETL) has garnered attention as a viable alternative to full fine-tuning. One such approach is the adapter, which introduces a few trainable parameters into the pre-trained models while preserving the original parameters during adaptation. In this paper, we present a novel modeling framework that recasts adapter tuning after attention as a graph message passing process on attention graphs, where the projected query and value features and attention matrix constitute the node features and the graph adjacency matrix, respectively. Within this framework, tuning adapters in VLMs necessitates handling heterophilic graphs, owing to the disparity between the projected query and value space. To address this challenge, we propose a new adapter architecture, $p$-adapter, which employs $p$-Laplacian message passing in Graph Neural Networks (GNNs). Specifically, the attention weights are re-normalized based on the features, and the features are then aggregated using the calibrated attention matrix, enabling the dynamic exploitation of information with varying frequencies in the heterophilic attention graphs. We conduct extensive experiments on different pre-trained VLMs and multi-modal tasks, including visual question answering, visual entailment, and image captioning. The experimental results validate our method's significant superiority over other PETL methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
