arXiv 2402.108962024-06-01cs.CV

PaLM2-VAdapter：渐进对齐的语言模型构成强大的视觉-语言适配器

PaLM2-VAdapter: Progressively Aligned Language Model Makes a Strong Vision-language Adapter

Johns Hopkins University（约翰斯·霍普金斯大学）；Google Research（谷歌研究院）

Junfei Xiao, Zheng Xu, Alan Yuille, Shen Yan, Boyu Wang

备注：: Technical report, 15 pages; v2 fix typos, add additional results in appendix

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出PaLM2-VAdapter，采用渐进对齐的语言模型作为视觉-语言适配器，有效连接冻结视觉编码器与LLMs，在VQA和图像描述任务上以更少参数实现最先进的视觉理解与多模态推理能力。

摘要：

本文证明，渐进对齐的语言模型能够有效连接冻结的视觉编码器与大型语言模型（LLMs）。尽管视觉编码器和LLMs的基础架构与预训练方法已被广泛研究，但近期各研究中视觉-语言适配器的架构与训练策略差异显著。我们的研究对最先进的perceiver resampler架构进行了深入探索，并构建了一个强大的基线。然而，我们观察到，使用perceiver resampler进行视觉-语言对齐时存在收敛缓慢和可扩展性有限的问题，且缺乏直接监督。为解决此问题，我们提出了PaLM2-VAdapter，采用渐进对齐的语言模型作为视觉-语言适配器。与使用perceiver resampler的强大基线相比，我们的方法在经验上展现出更快的收敛速度、更高的性能和更强的可扩展性。在图像和视频上的各种视觉问答（VQA）和图像描述任务中进行的大量实验表明，我们的模型展现出最先进的视觉理解与多模态推理能力。值得注意的是，与最先进的大型视觉-语言模型相比，我们的方法以少30~70%的参数实现了这些进步，标志着显著的效率提升。

英文摘要：

This paper demonstrates that a progressively aligned language model can effectively bridge frozen vision encoders and large language models (LLMs). While the fundamental architecture and pre-training methods of vision encoders and LLMs have been extensively studied, the architecture and training strategy of vision-language adapters vary significantly across recent works. Our research undertakes a thorough exploration of the state-of-the-art perceiver resampler architecture and builds a strong baseline. However, we observe that the vision-language alignment with perceiver resampler exhibits slow convergence and limited scalability with a lack of direct supervision. To address this issue, we propose PaLM2-VAdapter, employing a progressively aligned language model as the vision-language adapter. Compared to the strong baseline with perceiver resampler, our method empirically shows faster convergence, higher performance, and stronger scalability. Extensive experiments across various Visual Question Answering (VQA) and captioning tasks on both images and videos demonstrate that our model exhibits state-of-the-art visual understanding and multi-modal reasoning capabilities. Notably, our method achieves these advancements with 30~70% fewer parameters than the state-of-the-art large vision-language models, marking a significant efficiency improvement.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。