arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2405.17815cs.CV
收藏

视觉锚点是多模态大语言模型的强信息聚合器

Visual Anchors Are Strong Information Aggregators For Multimodal Large Language Model

Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);ByteDance, Inc(字节跳动公司)

Haogeng Liu, Quanzeng You, Xiaotian Han, Yongfei Liu, Huaibo Huang, Ran He, Hongxia Yang

AI总结:

本研究针对多模态大语言模型中视觉-语言连接器研究不足的问题,发现Vision Transformer中的视觉锚点并提出提取算法,构建新型连接器AcFormer,在降低近三分之二计算成本的同时提升了性能。

摘要:

在多模态大语言模型(MLLMs)领域,视觉-语言连接器在连接预训练视觉编码器与大语言模型(LLMs)方面发挥着关键作用。尽管其重要性不言而喻,但视觉-语言连接器的相关研究相对较少。本研究旨在提出一种高性能的视觉-语言连接器,使MLLMs在保持低计算成本的同时实现高精度。我们首先揭示了Vision Transformer中视觉锚点的存在,并提出了一种低成本的搜索算法来提取这些锚点。基于这些发现,我们提出了Anchor Former(AcFormer),这是一种新型视觉-语言连接器,旨在利用预训练期间从这些视觉锚点中获得的丰富先验知识,引导信息聚合。通过大量实验,我们证明所提出的方法与基线相比显著降低了近三分之二的计算成本,同时性能优于基线方法,这凸显了AcFormer的有效性和高效性。代码可在https://github.com/liuhaogeng/Anchor-Former获取。

英文摘要:

In the realm of Multimodal Large Language Models (MLLMs), vision-language connector plays a crucial role to link the pre-trained vision encoders with Large Language Models (LLMs). Despite its importance, the vision-language connector has been relatively less explored. In this study, we aim to propose a strong vision-language connector that enables MLLMs to achieve high accuracy while maintain low computation cost. We first reveal the existence of the visual anchors in Vision Transformer and propose a cost-effective search algorithm to extract them. Building on these findings, we introduce the Anchor Former (AcFormer), a novel vision-language connector designed to leverage the rich prior knowledge obtained from these visual anchors during pretraining, guiding the aggregation of information. Through extensive experimentation, we demonstrate that the proposed method significantly reduces computational costs by nearly two-thirds compared with baseline, while simultaneously outperforming baseline methods. This highlights the effectiveness and efficiency of AcFormer. Codes are available at https://github.com/liuhaogeng/Anchor-Former.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑