arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.02745cs.CVcs.AIcs.CL
收藏

AVG-LLaVA:一种具有自适应视觉粒度的高效大型多模态模型

AVG-LLaVA: An Efficient Large Multimodal Model with Adaptive Visual Granularity

School of Informatics, Xiamen University, China(厦门大学信息学院);Pattern Recognition Center, WeChat AI, Tencent Inc, China(腾讯人工智能研究院);Key Laboratory of Digital Protection and Intelligent Processing of Intangible Cultural Heritage of Fujian and Taiwan (Xiamen University), Ministry of Culture and Tourism, China(福建省和台湾非物质文化遗产数字化保护与智能处理重点实验室);Shanghai Artificial Intelligence Laboratory, China(上海人工智能实验室)

Zhibin Lan, Liqiang Niu, Fandong Meng, Wenbo Li, Jie Zhou, Jinsong Su

AI总结:

AVG-LLaVA通过视觉粒度路由器自适应选择图像粒度,减少视觉令牌并加速推理,在11个基准上性能优越。

摘要:

近年来,大型多模态模型(LMMs)取得了显著进展。在处理高分辨率图像时,主流的大型多模态模型通常将图像划分为多个局部图像和一个全局图像,导致产生大量的视觉令牌。在这项工作中,我们引入了AVG-LLaVA,一种能够根据输入图像和指令自适应选择合适视觉粒度的大型多模态模型。具体而言,我们首先应用多个池化层来获取不同粒度下的视觉令牌。然后,我们提出了一种视觉粒度路由器,该路由器包含一个Transformer层、一个MLP层和一个投票层,用于根据图像和指令选择合适的视觉粒度。此外,我们提出了RGLF,一种新颖的训练范式,旨在使路由器预测的粒度与大型多模态模型的偏好对齐,而无需额外的人工标注数据。大量的实验和分析表明,AVG-LLaVA在11个基准测试中取得了优越的性能,同时显著减少了视觉令牌的数量并加速了推理(例如,在AI2D基准上,视觉令牌减少了85.3%,推理速度提升了2.53倍)。

英文摘要:

Recently, large multimodal models (LMMs) have achieved significant advancements. When dealing with high-resolution images, dominant LMMs typically divide them into multiple local images and a global image, leading to a large number of visual tokens. In this work, we introduce AVG-LLaVA, an LMM that can adaptively select the appropriate visual granularity based on the input image and instruction. Specifically, we first apply the multiple pooling layers to obtain visual tokens at different granularities. Then we propose a visual granularity router, which includes a Transformer layer, an MLP layer, and a voter layer, used to select the appropriate visual granularity based on the image and instruction. Furthermore, we put forward RGLF, a novel training paradigm that aims at aligning the granularity predicted by the router with the preferences of the LMM, without the need for additional manually annotated data. Extensive experiments and analysis show that AVG-LLaVA achieves superior performance across 11 benchmarks, as well as significantly reduces the number of visual tokens and speeds up inference (e.g., an 85.3% reduction in visual tokens and a 2.53$\times$ increase in inference speed on the AI2D benchmark).

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑