arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2606.04719cs.CL
收藏

基于查询的跨模态投影器增强Mamba多模态大语言模型

Query-based Cross-Modal Projector Bolstering Mamba Multimodal LLM

Korea Advanced Institute of Science and Technology / Korea, Republic of(韩国科学技术院);University of Illinois in Urbana-Champaign / United States of America(伊利诺伊大学厄巴纳-香槟分校);Korea University / Korea, Republic of(韩国大学)

SooHwan Eom, Jay Shim, Gwanhyeong Koo, Haebin Na, Mark A. Hasegawa-Johnson, Sungwoong Kim, Chang D. Yoo

AI总结:

提出基于查询的跨模态投影器,通过交叉注意力压缩视觉令牌,消除手动设计2D扫描顺序的需求,提升Mamba多模态LLM的性能和吞吐量。

摘要:

Transformer的复杂度随输入长度呈二次增长,给大语言模型(LLM)带来了不可持续的计算负担。相比之下,选择性扫描结构化状态空间模型(即Mamba)有效解决了这一计算挑战。本文探索了一种基于查询的跨模态投影器,通过交叉注意力机制根据输入压缩视觉令牌,从而增强Mamba在视觉-语言建模中的效率。这种创新的投影器还消除了将原始图像特征转换为Mamba LLM输入序列时手动设计2D扫描顺序的需求。在各种视觉-语言理解基准上的实验结果表明,所提出的跨模态投影器增强了基于Mamba的多模态LLM,提升了性能和吞吐量。

英文摘要:

The Transformer's quadratic complexity with input length imposes an unsustainable computational load on large language models (LLMs). In contrast, the Selective Scan Structured State-Space Model, or Mamba, addresses this computational challenge effectively. This paper explores a query-based cross-modal projector designed to bolster Mamba's efficiency for vision-language modeling by compressing visual tokens based on input through the cross-attention mechanism. This innovative projector also removes the need for manually designing the 2D scan order of original image features when converting them into an input sequence for Mamba LLM. Experimental results across various vision-language understanding benchmarks show that the proposed cross-modal projector enhances Mamba-based multimodal LLMs, boosting both performance and throughput.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑