arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.00505cs.CVcs.AI
收藏

稀疏快捷键:促进多模态大语言模型中的高效融合

Sparse Shortcuts: Facilitating Efficient Fusion in Multimodal Large Language Models

Guangdong-Hong Kong-Macao Joint Laboratory for Emotion Intelligence and Pervasive Computing, Artificial Intelligence Research Institute, Shenzhen MSU-BIT University, Shenzhen(粤港澳大湾区情感智能与 pervasive 计算联合实验室,人工智能研究院,深圳MSU-BIT大学)

Jingrui Zhang, Feng Liang, Yong Zhang, Wei Wang, Runhao Zeng, Xiping Hu

AI总结:

SparseCut通过稀疏快捷连接和多粒度特征融合模块,提升多模态大语言模型在跨模态融合中的效率和性能。

摘要:

随着大语言模型(LLMs)在自然语言理解和生成中的显著成功,多模态大语言模型(MLLMs)在处理多模态数据的能力上迅速提升。尽管大多数现有研究集中在扩大语言模型的规模或构建更高质量的训练数据,但有限的注意力被投入到有效整合跨模态知识到语言空间中。例如,在视觉-语言模型中,仅使用高级视觉特征对齐模态往往会丢弃中等和低级特征中丰富的语义信息,限制了模型跨模态理解的能力。为了解决这个问题,我们提出了SparseCut,一种适用于MLLMs的通用跨模态融合架构,引入了跨模态编码器与LLM之间的稀疏快捷连接。这些快捷连接使模型能够高效且分层地整合不同层次的视觉特征,促进更丰富的语义融合,而无需增加计算开销。我们进一步引入了一个高效的多粒度特征融合模块,该模块在将特征路由到快捷连接之前执行视觉特征的融合。这保留了原始语言上下文,并且不增加整体输入长度,从而避免了对LLM的计算复杂性增加。实验表明,SparseCut在各种多模态基准测试中显著提升了MLLMs的性能,具有通用性和可扩展性,适用于不同的基础LLMs。

英文摘要:

With the remarkable success of large language models (LLMs) in natural language understanding and generation, multimodal large language models (MLLMs) have rapidly advanced in their ability to process data across multiple modalities. While most existing efforts focus on scaling up language models or constructing higher-quality training data, limited attention has been paid to effectively integrating cross-modal knowledge into the language space. In vision-language models, for instance, aligning modalities using only high-level visual features often discards the rich semantic information present in mid- and low-level features, limiting the model's ability of cross-modality understanding. To address this issue, we propose SparseCut, a general cross-modal fusion architecture for MLLMs, introducing sparse shortcut connections between the cross-modal encoder and the LLM. These shortcut connections enable the efficient and hierarchical integration of visual features at multiple levels, facilitating richer semantic fusion without increasing computational overhead. We further introduce an efficient multi-grained feature fusion module, which performs the fusion of visual features before routing them through the shortcuts. This preserves the original language context and does not increase the overall input length, thereby avoiding an increase in computational complexity for the LLM. Experiments demonstrate that SparseCut significantly enhances the performance of MLLMs across various multimodal benchmarks with generality and scalability for different base LLMs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑