arXiv 2410.059382024-10-08cs.CVcs.AI

EMMA：通过结构对齐与层级对齐赋能多模态Mamba

EMMA: Empowering Multi-modal Mamba with Structural and Hierarchical Alignment

Peng Cheng Laboratory（鹏城实验室）；University of Chinese Academy of Sciences（中国科学院大学）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Sun Yat-sen University（中山大学）

Yifei Xing, Xiangyuan Lan, Ruiping Wang, Dongmei Jiang, Wenjun Huang, Qingfang Zheng, Yaowei Wang

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对现有Mamba多模态大语言模型视觉特征提取不足、跨模态对齐失衡的问题，提出EMMA方法，通过逐像素对齐模块和多尺度特征融合模块实现结构与层级对齐，推理速度更快、幻觉更少，在多模态基准上性能更优。

摘要：

基于Mamba的架构凭借其颇具竞争力的性能与亚二次方部署速度，已成为深度学习模型领域颇具前景的新方向。然而，当前的Mamba多模态大语言模型（MLLM）在视觉特征提取方面存在不足，导致视觉潜在表征与文本潜在表征之间的跨模态对齐失衡，对多模态任务的性能造成负面影响。在本研究中，我们提出了“通过结构对齐与层级对齐赋能多模态Mamba（EMMA）”方法，该方法可让MLLM提取细粒度视觉信息。具体而言，我们提出了一个逐像素对齐模块，用于自回归地优化空间图像级特征与文本token的学习及处理，实现图像层面的结构对齐。此外，为防止跨模型对齐过程中视觉信息退化，我们提出了多尺度特征融合（MFF）模块，用于结合来自中间层的多尺度视觉特征，实现特征层面的层级对齐。我们在多种多模态基准上开展了大量实验。我们的模型延迟低于其他基于Mamba的MLLM，推理速度比同等规模的基于transformer的MLLM快近四倍。得益于更优的跨模态对齐效果，我们的模型幻觉程度更低，对视觉细节的敏感度更高，在各类多模态基准上均表现出更优异的性能。代码将随后提供。

英文摘要：

Mamba-based architectures have shown to be a promising new direction for deep learning models owing to their competitive performance and sub-quadratic deployment speed. However, current Mamba multi-modal large language models (MLLM) are insufficient in extracting visual features, leading to imbalanced cross-modal alignment between visual and textural latents, negatively impacting performance on multi-modal tasks. In this work, we propose Empowering Multi-modal Mamba with Structural and Hierarchical Alignment (EMMA), which enables the MLLM to extract fine-grained visual information. Specifically, we propose a pixel-wise alignment module to autoregressively optimize the learning and processing of spatial image-level features along with textual tokens, enabling structural alignment at the image level. In addition, to prevent the degradation of visual information during the cross-model alignment process, we propose a multi-scale feature fusion (MFF) module to combine multi-scale visual features from intermediate layers, enabling hierarchical alignment at the feature level. Extensive experiments are conducted across a variety of multi-modal benchmarks. Our model shows lower latency than other Mamba-based MLLMs and is nearly four times faster than transformer-based MLLMs of similar scale during inference. Due to better cross-modal alignment, our model exhibits lower degrees of hallucination and enhanced sensitivity to visual details, which manifests in superior performance across diverse multi-modal benchmarks. Code will be provided.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。