MaTVLM：用于高效视觉-语言建模的混合 Mamba-Transformer
MaTVLM: Hybrid Mamba-Transformer for Efficient Vision-Language Modeling
AI总结：
本文提出混合 Mamba-Transformer 模型 MaTVLM，通过 Mamba-2 层替换、注意力权重初始化和单阶段蒸馏，在保持 VLM 性能的同时显著提升推理效率并降低显存占用。
摘要：
随着具有线性复杂度的 RNN 模型的发展，transformer 的二次复杂度挑战有望被克服。值得注意的是，新兴的 Mamba-2 已展示出具有竞争力的性能，弥合了 RNN 模型与 transformer 之间的差距。然而，由于顺序处理和梯度消失，RNN 模型难以捕捉长程依赖，从而限制了上下文理解。这导致收敛缓慢、资源需求高，并且在下游理解和复杂推理任务上性能较差。在本工作中，我们提出了一种混合模型 MaTVLM，方法是用 Mamba-2 层替换预训练 VLM 中的一部分 transformer 解码器层。利用注意力与 Mamba-2 之间的内在关系，我们用相应的注意力权重初始化 Mamba-2，以加速收敛。随后，我们采用单阶段蒸馏过程，使用预训练 VLM 作为教师模型，将知识迁移给 MaTVLM，从而进一步提升收敛速度和性能。此外，我们研究了训练框架中差分蒸馏损失的影响。我们在多个基准上评估 MaTVLM，结果表明其相对于教师模型和现有 VLM 具有竞争力，同时优于基于 Mamba 的 VLM 以及参数规模相当的模型。值得注意的是，MaTVLM 的推理速度最高可达教师模型的 3.6 倍，并将 GPU 内存消耗降低 27.5%，且所有这些都不会损失性能。代码和模型已发布于 http://github.com/hustvl/MaTVLM。
英文摘要：
With the advancement of RNN models with linear complexity, the quadratic complexity challenge of transformers has the potential to be overcome. Notably, the emerging Mamba-2 has demonstrated competitive performance, bridging the gap between RNN models and transformers. However, due to sequential processing and vanishing gradients, RNN models struggle to capture long-range dependencies, limiting contextual understanding. This results in slow convergence, high resource demands, and poor performance on downstream understanding and complex reasoning tasks. In this work, we present a hybrid model MaTVLM by substituting a portion of the transformer decoder layers in a pre-trained VLM with Mamba-2 layers. Leveraging the inherent relationship between attention and Mamba-2, we initialize Mamba-2 with corresponding attention weights to accelerate convergence. Subsequently, we employ a single-stage distillation process, using the pre-trained VLM as the teacher model to transfer knowledge to the MaTVLM, further enhancing convergence speed and performance. Furthermore, we investigate the impact of differential distillation loss within our training framework. We evaluate the MaTVLM on multiple benchmarks, demonstrating competitive performance against the teacher model and existing VLMs while surpassing both Mamba-based VLMs and models of comparable parameter scales. Remarkably, the MaTVLM achieves up to 3.6x faster inference than the teacher model while reducing GPU memory consumption by 27.5%, all without compromising performance. Code and models are released at http://github.com/hustvl/MaTVLM.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
