arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2601.01322cs.CVcs.AIcs.LGcs.MMeess.IV
收藏

LinMU: 使多模态理解线性化

LinMU: Multimodal Understanding Made Linear

Princeton University(普林斯顿大学)

Hongjie Wang, Niraj K. Jha

AI总结:

LinMU通过线性复杂度设计实现多模态理解,无需二次注意力模块,提升视频处理效率。

摘要:

现代视觉-语言模型(VLMs)在性能上取得了显著成就,但受到自注意力的二次复杂度的限制,这阻止了它们在边缘设备上的部署,并使它们对高分辨率图像和长上下文视频的理解变得极其昂贵。为了解决这一挑战,我们介绍了LinMU(线性复杂度多模态理解),一种VLM设计,它在不使用任何二次复杂度模块的情况下实现了线性复杂度,同时保持了基于全局注意力的VLMs的性能。LinMU将VLM中的每一个自注意力层替换为M-MATE块:一个双分支模块,结合了用于全局上下文的双向状态空间模型(Flex-MA分支)和用于相邻相关性的局部Swin式窗口注意力(Local-Swin分支)。为了将预训练的VLM转换为LinMU架构,我们提出了一个三阶段蒸馏框架,该框架(i)使用自注意力权重初始化两个分支并单独训练Flex-MA分支,(ii)解冻Local-Swin分支并联合训练Flex-MA分支和Local-Swin分支,(iii)解冻其余块并使用LoRA适配器进行微调,同时回归冻结VLM教师的隐藏状态和token级别logits。在MMMU、TextVQA、LongVideoBench、Video-MME等基准上,LinMU与教师模型的性能相当,但在分钟长度视频上将Time-To-First-Token(TTFT)减少高达2.7倍,并将token吞吐量提高高达9.0倍。消融实验确认了每个蒸馏阶段的重要性以及M-MATE块两个分支的必要性。所提出的框架证明了无需二次注意力即可实现最先进的多模态推理,从而为能够处理高分辨率图像和长视频的长上下文VLMs开辟了新途径。

英文摘要:

Modern Vision-Language Models (VLMs) achieve impressive performance but are limited by the quadratic complexity of self-attention, which prevents their deployment on edge devices and makes their understanding of high-resolution images and long-context videos prohibitively expensive. To address this challenge, we introduce LinMU (Linear-complexity Multimodal Understanding), a VLM design that achieves linear complexity without using any quadratic-complexity modules while maintaining the performance of global-attention-based VLMs. LinMU replaces every self-attention layer in the VLM with the M-MATE block: a dual-branch module that combines a bidirectional state-space model for global context (Flex-MA branch) with localized Swin-style window attention (Local-Swin branch) for adjacent correlations. To transform a pre-trained VLM into the LinMU architecture, we propose a three-stage distillation framework that (i) initializes both branches with self-attention weights and trains the Flex-MA branch alone, (ii) unfreezes the Local-Swin branch and fine-tunes it jointly with the Flex-MA branch, and (iii) unfreezes the remaining blocks and fine-tunes them using LoRA adapters, while regressing on hidden states and token-level logits of the frozen VLM teacher. On MMMU, TextVQA, LongVideoBench, Video-MME, and other benchmarks, LinMU matches the performance of teacher models, yet reduces Time-To-First-Token (TTFT) by up to 2.7$\times$ and improves token throughput by up to 9.0$\times$ on minute-length videos. Ablations confirm the importance of each distillation stage and the necessity of the two branches of the M-MATE block. The proposed framework demonstrates that state-of-the-art multimodal reasoning can be achieved without quadratic attention, thus opening up avenues for long-context VLMs that can deal with high-resolution images and long videos.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑