Modern Vision-Language Models (VLMs) achieve impressive performance but are limited by the quadratic complexity of self-attention, which prevents their deployment on edge devices and makes their understanding of high-resolution images and long-context videos prohibitively expensive. To address this challenge, we introduce LinMU (Linear-complexity Multimodal Understanding), a VLM design that achieves linear complexity without using any quadratic-complexity modules while maintaining the performance of global-attention-based VLMs. LinMU replaces every self-attention layer in the VLM with the M-MATE block: a dual-branch module that combines a bidirectional state-space model for global context (Flex-MA branch) with localized Swin-style window attention (Local-Swin branch) for adjacent correlations. To transform a pre-trained VLM into the LinMU architecture, we propose a three-stage distillation framework that (i) initializes both branches with self-attention weights and trains the Flex-MA branch alone, (ii) unfreezes the Local-Swin branch and fine-tunes it jointly with the Flex-MA branch, and (iii) unfreezes the remaining blocks and fine-tunes them using LoRA adapters, while regressing on hidden states and token-level logits of the frozen VLM teacher. On MMMU, TextVQA, LongVideoBench, Video-MME, and other benchmarks, LinMU matches the performance of teacher models, yet reduces Time-To-First-Token (TTFT) by up to 2.7$\times$ and improves token throughput by up to 9.0$\times$ on minute-length videos. Ablations confirm the importance of each distillation stage and the necessity of the two branches of the M-MATE block. The proposed framework demonstrates that state-of-the-art multimodal reasoning can be achieved without quadratic attention, thus opening up avenues for long-context VLMs that can deal with high-resolution images and long videos.
Advances in Multimodal Adaptation and Generalization: From Traditional Approaches to Foundation Models
Technical University of Munich(慕尼黑工业大学);Hong Kong Baptist University(香港浸会大学);Aalto University(阿尔托大学);University of Oulu(奥卢大学);University of Bonn(波恩大学);EPFL(洛桑联邦理工学院)
FedMLLM: Federated Fine-tuning MLLM on Multimodal Heterogeneity Data
Nanjing University of Science and Technology(南京理工大学);National University of Singapore(新加坡国立大学);Institute of High-Performance Computing, A*STAR(新加坡高性能计算研究院(新加坡科研局下属机构))
MCITlib: Multimodal Continual Instruction Tuning Library and Benchmark
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences(中国科学院大学先进交叉学科学院);State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所多模态人工智能系统国家重点实验室);Centre for Artificial Intelligence and Robotics, Hong Kong Institute of Science and Innovation, Chinese Academy of Sciences(中国科学院香港创新科学研究院人工智能与机器人中心);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Fujian Key Laboratory of Pattern Recognition and Image Understanding, School of Computer and Information Engineering, Xiamen University of Technology(福建 pattern recognition and image understanding 工程学院,厦门大学科技学院)
MERLIN: Multimodal Embedding Refinement via LLM-based Iterative Navigation for Text-Video Retrieval-Rerank Pipeline
Buzzni AI Lab(Buzzni人工智能实验室);KAIST(韩国科学技术院);Mathpresso Inc.(Mathpresso公司);UC Berkeley(加州大学伯克利分校);Fainders AI(Fainders人工智能公司);Seoul National University(首尔大学)
Evaluating Compositional Scene Understanding in Multimodal Generative Models
Department of Psychology, University of California, Los Angeles(加州大学洛杉矶分校心理学系);Microsoft Research, NYC(微软研究院纽约分部);Air Force Research Laboratory(美国空军研究实验室);Department of Statistics, University of California, Los Angeles(加州大学洛杉矶分校统计学系)
KARAMEL: Knowledge-Aware Ranking for Multimodal Entity Linking in the Arts via Heterogeneous Graphs
University of Bari Aldo Moro(巴里大学阿尔多·莫罗分校);Hasso Plattner Institute(哈索·普拉特纳研究所);University of Potsdam(波茨坦大学);ZBW - Leibniz Information Centre for Economics(ZBW - 莱布尼茨经济信息中心);University of Kiel(基尔大学)
E$^2$AT: Multimodal Jailbreak Defense via Dynamic Joint Optimization for Multimodal Large Language Models
School of Cyber Science and Engineering, Nanjing University of Science and Technology, China(南京理工大学信息科学与工程学院);HKUST(GZ) and INSAIT, Sofia University St. Kliment Ohridski(香港科技大学(广州)及INSAIT,索菲亚大学圣克莱门特·奥赫里迪斯学院);College of Computing and Data Science, Nanyang Technological University, Singapore(南洋理工大学计算与数据科学学院)
Multimodal Large Language Models for Text-rich Image Understanding: A Comprehensive Review
Meituan(美团);Shanghai Jiao Tong University(上海交通大学);Beijing Institute of Technology(北京理工大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所)
InfiR : Crafting Effective Small Language Models and Multimodal Small Language Models in Reasoning
Reallm Labs(Reallm实验室);The Hong Kong Polytechnic University(香港理工大学);Zhejiang University(浙江大学);South China University of Technology(华南理工大学);Harbin Institute of Technology(哈尔滨工业大学);Dalian University of Technology(大连理工大学);University of Electronic Science and Technology of China(电子科技大学);Beijing University of Posts and Telecommunications(北京邮电大学);TikTok(字节跳动旗下TikTok);Amazon(亚马逊)
Grounding Everything in Tokens for Multimodal Large Language Models
MoE Key Lab of Artificial Intelligence(人工智能混合专家模型关键实验室);AI Institute, Shanghai Jiao Tong University(上海交通大学人工智能研究院);Central Research Institute, Huawei(华为中央研究院)
MMSearch-OPD: Easy Made Hard, Hard Made Learnable for Long-Horizon Multimodal Search Agents
Peking University(北京大学);HKUST(GZ)(香港科技大学(广州));Tsinghua University(清华大学);The University of Hong Kong(香港大学);Zhejiang University(浙江大学);Huawei Cloud BU(华为云业务部)
HaploOmni: Unified Single Transformer for Multimodal Video Understanding and Generation
Tsinghua Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院,清华大学);ARC Lab, Tencent PCG(腾讯PCG ARC实验室);The University of Hong Kong(香港大学);Xi’an JiaoTong University(西安交通大学)
Multimodal Learning for Non-small Cell Lung Cancer Prognosis
Peng Cheng Laboratory(鹏城实验室);Xi’an Jiaotong University(西安交通大学);Shanghai AI Laboratory(上海人工智能实验室);Shenzhen Peini Digital Technology Co., Ltd.(深圳佩妮数字科技有限公司);University of Technology Sydney(悉尼科技大学)