Sparse Shortcuts: Facilitating Efficient Fusion in Multimodal Large Language Models
Guangdong-Hong Kong-Macao Joint Laboratory for Emotion Intelligence and Pervasive Computing, Artificial Intelligence Research Institute, Shenzhen MSU-BIT University, Shenzhen(粤港澳大湾区情感智能与 pervasive 计算联合实验室,人工智能研究院,深圳MSU-BIT大学)
With the remarkable success of large language models (LLMs) in natural language understanding and generation, multimodal large language models (MLLMs) have rapidly advanced in their ability to process data across multiple modalities. While most existing efforts focus on scaling up language models or constructing higher-quality training data, limited attention has been paid to effectively integrating cross-modal knowledge into the language space. In vision-language models, for instance, aligning modalities using only high-level visual features often discards the rich semantic information present in mid- and low-level features, limiting the model's ability of cross-modality understanding. To address this issue, we propose SparseCut, a general cross-modal fusion architecture for MLLMs, introducing sparse shortcut connections between the cross-modal encoder and the LLM. These shortcut connections enable the efficient and hierarchical integration of visual features at multiple levels, facilitating richer semantic fusion without increasing computational overhead. We further introduce an efficient multi-grained feature fusion module, which performs the fusion of visual features before routing them through the shortcuts. This preserves the original language context and does not increase the overall input length, thereby avoiding an increase in computational complexity for the LLM. Experiments demonstrate that SparseCut significantly enhances the performance of MLLMs across various multimodal benchmarks with generality and scalability for different base LLMs.
Sorbonne Université(索邦大学);CNRS(法国国家科学研究中心);LIP6(巴黎第六大学计算机科学实验室);Carney Institute for Brain Science, Brown University(布朗大学卡尼脑科学研究所);Idiap Research Institute(伊迪亚普研究所);Australian Institute for Machine Learning, University of Adelaide(阿德莱德大学澳大利亚机器学习研究所)
Seeing Time: Benchmarking Chronological Reasoning and Shortcut Biases in Vision-Language Models
College of Computer Science and Technology, Jilin University(吉林大学计算机科学与技术学院);College of Computing and Data Science, Nanyang Technological University(南洋理工大学计算与数据科学学院);School of Computer Science, Wuhan University(武汉大学计算机学院);School of Computing Technologies, RMIT University(皇家墨尔本理工学院计算技术学院)
PunchBench: Benchmarking MLLMs in Multimodal Punchline Comprehension
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University(多媒体信息处理国家重点实验室,计算机学院,北京大学);WeChat AI, Tencent Inc., China(微信AI,腾讯公司,中国)
MedQA-MM: Shortcuts Behind Medical Visual Reasoning
University of Massachusetts Amherst(马萨诸塞大学阿默斯特分校);Yale University(耶鲁大学);University of Massachusetts Lowell(马萨诸塞大学洛厄尔分校);VA Bedford Healthcare System(贝斯以色列女执事医疗中心贝德福德分部);Qingdao Medical College of Qingdao University(青岛大学青岛医学院);Yale School of Medicine(耶鲁医学院);National University Hospital, Singapore(新加坡国立大学医院);Zhejiang University(浙江大学);Stanford University(斯坦福大学)
MicroVQA: A Multimodal Reasoning Benchmark for Microscopy-Based Scientific Research
Stanford University(斯坦福大学);Tsinghua University(清华大学);University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校);Princeton University(普林斯顿大学);KTH Royal Institute of Technology(瑞典皇家理工学院);Chan Zuckerberg Biohub Network(陈·扎克翰生物中心网络)
Can Multimodal LLMs See Science Instruction? Benchmarking Pedagogical Reasoning in K-12 Classroom Videos
Department of Computer Science, Drexel University(德雷塞尔大学计算机科学系);Department of Teaching and Learning, Washington State University(华盛顿州立大学教学与学习系);College of Chemistry, Beijing Normal University(北京师范大学化学学院);Department of Computer Science, University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校计算机科学系);Department of Data Science, City University of Hong Kong(香港城市大学数据科学系)
Tsinghua University(清华大学);Meituan Inc(美团公司);Central South University(中南大学);Peking University(北京大学);University of Chinese Academy of Sciences(中国科学院大学);Nanyang Technological University(南洋理工大学)
Offline-Online Curriculum RL for Multimodal Reasoning
Beijing University of Posts and Telecommunications(北京邮电大学);BUPT Shenzhen Institute(北京邮电大学深圳研究院);Carnegie Mellon University(卡内基梅隆大学);Lancaster University(兰卡斯特大学);Nanyang Technological University(南洋理工大学);China Telecom Corporation Limited Sichuan Branch(中国电信股份有限公司四川分公司);Changsha University of Science & Technology(长沙理工大学)
Learning to Reason About Implicit Multimodal Harm: From Diagnosis to Generalization
Max Planck Institute for Informatics(马克斯·普朗克院信息研究所);Saarland Informatics Campus(萨尔兰州信息校园);Saarland University(萨尔兰州大学);The University of Edinburgh(爱丁堡大学);Samsung AI Center, Cambridge(三星AI中心,剑桥)
Sun Yat-sen University(中山大学);Shenzhen Loop Area Institute(深圳河套学院);Harbin Institute of Technology (HIT)(哈尔滨工业大学);Fudan University(复旦大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);The Chinese University of Hong Kong(香港中文大学)
Attend to Evidence: Evidence-Anchored Spatial Attention Supervision for Multimodal RLVR
Harbin Institute of Technology(哈尔滨工业大学);Zhongguancun Academy(中关村学院);Zhongguancun Institute of Artificial Intelligence(中关村人工智能研究院);Nankai University(南开大学);Shanghai Jiaotong University(上海交通大学);Zhejiang University(浙江大学)
Preliminary Investigations of a Multi-Faceted Robust and Synergistic Approach in Semiconductor Electron Micrograph Analysis: Integrating Vision Transformers with Large Language and Multimodal Models
To See is Not to Learn: Protecting Multimodal Data from Unauthorized Fine-Tuning of Large Vision-Language Model
School of Computing;Augmented Intelligence, Arizona State University, Tempe, AZ, USA;Department of Computer Science;Engineering, Texas A\&M University, College Station, TX, USA
MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI
Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Shanghai Jiao Tong University(上海交通大学);The University of Hong Kong(香港大学);The University of Adelaide(阿德莱德大学);Zhejiang University(浙江大学);Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院)
Teaching the Way, Not the Answer: Privileged Tutoring Distillation for Multimodal Policy Optimization
Tianjin University(天津大学);Beijing Institute of Technology(北京理工大学);Singapore Management University(新加坡国立大学);University of Chinese Academy of Sciences(中国科学院大学);Xiaomi Inc(小米公司)
Seeing Before Reasoning: Decoupling Perception and Reasoning for Shortcut-Resilient Multimodal On-Policy Self-Distillation
State Key Laboratory of Robotics and Intelligent Systems, Shenyang Institute of Automation, Chinese Academy of Sciences(机器人与智能系统国家重点实验室,沈阳自动化研究所,中国科学院);University of Chinese Academy of Sciences(中国科学院大学)
Beyond Accuracy: Evaluating Visual Grounding In Multimodal Medical Reasoning
The University of Texas MD Anderson Cancer Center(德克萨斯大学MD安德森癌症中心);Cohere Labs(Cohere实验室);Eisai Inc.(艾伯维公司);Indian Institute of Science, Bangalore(班加罗尔印度科学研究院)