Large Multimodal Models (LMMs) have shown significant visual reasoning capabilities by connecting a visual encoder and a large language model. LMMs typically take in a fixed and large amount of visual tokens, such as the penultimate layer features in the CLIP visual encoder, as the prefix content. Recent LMMs incorporate more complex visual inputs, such as high-resolution images and videos, which further increases the number of visual tokens significantly. However, due to the inherent design of the Transformer architecture, the computational costs of these models tend to increase quadratically with the number of input tokens. To tackle this problem, we explore a token reduction mechanism that identifies significant spatial redundancy among visual tokens. In response, we propose PruMerge, a novel adaptive visual token reduction strategy that significantly reduces the number of visual tokens without compromising the performance of LMMs. Specifically, to metric the importance of each token, we exploit the sparsity observed in the visual encoder, characterized by the sparse distribution of attention scores between the class token and visual tokens. This sparsity enables us to dynamically select the most crucial visual tokens to retain. Subsequently, we cluster the selected (unpruned) tokens based on their key similarity and merge them with the unpruned tokens, effectively supplementing and enhancing their informational content. Empirically, when applied to LLaVA-1.5, our approach can compress the visual tokens by 14 times on average, and achieve comparable performance across diverse visual question-answering and reasoning tasks. Code and checkpoints are at https://llava-prumerge.github.io/.
LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Token
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Chinese Academy of Sciences（中国科学院）；University of Chinese Academy of Sciences（中国科学院大学）
WSI-LLaVA: A Multimodal Large Language Model for Whole Slide Image
Shenzhen University（深圳大学）；University of Nottingham Ningbo China（诺丁汉大学宁波分校）；City University of Hong Kong（香港城市大学）；Stanford University（斯坦福大学）；Hong Kong University of Science and Technology（香港科学与技术大学）
LLaVA-Docent: Instruction Tuning with Multimodal Large Language Model to Support Art Appreciation Education
Korea University（高丽大学）；University of Nebraska-Lincoln（内布拉斯加大学林肯分校）；Seoul Metropolitan Office of Education（首尔市教育厅）；Seoul National University（首尔国立大学）；Seoul National University of Education（首尔国立教育大学）
LLaVA-Ultra: Large Chinese Language and Vision Assistant for Ultrasound
ZJU-UIUC Institute, Zhejiang University（浙江大学ZJU-UIUC学院）；University of Washington（华盛顿大学）；Zhejiang University School of Medicine Sir Run Run Shaw Hospital（浙江大学医学院附属邵逸夫医院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
ReGraP-LLaVA: Reasoning enabled Graph-based Personalized Large Language and Vision Assistant
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究所）；School of Engineering, Westlake University（西湖大学工程学院）；University of Minnesota – Twin Cities（明尼苏达大学双城分校）；University of Toronto（多伦多大学）
Spec-LLaVA: Accelerating Vision-Language Models with Dynamic Tree-Based Speculative Decoding
Carnegie Mellon University（卡内基梅隆大学）；University of Nottingham（诺丁汉大学）；The University of Hong Kong（香港大学）；The Hong Kong Polytechnic University（香港理工大学）；Columbia University（哥伦比亚大学）；University of California, Berkeley（加州大学伯克利分校）
ForestPrune: High-ratio Visual Token Compression for Video Multimodal Large Language Models via Spatial-Temporal Forest Modeling
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University（厦门大学多媒体可信感知与高效计算教育部重点实验室）；National University of Defense Technology（国防科技大学）
Evaluating Multimodal Large Language Models on Educational Textbook Question Answering
Faculty of Computing and Information Technology（计算与信息科技学院）；King Abdulaziz University（阿卜杜勒阿齐兹大学）；FAST School of Computing（快速计算学院）；National University of Computer and Emerging Sciences（国家计算机与新兴科学大学）；School of Computing（计算学院）；Macquarie University（麦考瑞大学）；Center of Research Excellence in AI and Data Science（人工智能与数据科学卓越研究中心）
MM-Instruct: Generated Visual Instructions for Large Multimodal Model Alignment
CUHK MMLab（香港中文大学多媒体实验室）；Waseda University（早稻田大学）；Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院）；SenseTime Research（商汤科技研究院）；Shanghai AI Laboratory（上海人工智能实验室）；CPII under InnoHK（创新香港旗下的CPII机构）
PG-Video-LLaVA: Pixel Grounding Large Video-Language Models
Mohamed bin Zayed University of AI（穆罕默德·本·扎耶德人工智能大学）；Australian National University（澳大利亚国立大学）；University of Central Florida（中佛罗里达大学）；Linköping University（林雪平大学）
TC-LLaVA: Rethinking the Transfer from Image to Video Understanding with Temporal Considerations
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Tencent PCG（腾讯平台与内容事业群）；The Hong Kong University of Science and Technology（香港科技大学）；Dalian University of Technology（大连理工大学）
Speculative Decoding Reimagined for Multimodal Large Language Models
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University（多媒体可信感知与高效计算重点实验室、中华人民共和国教育部、厦门大学）
