Recently, Multi-modal Large Language Models (MLLMs) have shown remarkable effectiveness for multi-modal tasks due to their abilities to generate and understand cross-modal data. However, processing long sequences of visual tokens extracted from visual backbones poses a challenge for deployment in real-time applications. To address this issue, we introduce FOLDER, a simple yet effective plug-and-play module designed to reduce the length of the visual token sequence, mitigating both computational and memory demands during training and inference. Through a comprehensive analysis of the token reduction process, we analyze the information loss introduced by different reduction strategies and develop FOLDER to preserve key information while removing visual redundancy. We showcase the effectiveness of FOLDER by integrating it into the visual backbone of several MLLMs, significantly accelerating the inference phase. Furthermore, we evaluate its utility as a training accelerator or even performance booster for MLLMs. In both contexts, FOLDER achieves comparable or even better performance than the original models, while dramatically reducing complexity by removing up to 70% of visual tokens.
Short-LVLM: Compressing and Accelerating Large Vision-Language Models by Pruning Redundant Layers
Northwestern Polytechnical University（西北工业大学）；National Engineering Laboratory for Integrated Aero-Space-Ground-Ocean Big Data Application Technology（集成空天地海大数据应用技术国家工程实验室）
MADTP: Multimodal Alignment-Guided Dynamic Token Pruning for Accelerating Vision-Language Transformer
School of Information Science and Technology, Fudan University（复旦大学信息科学与技术学院）；Academy for Engineering and Technology, Fudan University（复旦大学工程技术研究院）；Tsinghua-Berkeley Shenzhen Institute, Tsinghua University（清华大学深圳国际研究生院（清华-伯克利深圳学院））
You Need Multiple Exiting: Dynamic Early Exiting for Accelerating Unified Vision Language Model
North Carolina State University（北卡罗来纳州立大学）；Google Research（谷歌研究院）；Northeastern University（东北大学）；University of Michigan（密歇根大学）；The University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；University of Connecticut（康涅狄格大学）
Fine-Grained Post-Training Quantization for Large Vision Language Models with Quantization-Aware Integrated Gradients
State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA（多模态人工智能系统国家重点实验室，中国科学院自动化所）；School of Artificial Intelligence, UCAS（人工智能学院，中国科学院大学）；Beijing National Research Center for Information Science and Technology（北京信息科学研究中心）；Institute of Artificial Intelligence, USTB（信息科学技术大学人工智能学院）；School of Artificial Intelligence, Beihang University（北京航空航天大学人工智能学院）；Zhongguancun Academy（中关村学院）
Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findings
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University（中国教育部多媒体可信感知与高效计算重点实验室，厦门大学）；Institute of Artificial Intelligence, Xiamen University（厦门大学人工智能研究院）
Thinking with Images as Continuous Actions: Numerical Visual Chain-of-Thought
Nanyang Technological University（南洋理工大学）；University of Science and Technology of China（中国科学技术大学）；Chalmers University of Technology（楚克理工大学）；University of Gothenburg（哥德堡大学）
VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interaction
State Key Laboratory for Novel Software Technology, Nanjing University（南京大学新型软件技术国家重点实验室）；School of Intelligence Science and Technology, Nanjing University（南京大学智能科学与技术学院）；Tencent Youtu Lab（腾讯优图实验室）；XMU（厦门大学）；CASIA（中国科学院自动化研究所）
TrajFlow: Multi-modal Motion Prediction via Flow Matching
University of British Columbia（不列颠哥伦比亚大学）；Vector Institute for AI（向量人工智能研究所）；University of Waterloo（滑铁卢大学）；Georgia Institute of Technology（佐治亚理工学院）；Carnegie Mellon University（卡内基梅隆大学）；Tesla（特斯拉）；XPeng Motors（小鹏汽车）；Leapmotor；Nvidia（英伟达）；DiDi（滴滴出行）；Canada CIFAR AI Chair（加拿大 CIFAR 人工智能主席）
A multi-modal dataset for insect biodiversity with imagery and DNA at the trap and individual level
Department of Ecology, Swedish University of Agricultural Sciences（生态系，瑞典农业科学大学）；University of Guelph（圭尔夫大学）；Vector Institute（向量研究所）；University of Helsinki（赫尔辛基大学）；Department of Entomology, National Museum of Natural History, Smithsonian Institution（昆虫学系，国家自然历史博物馆，史密松尼研究所）；Department of Biological and Environmental Science, University of Jyväskylä（生物与环境科学系，于韦斯屈埃大学）；Faculty of Biological and Environmental Sciences, University of Helsinki（生物与环境科学学院，赫尔辛基大学）；Centre for Biodiversity Genomics, University of Guelph（生物多样性基因组中心，圭尔夫大学）；NVIDIA（NVIDIA公司）；University of Toronto（多伦多大学）；Kilpisjärvi Biological Station, University of Helsinki（基利皮斯杰里生物学站，赫尔辛基大学）；Unit for Field-based Forest Research, Swedish University of Agricultural Sciences（基于现场的森林研究单位，瑞典农业科学大学）
TransMA: an explainable multi-modal deep learning model for predicting properties of ionizable lipid nanoparticles in mRNA delivery
Shanghai Advanced Research Institute, Chinese Academy of Sciences（中国科学院上海高等研究院）；University of Chinese Academy of Sciences（中国科学院大学）；University of Tsukuba（筑波大学）；Central China Normal University（华中师范大学）
MeanFuser: Fast One-Step Multi-Modal Trajectory Generation and Adaptive Reconstruction via MeanFlow for End-to-End Autonomous Driving
SKL-MAIS, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所模式识别国家重点实验室）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Xiaomi EV（小米汽车）；Institute for AI Industry Research (AIR), Tsinghua University（清华大学智能产业研究院）
Zhejiang University（浙江大学）；University of Technology Sydney（悉尼科技大学）
Recently, large-scale pre-training methods like CLIP have made great progress in multi-modal research such as text-video retrieval. In CLIP, transformers are vital for modeling complex multi-modal relations. However, in the vision transformer of CLIP, the essential visual tokenization process, which produces discrete visual token sequences, generates many homogeneous tokens due to the redundancy nature of consecutive and similar frames in videos. This significantly increases computation costs and hinders the deployment of video retrieval models in web applications. In this paper, to reduce the…
Spectral Evolution-Guided Token Pruning in Multimodal Large Language Models
School of Software Technology, Zhejiang University（浙江大学软件学院）；Zhejiang Key Laboratory of Digital-Intelligence Service Technology（浙江省数字化服务技术重点实验室）；The University of Queensland（昆士兰大学）；Singapore Management University（新加坡管理大学）；The University of Southern Queensland（南昆士兰大学）
