In vision-language models (VLMs), visual tokens usually bear a significant amount of computational overhead despite sparsity of information in them when compared to text tokens. To address this, most existing methods learn a network to prune redundant visual tokens using certain training data. Differently, we propose a text-guided training-free token optimization mechanism dubbed SparseVLM that eliminates the need of extra parameters or fine-tuning costs. Given that visual tokens complement text tokens in VLM's linguistic reasoning, we select relevant text tokens to rate the significance of visual tokens using self-attention matrices and, then, prune visual tokens using the proposed strategy to maximize sparsity while retaining information. In particular, we introduce a rank-based strategy to adaptively determine the sparsification ratio for each layer, alongside a token recycling method that compresses pruned tokens into more compact representations. Experimental results show that SparseVLM increases the efficiency of various VLMs in a number of image and video understanding tasks. For example, LLaVA when equipped with SparseVLM achieves 54% reduction in FLOPs, 37% decrease in CUDA latency while maintaining 97% of its original accuracy. Our code is available at https://github.com/Gumpest/SparseVLMs.
Focus Through Motion: RGB-Event Collaborative Token Sparsification for Efficient Object Detection
School of Information Engineering, Chang’an University（信息工程学院，长安大学）；School of Electronics and Information, Northwestern Polytechnical University（电子信息学院，西北工业大学）；School of Vehicle and Mobility, Tsinghua University（车辆与移动学院，清华大学）
Certainty Is Redundant: Token Sparsification for Efficient Camouflaged Object Detection with Vision Foundation Models
VCIP, College of Computer Science, Nankai University（VCIP，计算机科学学院，南开大学）；Academy for Advanced Interdisciplinary Studies, Nankai University（先进交叉学科研究院，南开大学）；School of Computer Science and Engineering, Tianjin University of Technology（计算机科学与工程学院，天津工业大学）；School of Information and Communication Engineering, UESTC（信息与通信工程学院，电子科技大学）；Nankai International Advanced Research Institute, Shenzhen Futian（南开国际先进研究院，深圳福田）
EventFlash: Towards Efficient MLLMs for Event-Based Vision
Xidian University（西安电子科技大学）；Tsinghua University（清华大学）；Beijing Institute of Technology（北京理工大学）；Guangdong Laboratory of Artificial Intelligence and Digital Economy(SZ)（广东人工智能与数字经济实验室（深圳））
SparseSAM: Structured Sparsification of Activations in Segment Anything Models
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；College of Engineering & Computer Science, VinUniversity（Vin大学工程与计算机科学学院）；VinUni-Illinois Smart Health Center, VinUniversity（Vin大学-伊利诺伊智能健康中心）；DFKI；Max Planck Research School for Intelligent Systems (IMPRS-IS)（马克斯·普朗克智能系统研究学校）；University of Stuttgart（斯图加特大学）
RVSD: Retrieval Vision Sparse Decoding for Mitigating Visual Hallucinations in Large Vision-Language Models
School of Automation, Guangdong University of Technology（广东工业大学自动化学院）；City University of Hong Kong（香港城市大学）；The Second Affiliated Hospital of Guangzhou University of Chinese Medicine（广州中医药大学第二附属医院）
Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understanding
Beijing Academy of Artificial Intelligence（北京人工智能研究院）；Renmin University of China（中国人民大学）；Chinese Academy of Sciences（中国科学院）；Beijing University of Posts and Telecommunications（北京邮电大学）；Peking University（北京大学）；Shanghai Jiaotong University（上海交通大学）
Med-VCD: Mitigating Hallucination for Medical Large Vision Language Models through Visual Contrastive Decoding
Department of computer science, University of Central Florida, Orlando, USA（计算机科学系，中央佛罗里达大学）；Department of Bioengineering, University of Pennsylvania, Philadelphia, PA, USA（生物工程系，宾夕法尼亚大学）；Department of electrical engineering, Columbia university, New York, NY, USA（电气工程系，哥伦比亚大学）；Technical University of Applied Sciences Regensburg, Regensburg, Germany（应用科学技术大学（雷根斯堡））；Department of Surgery, University of Calgary, Calgary, Alberta, Canada（外科系，卡尔加里大学）；University College of Nabi Akram, Tabriz, Iran（纳比阿克兰大学）；School of Electrical Engineering, Iran University of Science and Technology, Tehran, Iran（电气工程学院，伊朗科学技术大学）
FARTrack: Fast Autoregressive Visual Tracking with High Performance
School of Software Engineering, Xi’an Jiaotong University（西安交通大学软件工程学院）；State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Xi’an Jiaotong University（西安交通大学人机混合增强智能国家重点实验室）；DAMO Academy, Alibaba Group（阿里云达摩院）；National University of Singapore（新加坡国立大学）
Event-Priori-Based Vision-Language Model for Efficient Visual Understanding
Center for Project-Based Learning D-ITET, ETH Zürich, Switzerland（项目基于学习中心D-ITET，苏黎世联邦理工学院，瑞士）；State Key Laboratory of Industrial Control Technology, Zhejiang University, China（工业控制技术国家重点实验室，浙江大学，中国）
Sparse-Tuning: Adapting Vision Transformers with Efficient Fine-tuning and Inference
College of Systems Engineering and State Key Laboratory of Digital Intelligent Modeling and Simulation, National University of Defense Technology（系统工程学院和数字智能建模与仿真国家重点实验室，国防科技大学）；College of Electronics and Information Engineering, Sichuan University（电子信息工程学院，四川大学）；Key Laboratory of Knowledge Engineering with Big Data, Hefei University of Technology（大数据知识工程重点实验室，合肥工业大学）；College of Computer Science and Technology, Zhejiang University（计算机科学与技术学院，浙江大学）；School of Engineering, Westlake University（工程学院，西湖大学）；Tsinghua University（清华大学）；State Key Laboratory for Novel Software Technology, Nanjing University（新型软件技术国家重点实验室，南京大学）；Key Laboratory of Education Blockchain and Intelligent Technology, Ministry of Education, Guangxi Normal University（教育区块链与智能技术重点实验室，教育部，广西师范大学）
