In Vision Language Models (VLMs), vision tokens are quantity-heavy yet information-dispersed compared with language tokens, thus consume too much unnecessary computation. Pruning redundant vision tokens for high VLM inference efficiency has been continuously studied but all existing methods resort to indirect and non-guaranteed ways. We propose OC-VTP, a direct and guaranteed approach to select the most representative vision tokens for high-efficiency yet accuracy-preserving VLM inference. Our OC-VTP requires merely light-weight pre-training of a small object-centric vision token pruner, which can then be inserted into existing VLMs, without fine-tuning of any models on any datasets. It is gauranteed that the most representative vision tokens are kept by minimizing the error in reconstructing the original unpruned tokens from the selected ones. Across any vision pruning ratios, i.e., inference efficiency, our OC-VTP consistently helps mainstream VLMs to preserve the highest inference accuracy. Our pruning also demonstrates interesting interpretability. Our codes are available at https://github.com/GarryLarry010131/OC-VTP.
How Many Visual Tokens Do Multimodal Language Models Need? Scaling Visual Token Pruning with F^3A
School of Computer Science and Engineering, Northeastern University, Shenyang 110819, China（东北大学计算机科学与工程学院，沈阳 110819，中国）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；School of Computer and Communication Engineering, Northeastern University, Qinhuangdao 066004, China（东北大学计算机与通信工程学院，秦皇岛 066004，中国）
ST-Prune: Training-Free Spatio-Temporal Token Pruning for Vision-Language Models in Autonomous Driving
School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Carizon；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMs
National Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University（国家多媒体信息处理重点实验室，计算机学院，北京大学）；ByteDance（字节跳动）
TOPS: First-Principles Visual Token Pruning via Constructing Token Optimal Preservation Sets for Efficient MLLM Inference
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University（北京大学计算机学院多媒体信息处理国家重点实验室）；University of Electronic Science and Technology of China（电子科技大学）；Nanyang Technological University（南洋理工大学）；Beijing Academy of Artificial Intelligence (BAAI)（北京人工智能研究院）
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, CAS, Beijing, China（多模态人工智能系统国家重点实验室，自动化研究所，中国科学院，北京，中国）；School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China（人工智能学院，中国科学院大学，北京，中国）；Fanyu AI Laboratory, Zhongke Fanyu Technology Co., Ltd, Beijing, China（凡语AI实验室，中科创始人技术有限公司，北京，中国）
Nüwa: Mending the Spatial Integrity Torn by VLM Token Pruning
Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)（人工智能与数字经济广东实验室）；School of Artificial Intelligence, Xidian University（西安电子科技大学人工智能学院）；The Hong Kong Polytechnic University（香港理工大学）；Great Bay University（大鹏大学）；Shenzhen University（深圳大学）；Huawei（华为）
When Token Pruning is Worse than Random: Understanding Visual Token Information in VLLMs
Tongji University（同济大学）；University of California, Santa Cruz（加州大学圣克ruz分校）；Amazon（亚马逊）；East China Normal University（华东师范大学）；Shanghai Eye Disease Prevention and Treatment Center（上海眼病预防与治疗中心）
AdaptInfer: Adaptive Token Pruning for Vision-Language Model Inference via Dynamical Text Guidance
Global Innovation Exchange, Tsinghua University（清华大学全球创新交流中心）；Department of Automation, Tsinghua University（清华大学自动化系）；School of Computer Science, Peking University（北京大学计算机科学系）
CoViPAL: Layer-wise Contextualized Visual Token Pruning for Large Vision-Language Models
School of Artificial Intelligence, Wuhan University（武汉大学人工智能学院）；School of Computer Science, Wuhan University（武汉大学计算机学院）；School of Computer Science, Shanghai Jiao Tong University（上海交通大学计算机学院）；Cognitive AI Lab, Shanghai Huawei Technologies, China（上海华为技术有限公司认知人工智能实验室）
ERASE: Eliminating Redundant Visual Tokens via Adaptive Two-Stage Token Pruning
Department of Electrical and Computer Engineering, Sungkyunkwan University, Republic of Korea（电气与计算机工程系，成均馆大学，大韩民国）；Department of Semiconductor Systems Engineering, Sungkyunkwan University, Republic of Korea（半导体系统工程系，成均馆大学，大韩民国）
