Large Language Models (LLMs) and Vision-Language Models (VLMs) have demonstrated remarkable capabilities. However, their deployment is hindered by significant computational costs. Existing structured pruning methods, while hardware-efficient, often suffer from significant accuracy degradation. In this paper, we argue that this failure stems from a stage-agnostic pruning approach that overlooks the asymmetric roles between the prefill and decode stages. By introducing a virtual gate mechanism, our importance analysis reveals that deep layers are critical for next-token prediction (decode) but largely redundant for context encoding (prefill). Leveraging this insight, we propose Prefill-Only Pruning (POP), a stage-aware inference strategy that safely omits deep layers during the computationally intensive prefill stage while retaining the full model for the sensitive decode stage. To enable the transition between stages, we introduce independent Key-Value (KV) projections to maintain cache integrity, and a boundary handling strategy to ensure the accuracy of the first generated token. Extensive experiments on Llama-3.1, Qwen3-VL, and Gemma-3 across diverse modalities demonstrate that POP achieves up to 1.37$\times$ speedup in prefill latency with minimal performance loss, effectively overcoming the accuracy-efficiency trade-off limitations of existing structured pruning methods.
AdaptInfer: Adaptive Token Pruning for Vision-Language Model Inference via Dynamical Text Guidance
Global Innovation Exchange, Tsinghua University(清华大学全球创新交流中心);Department of Automation, Tsinghua University(清华大学自动化系);School of Computer Science, Peking University(北京大学计算机科学系)
CoViPAL: Layer-wise Contextualized Visual Token Pruning for Large Vision-Language Models
School of Artificial Intelligence, Wuhan University(武汉大学人工智能学院);School of Computer Science, Wuhan University(武汉大学计算机学院);School of Computer Science, Shanghai Jiao Tong University(上海交通大学计算机学院);Cognitive AI Lab, Shanghai Huawei Technologies, China(上海华为技术有限公司认知人工智能实验室)
SpecVLM: Enhancing Speculative Decoding of Video LLMs via Verifier-Guided Token Pruning
The State Key Laboratory of Blockchain and Data Security(区块链与数据安全国家重点实验室);Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security(杭州高新技术区(滨江)区块链与数据安全研究院);Department of Computing, The Hong Kong Polytechnic University(香港理工大学计算机系);School of Computer Science, Beijing University of Posts and Telecommunications(北京邮电大学计算机学院)
POP: Online Structural Pruning Enables Efficient Inference of Large Foundation Models
Korea Advanced Institute of Science and Technology(韩国科学技术院);University of Tokyo, Tokyo, Japan(东京大学);Tokyo Institute of Technology, Tokyo, Japan(东京技术大学)
MVPruner: Dynamic Token Pruning for Accelerating Multi-view Vision-Language Models in Autonomous Driving
School of Information Engineering, Chang’an University, China(长安大学信息工程学院);School of Artificial Intelligence, Shanghai Jiaotong University, China(上海交通大学人工智能学院)
Towards Joint Quantization and Token Pruning of Vision-Language Models
VCIP, College of Computer Science, Nankai University(VCIP,计算机科学学院,南开大学);School of Computer Science and Engineering, Tianjin University of Technology(计算机科学与工程学院,天津工业大学);OPPO Research Institute(OPPO研究院);Academy for Advanced Interdisciplinary Studies, Nankai University(先进跨学科研究院,南开大学);Nankai International Advanced Research Institute, Shenzhen Futian(南开国际先进研究 institutes,深圳福田);Department of Computing, Hong Kong Polytechnic University(计算学院,香港理工大学)
CoCurve: Cross-Module Co-Pruning Curvature for Structured LLM Pruning
College of Computing and Data Science, Nanyang Technological University(南洋理工大学计算与数据科学学院);Interdisciplinary Graduate Programme, Nanyang Technological University(南洋理工大学跨学科研究生项目);School of Electrical and Electronic Engineering, Nanyang Technological University(南洋理工大学电气与电子工程学院)
Prune2Drive: A Plug-and-Play Framework for Accelerating Vision-Language Models in Autonomous Driving
Shanghai Jiao Tong University(上海交通大学);Shanghai AI Laboratory(上海人工智能实验室);Carnegie Mellon University(卡内基梅隆大学);Sichuan University(四川大学);University of Chinese Academy of Sciences(中国科学院大学);The University of Hong Kong(香港大学)
Beyond FLOPs: Benchmarking Real Inference Acceleration of LLM Pruning under a GEMM-Centric Taxonomy
Ningbo Institute of Digital Twin, Eastern Institute of Technology, Ningbo(宁波数字孪生研究院,东方理工大学(宁波));Department of Computing, The Hong Kong Polytechnic University(香港理工大学计算学系);Munich Center for Machine Learning, LMU Munich(慕尼黑大学机器学习慕尼黑中心)