POP: Online Structural Pruning Enables Efficient Inference of Large Foundation Models
Korea Advanced Institute of Science and Technology(韩国科学技术院);University of Tokyo, Tokyo, Japan(东京大学);Tokyo Institute of Technology, Tokyo, Japan(东京技术大学)
Yi Chen, Wonjin Shin, Shuhong Liu, Tho Mai, Jeongmo Lee, Chuanbo Hua, Kun Wang, Jun Liu, Joo-Young Kim
Large foundation models (LFMs) achieve strong performance through scaling, yet current structural pruning methods derive fixed pruning decisions during inference, overlooking sparsity patterns that emerge in the autoregressive token generation. In this paper, we propose POP (Partition-guided Online Pruning), an efficient online structural pruning framework that enables context-conditioned dynamic pruning with minimal computational overhead. POP partitions model channels into retained, candidate, and pruned regions, where prefilling defines a coarse pruning partition, and the decoding stage generates a fine-grained mask within the candidate region, avoiding full-channel re-evaluation. The coarse pruning partition preserves consistently important weights, while the fine-grained masking provides context-conditioned variation during decoding. Moreover, POP is a lightweight, plug-and-play method that requires no preprocessing, including offline calibration, retraining, or learning predictors. Extensive evaluations across diverse LFMs, including large language models (LLMs), mixture-of-experts models (MoEs), and vision-language models (VLMs), demonstrate that POP consistently delivers higher accuracy than existing pruning approaches while incurring smaller computational overhead and minimizing inference latency.
Beyond Linear Approximations: A Novel Pruning Approach for Attention Matrix
The University of Hong Kong(香港大学);University of Wisconsin-Madison(威斯康星大学麦迪逊分校);South China University of Technology(华南理工大学);The Simons Institute for the Theory of Computing(西蒙斯理论计算研究所);University of Pennsylvania(宾夕法尼亚大学)
Towards Efficient VLMs: Information-Theoretic Driven Compression via Adaptive Structural Pruning
School of Artificial Intelligence, Beijing Normal University(北京师范大学人工智能学院);Zhongtai Securities Institute for Financial Studies, Shandong University(山东大学中泰证券金融研究学院)