视觉语言模型（VLMs）在推理时需要大量计算资源，因为表示视觉信息需要额外的输入标记。然而，这些视觉标记通常包含冗余和不重要的信息，导致标记数量不必要地增加。为了解决这个问题，我们提出了PACT方法，通过在语言模型的早期层剪枝无关标记并合并视觉冗余标记来减少推理时间和内存使用。我们的方法使用一种新的重要性度量来识别不重要的标记，无需依赖注意力分数，使其与FlashAttention兼容。我们还提出了一种新的聚类算法，称为距离受限密度峰值聚类（Distance Bounded Density Peak Clustering），该算法在通过预定义阈值约束簇内元素之间距离的同时，高效地对视觉标记进行聚类。我们通过广泛的实验证明了PACT的有效性。
英文摘要：
Visual Language Models require substantial computational resources for inference due to the additional input tokens needed to represent visual information. However, these visual tokens often contain redundant and unimportant information, resulting in an unnecessarily high number of tokens. To address this, we introduce PACT, a method that reduces inference time and memory usage by pruning irrelevant tokens and merging visually redundant ones at an early layer of the language model. Our approach uses a novel importance metric to identify unimportant tokens without relying on attention scores, making it compatible with FlashAttention. We also propose a novel clustering algorithm, called Distance Bounded Density Peak Clustering, which efficiently clusters visual tokens while constraining the distances between elements within a cluster by a predefined threshold. We demonstrate the effectiveness of PACT through extensive experiments.
Reinforcement Learning-based Token Pruning in Vision Transformers: A Markov Game Approach
School of Computer Science, Fudan University（复旦大学计算机科学学院）；Data Intelligence Institute of Paris（巴黎数据智能研究院）；Université Paris Cité（巴黎西岱大学）；Shijiazhuang Tiedao University（石家庄铁道大学）
Block-based Symmetric Pruning and Fusion for Efficient Vision Transformers
College of Artificial Intelligence and Green Energy National Yang Ming Chiao Tung University（人工智能与绿色能源学院，阳明交通大学）；Computer Science Department University at Albany, SUNY（计算机科学系，阿尔巴尼大学）；Computer Science Information Engineering Department University at Albany, SUNY（计算机科学与信息工程系，阿尔巴尼大学）
While vision transformers have achieved impressive results, effectively and efficiently accelerating these models can further boost performances. In this work, we propose a dense/sparse training framework to obtain a unified model, enabling weight sharing across various token densities. Thus one model offers a range of accuracy and throughput tradeoffs for different applications. Besides, we introduce adaptive token pruning to optimize the patch token sparsity based on the input image. In addition, we investigate knowledge distillation to enhance token selection capability in early transformer…
Layer-Aware Position Embeddings for Visual Token Pruning in Multimodal Large Language Models
Tongji University（同济大学）；University of California, Santa Cruz（加州大学圣克鲁兹分校）；East China Normal University（华东师范大学）；Shanghai Eye Disease Prevention and Treatment Center（上海市眼病防治中心）
OTPrune: Distribution-Aligned Visual Token Pruning via Optimal Transport
Morgan Stanley（摩根士丹利）；Clemson University（克莱姆森大学）；Arizona State University（亚利桑那州立大学）；Washington University in St. Louis（圣路易斯华盛顿大学）；Texas A&M University（德克萨斯农工大学）；Rice University（莱斯大学）；Florida State University（佛罗里达州立大学）
Convolutional neural network (CNN) has been widely used for vision-based tasks. Due to the high computational complexity and memory storage requirement, it is hard to directly deploy a full-precision CNN on embedded devices. The hardware-friendly designs are needed for re-source-limited and energy-constrained embed-ded devices. Emerging solutions are adopted for the neural network compression, e.g., bina-ry/ternary weight network, pruned network and quantized network. Among them, Binarized Neural Network (BNN) is believed to be the most hardware-friendly framework due to its small network size…
MADTP: Multimodal Alignment-Guided Dynamic Token Pruning for Accelerating Vision-Language Transformer
School of Information Science and Technology, Fudan University（复旦大学信息科学与技术学院）；Academy for Engineering and Technology, Fudan University（复旦大学工程技术研究院）；Tsinghua-Berkeley Shenzhen Institute, Tsinghua University（清华大学深圳国际研究生院（清华-伯克利深圳学院））
Spectral Evolution-Guided Token Pruning in Multimodal Large Language Models
School of Software Technology, Zhejiang University（浙江大学软件学院）；Zhejiang Key Laboratory of Digital-Intelligence Service Technology（浙江省数字化服务技术重点实验室）；The University of Queensland（昆士兰大学）；Singapore Management University（新加坡管理大学）；The University of Southern Queensland（南昆士兰大学）
TAMP: Token-Adaptive Layerwise Pruning in Multimodal Large Language Models
University of North Carolina Chapel Hill（北卡罗来纳大学教堂山分校）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Massachusetts Institute of Technology（麻省理工学院）；University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）；Hong Kong University of Science and Technology（香港科学与技术大学）
The Better You Learn, The Smarter You Prune: Towards Efficient Vision-language-action Models via Differentiable Token Pruning
LiAuto Inc.（LiAuto公司）；School of Vehicle and Mobility, Tsinghua University（清华大学车辆与移动学院）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）
2D or 3D: Who Governs Salience in VLA Models? -- Tri-Stage Token Pruning Framework with Modality Salience Awareness
School of Computer Science, Peking University（北京大学计算机科学学院）；ZTE Corporation（中兴通讯）；School of Artificial Intelligence, Beijing Normal University（北京师范大学人工智能学院）；School of Computer Science, China University of Geosciences (Wuhan)（中国地质大学（武汉）计算机科学学院）；School of Electronics Engineering and Computer Science, Peking University（北京大学电子工程与计算机科学学院）
